| *** achanda has joined #openstack-infra | 00:00 | |
| *** glauco_ has joined #openstack-infra | 00:00 | |
| *** Swami has quit IRC | 00:00 | |
| *** dimtruck is now known as zz_dimtruck | 00:01 | |
| *** camunoz has quit IRC | 00:01 | |
| *** ybathia has quit IRC | 00:03 | |
| *** glauco_ has quit IRC | 00:04 | |
| *** achanda has quit IRC | 00:04 | |
| *** yamamoto has joined #openstack-infra | 00:05 | |
| *** yamamoto has quit IRC | 00:05 | |
| *** yamamoto has joined #openstack-infra | 00:05 | |
| *** sflanigan has quit IRC | 00:05 | |
| fungi | jgriffith: here's what i find from the zuul logs from when you reapplied the approval... 2015-09-01 18:45:58,768 DEBUG zuul.DependentPipelineManager: Change <Change 0x7fa73895ae90 212861,11> in project openstack/cinder does not share a change queue with <Change 0x7fa68cbde510 215847,2> in project openstack/python-cinderclient | 00:07 |
|---|---|---|
| fungi | 2015-09-01 18:45:58,769 DEBUG zuul.DependentPipelineManager: Failed to enqueue changes ahead of <Change 0x7fa68cbde510 215847,2> | 00:07 |
| fungi | 212861,11 seems to be the most recent patchset for the depends-on | 00:08 |
| jhesketh | fungi: er sorry, got a busy morning this morning. Was there something about the swift indexes you wanted to talk about? | 00:08 |
| fungi | which didn't merge until 19:29:44 | 00:08 |
| fungi | jhesketh: nope, clarkb pointed me to the work in progress change for it | 00:08 |
| *** shashank_hegde has joined #openstack-infra | 00:09 | |
| jhesketh | fungi: yeah, I broke it last night, hopefully will fix whatever it is this morning | 00:09 |
| jhesketh | (actually it'll have to afternoon) | 00:09 |
| jhesketh | I also want to turn some more jobs over to swift logs, so will look at that next | 00:10 |
| fungi | jgriffith: so, while that's a known behavior (you can't enqueue a change via approval with a cross-repo dependency on a commit in another repo which doesn't share any of the same job names), i find it odd that cinder and cinderclient wouldn't share a change queue | 00:10 |
| *** sharikapongubala has quit IRC | 00:10 | |
| fungi | rather, can | 00:10 |
| fungi | 't enqueue it until the dependency merges | 00:10 |
| fungi | jgriffith: mtanino: ahh, yep that's correct. cinder and cinderclient really don't share a change queue (they have no jobs in common with each other at all) | 00:12 |
| *** achanda has joined #openstack-infra | 00:12 | |
| fungi | jgriffith: mtanino: so anyway, if you have a cinderclient change depends-on a cinder change (or vice versa) you need to wait to approve the second until the first merges | 00:12 |
| *** lucas-dinner has quit IRC | 00:12 | |
| *** darrenc is now known as darrenc_afk | 00:12 | |
| fungi | jgriffith: mtanino: now that 212861 has merged, if you remove and readd your approval workflow +1 vote on 215847 it should head into the gate | 00:13 |
| mtanino | fungi: I see. It's a know behaviour... | 00:13 |
| mtanino | fungi: jgriffith is getting back to home so I will ask him later to remove and readd the request. Thank you. | 00:14 |
| *** fitoduarte has quit IRC | 00:14 | |
| *** salv-orlando has quit IRC | 00:15 | |
| *** dims__ has quit IRC | 00:16 | |
| *** jeadre has quit IRC | 00:16 | |
| *** camunoz has joined #openstack-infra | 00:17 | |
| *** jeadre has joined #openstack-infra | 00:17 | |
| *** pvaneck has quit IRC | 00:21 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues https://review.openstack.org/219514 | 00:22 |
| *** sflanigan has joined #openstack-infra | 00:22 | |
| *** ashleighfarnham has quit IRC | 00:23 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/project-config: Create a job to ease manpage reviews https://review.openstack.org/218493 | 00:26 |
| *** tjones has joined #openstack-infra | 00:27 | |
| *** baoli has quit IRC | 00:27 | |
| *** darrenc_afk is now known as darrenc | 00:29 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues https://review.openstack.org/219514 | 00:30 |
| *** zhenguo has joined #openstack-infra | 00:30 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 00:32 |
| *** wenlock has quit IRC | 00:33 | |
| *** dguitarbite has quit IRC | 00:37 | |
| *** dbite has joined #openstack-infra | 00:37 | |
| *** puranamr has quit IRC | 00:38 | |
| *** claudiub has quit IRC | 00:39 | |
| *** sputnik13 has quit IRC | 00:39 | |
| *** vivekd has quit IRC | 00:40 | |
| *** dannywil_ has quit IRC | 00:43 | |
| *** dannywilson has joined #openstack-infra | 00:43 | |
| *** yamamoto has quit IRC | 00:44 | |
| *** puranamr has joined #openstack-infra | 00:44 | |
| *** jswarren has joined #openstack-infra | 00:45 | |
| *** dannywilson has quit IRC | 00:45 | |
| *** yamamoto has joined #openstack-infra | 00:46 | |
| *** jswarren has quit IRC | 00:46 | |
| *** jswarren has joined #openstack-infra | 00:47 | |
| *** morgabra has quit IRC | 00:50 | |
| *** morgabra has joined #openstack-infra | 00:50 | |
| *** dims__ has joined #openstack-infra | 00:51 | |
| *** tjones has quit IRC | 00:52 | |
| jgriffith | fungi: ahh... ok, thank you very muc | 00:52 |
| jgriffith | fungi: I have to admit the cinder dependency didn't register when I looked at it | 00:53 |
| *** Sukhdev has quit IRC | 00:55 | |
| *** dims__ has quit IRC | 00:56 | |
| *** sabeen1 has quit IRC | 00:56 | |
| *** xyang1 has joined #openstack-infra | 00:57 | |
| *** kaisers has quit IRC | 01:01 | |
| *** kaisers has joined #openstack-infra | 01:03 | |
| *** asettle is now known as asettle-afk | 01:03 | |
| *** r-daneel has quit IRC | 01:04 | |
| *** bknudson has quit IRC | 01:04 | |
| *** puranamr has quit IRC | 01:05 | |
| *** hdd has joined #openstack-infra | 01:06 | |
| *** puranamr has joined #openstack-infra | 01:06 | |
| *** achanda has quit IRC | 01:09 | |
| *** ajmiller has joined #openstack-infra | 01:12 | |
| *** mtanino has quit IRC | 01:13 | |
| *** stevemar has joined #openstack-infra | 01:15 | |
| *** vivekd has joined #openstack-infra | 01:16 | |
| *** ddieterly has joined #openstack-infra | 01:16 | |
| *** btully has joined #openstack-infra | 01:17 | |
| *** thorst has joined #openstack-infra | 01:17 | |
| pabelanger | yolanda: rcarrillocruz: Making good progress on ansible-role-nodepool. I should have something to show you by next week, won't get much time to play with it after today. | 01:20 |
| *** zeih has joined #openstack-infra | 01:21 | |
| *** skylerberg has quit IRC | 01:22 | |
| *** jeadre has quit IRC | 01:22 | |
| *** tiswanso has joined #openstack-infra | 01:22 | |
| *** armax has quit IRC | 01:22 | |
| *** jeadre has joined #openstack-infra | 01:23 | |
| *** tiswanso has quit IRC | 01:25 | |
| *** mpopow_ has joined #openstack-infra | 01:25 | |
| *** zeih has quit IRC | 01:26 | |
| *** tiswanso has joined #openstack-infra | 01:26 | |
| *** baoli has joined #openstack-infra | 01:27 | |
| *** baoli has quit IRC | 01:28 | |
| *** baoli has joined #openstack-infra | 01:28 | |
| *** sridhar_ram has joined #openstack-infra | 01:28 | |
| *** sridhar_ram has quit IRC | 01:29 | |
| *** mpopow has quit IRC | 01:29 | |
| *** armax has joined #openstack-infra | 01:35 | |
| *** asettle-afk has quit IRC | 01:36 | |
| *** baoli has quit IRC | 01:37 | |
| *** jeadre has quit IRC | 01:37 | |
| *** baoli has joined #openstack-infra | 01:38 | |
| *** jeadre has joined #openstack-infra | 01:38 | |
| *** vivekd has quit IRC | 01:39 | |
| *** jyuso1 has joined #openstack-infra | 01:40 | |
| *** puranamr has quit IRC | 01:40 | |
| *** puranamr has joined #openstack-infra | 01:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Add docs publish docs for grenade https://review.openstack.org/218513 | 01:41 |
| *** esmute_ has quit IRC | 01:41 | |
| *** armax has quit IRC | 01:46 | |
| *** sharikapongubala has joined #openstack-infra | 01:46 | |
| *** resker has joined #openstack-infra | 01:48 | |
| *** ddieterl_ has joined #openstack-infra | 01:48 | |
| *** vivekd has joined #openstack-infra | 01:50 | |
| *** ddieterly has quit IRC | 01:52 | |
| *** esker has quit IRC | 01:52 | |
| *** sharikapongubala has quit IRC | 01:53 | |
| *** madhu_ak has quit IRC | 01:53 | |
| *** ashleighfarnham has joined #openstack-infra | 01:56 | |
| *** puranamr has quit IRC | 01:57 | |
| *** hdd has quit IRC | 01:58 | |
| *** dingyichen has quit IRC | 02:00 | |
| *** puranamr has joined #openstack-infra | 02:00 | |
| *** ashleighfarnham has quit IRC | 02:01 | |
| *** wenlock has joined #openstack-infra | 02:01 | |
| *** sharikapongubala has joined #openstack-infra | 02:01 | |
| *** achanda has joined #openstack-infra | 02:04 | |
| *** sflanigan has quit IRC | 02:07 | |
| *** camunoz has quit IRC | 02:07 | |
| *** ddieterl_ has quit IRC | 02:11 | |
| *** jswarren_ has joined #openstack-infra | 02:13 | |
| *** w_ has joined #openstack-infra | 02:13 | |
| *** achanda has quit IRC | 02:14 | |
| *** jswarren has quit IRC | 02:17 | |
| *** olaph has quit IRC | 02:17 | |
| *** thorst has quit IRC | 02:17 | |
| *** shashank_hegde has quit IRC | 02:18 | |
| *** olaph has joined #openstack-infra | 02:18 | |
| *** w_ has quit IRC | 02:19 | |
| *** camunoz has joined #openstack-infra | 02:20 | |
| *** sflanigan has joined #openstack-infra | 02:22 | |
| *** zeih has joined #openstack-infra | 02:22 | |
| *** achanda has joined #openstack-infra | 02:23 | |
| *** zeih has quit IRC | 02:26 | |
| *** FallenPegasus has joined #openstack-infra | 02:27 | |
| *** salv-orlando has joined #openstack-infra | 02:27 | |
| *** BharatK has joined #openstack-infra | 02:28 | |
| StevenK | clarkb: Etherpad gone over | 02:28 |
| *** baoli has quit IRC | 02:29 | |
| *** baoli_ has joined #openstack-infra | 02:30 | |
| *** unicell has quit IRC | 02:31 | |
| *** salv-orlando has quit IRC | 02:32 | |
| *** dingyichen has joined #openstack-infra | 02:33 | |
| *** Sree has quit IRC | 02:41 | |
| *** resker has quit IRC | 02:41 | |
| *** annegentle has quit IRC | 02:41 | |
| *** tiswanso has quit IRC | 02:42 | |
| *** tiswanso_ has joined #openstack-infra | 02:42 | |
| *** annegentle has joined #openstack-infra | 02:43 | |
| clarkb | StevenK: thanks, for populating the zanata server with translations we just take what is in git and don't need to touch transifex? | 02:44 |
| clarkb | StevenK: just making sure I understand that bit (not sure if we were able to preserve history or memory or anything like that) | 02:45 |
| *** Sukhdev_ has joined #openstack-infra | 02:45 | |
| *** gyee has quit IRC | 02:47 | |
| *** achanda has quit IRC | 02:49 | |
| StevenK | clarkb: Zanata works out the translation memory from what we import. | 02:52 |
| StevenK | clarkb: populate-zanata-project pulls all translations from Transifex before pushing to Zanata | 02:53 |
| *** annegentle has quit IRC | 02:57 | |
| clarkb | StevenK gotcha and it uses the git repo to know wjere to talk in transifex? | 03:00 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add Rally to projects list https://review.openstack.org/190601 | 03:00 |
| StevenK | clarkb: For that it uses the first argument, but tx (as in the CLI utility) needs somewhere to write to | 03:01 |
| StevenK | clarkb: So it's just easier to just run it from the git repo | 03:02 |
| clarkb | I see | 03:02 |
| *** achanda has joined #openstack-infra | 03:02 | |
| *** changbl has joined #openstack-infra | 03:02 | |
| StevenK | clarkb: And zanata-cli needs to read from the repo, the setup scripts write a zanata.xml into the base directory of it | 03:04 |
| *** achanda has quit IRC | 03:08 | |
| *** asettle has joined #openstack-infra | 03:11 | |
| *** mmedvede has quit IRC | 03:11 | |
| *** sharikapongubala has quit IRC | 03:11 | |
| *** ddieterly has joined #openstack-infra | 03:13 | |
| anteaya | quiet floor complete with crying baby | 03:14 |
| anteaya | sounds very small | 03:14 |
| EmilienM | anteaya: in MTL? | 03:15 |
| *** sabeen1 has joined #openstack-infra | 03:16 | |
| anteaya | yeah | 03:16 |
| EmilienM | anteaya: taking the train in 6h | 03:16 |
| EmilienM | I should go sleep eventually | 03:16 |
| anteaya | wow early | 03:16 |
| anteaya | you should | 03:16 |
| EmilienM | but our CI is in a bad shape today | 03:16 |
| anteaya | what time to you think you will be at the office | 03:16 |
| anteaya | :( | 03:16 |
| EmilienM | I'm trying to get this solved | 03:16 |
| anteaya | sleep now | 03:16 |
| EmilienM | anteaya: ~9am | 03:16 |
| anteaya | solve tomorrow | 03:17 |
| EmilienM | anteaya: come at 9.15am and for sure I'll be here | 03:17 |
| anteaya | kk, I'll aim for being there at about 9 then too | 03:17 |
| EmilienM | nice | 03:17 |
| anteaya | ah 9:15 then | 03:17 |
| *** ddieterly has quit IRC | 03:17 | |
| anteaya | ate at ChuCha, was amazing vegan food | 03:18 |
| anteaya | had such a lovely dinner | 03:18 |
| *** wenlock1 has joined #openstack-infra | 03:18 | |
| EmilienM | anteaya: I might have an outstanding request | 03:18 |
| EmilienM | I'm testing now | 03:19 |
| anteaya | request away | 03:20 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: enable nova v20 compat testing jobs https://review.openstack.org/219370 | 03:21 |
| anteaya | as soon as I am finished reading the logs from the tc meeting I am signing off though | 03:21 |
| anteaya | I need to sleep too | 03:21 |
| *** zeih has joined #openstack-infra | 03:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting puppet-keystone beaker on trusty https://review.openstack.org/219543 | 03:23 |
| EmilienM | anteaya: ^ | 03:23 |
| EmilienM | sorry for asking stuffs so late but this is an outstanding request that will unblock our CI so we can do our Puppet sprint safely tomorrow | 03:24 |
| EmilienM | clarkb: if you're still around ^ | 03:24 |
| *** tiswanso_ has quit IRC | 03:24 | |
| *** glauco has joined #openstack-infra | 03:26 | |
| anteaya | I am done reading the tc meeting logs | 03:27 |
| anteaya | EmilienM: did you need me for a thing? | 03:27 |
| anteaya | I'm running out of reasons to stay online | 03:27 |
| anteaya | and I've testing all my pillows and none are feather | 03:27 |
| anteaya | so all eight of them get to spend the time in a stack on the other bed | 03:27 |
| anteaya | and this pillowless bed is looking more and more appealing | 03:27 |
| *** glauco is now known as 7JTAALLKW | 03:27 | |
| anteaya | I don't know why people need so many pillows | 03:27 |
| anteaya | but some folks like many pillows | 03:27 |
| EmilienM | anteaya: https://review.openstack.org/219543 | 03:27 |
| * anteaya reviews | 03:28 | |
| *** sharikapongubala has joined #openstack-infra | 03:28 | |
| *** zeih has quit IRC | 03:28 | |
| anteaya | EmilienM: +2 from me | 03:30 |
| EmilienM | anteaya: thx | 03:30 |
| anteaya | hopefully someone like clarkb or jhesketh will wander by soon and shove it in | 03:30 |
| anteaya | get some sleep | 03:30 |
| anteaya | see you tomorrow | 03:30 |
| anteaya | g'night | 03:30 |
| EmilienM | anteaya: thx | 03:30 |
| EmilienM | see you tomorrow in person | 03:30 |
| anteaya | welcome :) | 03:30 |
| * jhesketh 's ears burn | 03:30 | |
| anteaya | 'til tomorrow | 03:30 |
| anteaya | hi jhesketh | 03:31 |
| anteaya | g'night jhesketh | 03:31 |
| *** 7JTAALLKW has quit IRC | 03:31 | |
| jhesketh | heh, goodnight! | 03:31 |
| *** dchen has joined #openstack-infra | 03:32 | |
| *** camunoz has quit IRC | 03:33 | |
| *** Sree has joined #openstack-infra | 03:35 | |
| *** esker has joined #openstack-infra | 03:35 | |
| *** sflanigan has quit IRC | 03:35 | |
| *** btully has quit IRC | 03:36 | |
| *** baoli_ has quit IRC | 03:36 | |
| openstackgerrit | Merged openstack-infra/project-config: zuul/layout: make non-voting puppet-keystone beaker on trusty https://review.openstack.org/219543 | 03:37 |
| Nakato | fungi: clarkb: Could I get you guys to take a look at https://review.openstack.org/207262 ? It's the project-config part of the constraints work. | 03:37 |
| *** woodster_ has quit IRC | 03:39 | |
| *** esker has quit IRC | 03:40 | |
| *** salv-orlando has joined #openstack-infra | 03:41 | |
| *** hdd has joined #openstack-infra | 03:43 | |
| *** salv-orlando has quit IRC | 03:46 | |
| *** maishsk has quit IRC | 03:47 | |
| *** links has joined #openstack-infra | 03:48 | |
| *** puranamr has quit IRC | 03:49 | |
| *** camunoz has joined #openstack-infra | 03:49 | |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| *** armax has joined #openstack-infra | 03:52 | |
| *** puranamr has joined #openstack-infra | 03:52 | |
| *** dims__ has joined #openstack-infra | 04:03 | |
| *** jerryz has quit IRC | 04:03 | |
| *** tlian2 has quit IRC | 04:05 | |
| *** unicell has joined #openstack-infra | 04:10 | |
| *** hichihara has joined #openstack-infra | 04:11 | |
| *** bhunter71 has quit IRC | 04:11 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Remove nova v21 old jobs https://review.openstack.org/219553 | 04:11 |
| *** ddieterly has joined #openstack-infra | 04:14 | |
| *** unicell has quit IRC | 04:16 | |
| *** xyang1 has quit IRC | 04:17 | |
| *** dims__ is now known as dims | 04:17 | |
| *** ddieterly has quit IRC | 04:19 | |
| *** ihrachys has joined #openstack-infra | 04:22 | |
| *** rlandy has joined #openstack-infra | 04:22 | |
| *** zeih has joined #openstack-infra | 04:23 | |
| *** dims has quit IRC | 04:24 | |
| *** unicell has joined #openstack-infra | 04:24 | |
| *** stevemar has quit IRC | 04:25 | |
| *** zeih has quit IRC | 04:28 | |
| *** esker has joined #openstack-infra | 04:30 | |
| *** ajmiller has quit IRC | 04:31 | |
| *** ajmiller has joined #openstack-infra | 04:32 | |
| zaro | jhesketh: could you pls help with this gate-dsvm-nodepool failure?? https://review.openstack.org/#/c/200727/ | 04:32 |
| *** btully has joined #openstack-infra | 04:33 | |
| *** sabeen1 has quit IRC | 04:33 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting puppet-neutron beaker on trusty https://review.openstack.org/219556 | 04:33 |
| *** yfried has quit IRC | 04:33 | |
| *** esker has quit IRC | 04:34 | |
| *** jyuso1 has quit IRC | 04:34 | |
| jhesketh | zaro: I'll see what I can do | 04:35 |
| *** kushal has joined #openstack-infra | 04:37 | |
| *** btully has quit IRC | 04:37 | |
| *** mrmartin has joined #openstack-infra | 04:39 | |
| *** achanda has joined #openstack-infra | 04:42 | |
| *** esker has joined #openstack-infra | 04:43 | |
| *** puranamr has quit IRC | 04:44 | |
| *** Sukhdev_ has quit IRC | 04:44 | |
| *** puranamr has joined #openstack-infra | 04:45 | |
| *** esker has quit IRC | 04:49 | |
| zaro | jhesketh: i don't really understand the failure. was wondering if it's a blip or a real failure we need to deal with? | 04:50 |
| jhesketh | zaro: http://logs.openstack.org/27/200727/3/check/gate-dsvm-nodepool/3300544/logs/screen-nodepool.txt.gz#_2015-09-01_18_31_31_164 | 04:50 |
| jhesketh | zaro: trying to figure that out myself.. doing a recheck at the moment | 04:50 |
| *** ihrachys has quit IRC | 04:52 | |
| *** jcoufal has joined #openstack-infra | 04:52 | |
| SlickNik | hi folks — a change in the tarball file names on cloud-images.ubuntu.com is causing a break in the trove gate. | 04:54 |
| *** mrmartin has quit IRC | 04:54 | |
| SlickNik | We have a fix for this at https://review.openstack.org/#/c/219548 — is there any way to prioritize merging this since the gate is pretty long and without it all other trove changes in flight will fail? | 04:56 |
| *** salv-orlando has joined #openstack-infra | 04:56 | |
| *** mtreinish has quit IRC | 04:57 | |
| ianw | SlickNik: it can be promoted, but maybe jhesketh is the only person around right now that has permissions | 04:57 |
| SlickNik | thanks ianw | 04:58 |
| SlickNik | ^ jhesketh: pretty please? | 04:58 |
| SlickNik | it's still in the check queue and should be done in ~7-8 minutes. I'm not sure if you need it to be in the gate queue for it to be promoted. | 05:00 |
| *** salv-orlando has quit IRC | 05:01 | |
| *** mpopow__ has joined #openstack-infra | 05:03 | |
| *** mrmartin has joined #openstack-infra | 05:03 | |
| *** jyuso1 has joined #openstack-infra | 05:04 | |
| *** mrmartin has quit IRC | 05:04 | |
| *** mtreinish has joined #openstack-infra | 05:04 | |
| *** nadya has joined #openstack-infra | 05:05 | |
| *** rossella_s has quit IRC | 05:06 | |
| *** rossella_s has joined #openstack-infra | 05:06 | |
| jhesketh | SlickNik, ianw: promoting it will cause all of the current jobs in the integrated queue to be cancelled and re-queued | 05:06 |
| jhesketh | that's quite a burden | 05:06 |
| *** markvoelker has joined #openstack-infra | 05:06 | |
| jhesketh | we might be better just waiting for it to go through | 05:07 |
| *** mpopow_ has quit IRC | 05:07 | |
| SlickNik | ugh | 05:07 |
| *** unicell has quit IRC | 05:07 | |
| SlickNik | I agree, it's probably not worth that. | 05:07 |
| SlickNik | Given the current queue size. | 05:07 |
| *** rlandy has quit IRC | 05:07 | |
| ianw | oh, i had just assumed everything was broken until this got in, so yeah | 05:07 |
| SlickNik | ianw: No, just trove changes. | 05:08 |
| SlickNik | sorry if I was unclear. | 05:08 |
| ianw | ah, well yeah; np i didn't check :) | 05:08 |
| *** ayoung is now known as ayoung_ZZzz | 05:08 | |
| openstackgerrit | Romil Gupta proposed openstack-infra/project-config: Add check-requirements to networking-vsphere https://review.openstack.org/219558 | 05:09 |
| *** hdd has quit IRC | 05:09 | |
| *** unicell has joined #openstack-infra | 05:09 | |
| openstackgerrit | Romil Gupta proposed openstack/requirements: Add networking-vsphere to the list https://review.openstack.org/218189 | 05:09 |
| *** nadya has quit IRC | 05:09 | |
| *** markvoelker has quit IRC | 05:11 | |
| *** markvoelker has joined #openstack-infra | 05:12 | |
| *** ddieterly has joined #openstack-infra | 05:14 | |
| *** Sukhdev_ has joined #openstack-infra | 05:17 | |
| *** ddieterly has quit IRC | 05:19 | |
| *** flepied has joined #openstack-infra | 05:23 | |
| *** puranamr has quit IRC | 05:23 | |
| *** zeih has joined #openstack-infra | 05:24 | |
| *** Sree has quit IRC | 05:25 | |
| *** puranamr has joined #openstack-infra | 05:26 | |
| *** Sree has joined #openstack-infra | 05:28 | |
| *** zeih has quit IRC | 05:28 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make README.rst a bit more generic https://review.openstack.org/218043 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: doc: migrate from README.rst to Sphinx https://review.openstack.org/193051 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add DIB_KEEP_PACKAGE_CACHE variable https://review.openstack.org/208321 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add DIB_USE_PACKAGE_CACHE variable https://review.openstack.org/211434 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add #!/bin/bash to library functions https://review.openstack.org/218100 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Remove grub2 in redhat-common/pre-install.d/15-remove-grub https://review.openstack.org/211124 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Remove old entries from MANIFEST.in https://review.openstack.org/219561 | 05:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Handle install with pip -e https://review.openstack.org/219562 | 05:29 |
| *** yfried has joined #openstack-infra | 05:33 | |
| *** jyuso has joined #openstack-infra | 05:35 | |
| *** yfried_ has joined #openstack-infra | 05:36 | |
| *** yfried has quit IRC | 05:37 | |
| *** camunoz has quit IRC | 05:37 | |
| *** sflanigan has quit IRC | 05:37 | |
| *** jyuso1 has quit IRC | 05:37 | |
| *** btully has joined #openstack-infra | 05:45 | |
| *** jyuso1 has joined #openstack-infra | 05:49 | |
| *** jyuso has quit IRC | 05:49 | |
| *** btully has quit IRC | 05:49 | |
| *** jcoufal has quit IRC | 05:50 | |
| *** camunoz has joined #openstack-infra | 05:51 | |
| *** salv-orlando has joined #openstack-infra | 05:52 | |
| *** amotoki has joined #openstack-infra | 05:54 | |
| *** dchen has quit IRC | 05:56 | |
| *** amotoki has quit IRC | 05:56 | |
| *** dingyichen has quit IRC | 05:57 | |
| *** flepied has quit IRC | 05:57 | |
| *** vadimk_ has quit IRC | 05:58 | |
| *** amotoki has joined #openstack-infra | 06:00 | |
| *** jerryz has joined #openstack-infra | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/219148 | 06:02 |
| *** amotoki_ has joined #openstack-infra | 06:02 | |
| *** Sukhdev_ has quit IRC | 06:04 | |
| *** yfried_ is now known as yfried|afk | 06:04 | |
| openstackgerrit | Hidekazu Nakamura proposed openstack-infra/infra-manual: Fix pip link https://review.openstack.org/219567 | 06:06 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/219568 | 06:06 |
| *** amotoki has quit IRC | 06:06 | |
| *** yfried|afk is now known as yfried_ | 06:09 | |
| *** jtomasek has joined #openstack-infra | 06:09 | |
| *** FallenPegasus has quit IRC | 06:10 | |
| *** HeOS has quit IRC | 06:10 | |
| *** puranamr has quit IRC | 06:10 | |
| *** abregman has joined #openstack-infra | 06:11 | |
| *** ParsectiX has joined #openstack-infra | 06:11 | |
| *** puranamr has joined #openstack-infra | 06:12 | |
| *** ddieterly has joined #openstack-infra | 06:15 | |
| jhesketh | zaro: the recheck passed fyi | 06:17 |
| *** ddieterly has quit IRC | 06:20 | |
| *** asettle has quit IRC | 06:23 | |
| *** markvoelker has quit IRC | 06:23 | |
| *** sdake has joined #openstack-infra | 06:24 | |
| *** zeih has joined #openstack-infra | 06:25 | |
| *** ildikov has quit IRC | 06:25 | |
| *** zeih has quit IRC | 06:29 | |
| *** mpaolino has joined #openstack-infra | 06:30 | |
| *** mrmartin has joined #openstack-infra | 06:31 | |
| *** mrunge has joined #openstack-infra | 06:34 | |
| *** flepied has joined #openstack-infra | 06:37 | |
| *** stevemar has joined #openstack-infra | 06:38 | |
| *** yfried_ is now known as yfried|afk | 06:38 | |
| *** stevemar has quit IRC | 06:42 | |
| *** glauco has joined #openstack-infra | 06:42 | |
| *** flepied has quit IRC | 06:42 | |
| *** esmute_ has joined #openstack-infra | 06:43 | |
| *** mrmartin has quit IRC | 06:45 | |
| *** yfried|afk is now known as yfried_ | 06:46 | |
| *** glauco has quit IRC | 06:46 | |
| *** achanda has quit IRC | 06:47 | |
| *** mpopow__ has quit IRC | 06:47 | |
| *** puranamr has quit IRC | 06:49 | |
| *** dims has joined #openstack-infra | 06:50 | |
| *** ParsectiX has quit IRC | 06:51 | |
| *** afazekas__ has joined #openstack-infra | 06:52 | |
| *** puranamr has joined #openstack-infra | 06:53 | |
| *** mrmartin has joined #openstack-infra | 06:53 | |
| *** peristeri has joined #openstack-infra | 06:54 | |
| *** dims has quit IRC | 06:54 | |
| *** ildikov has joined #openstack-infra | 06:58 | |
| *** flepied has joined #openstack-infra | 06:59 | |
| *** liam__ has joined #openstack-infra | 06:59 | |
| *** jyuso has joined #openstack-infra | 07:01 | |
| *** esmute_ has quit IRC | 07:03 | |
| openstackgerrit | Merged openstack-infra/groups: Enable message email notification https://review.openstack.org/219307 | 07:03 |
| *** jyuso1 has quit IRC | 07:04 | |
| *** zeih has joined #openstack-infra | 07:08 | |
| *** liam__ has quit IRC | 07:09 | |
| *** maishsk has joined #openstack-infra | 07:09 | |
| *** wenlock1 has quit IRC | 07:10 | |
| *** shashank_hegde has joined #openstack-infra | 07:13 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: remove neutron advanced services from grenade https://review.openstack.org/201279 | 07:13 |
| *** ParsectiX has joined #openstack-infra | 07:13 | |
| *** markus_z has joined #openstack-infra | 07:13 | |
| *** ddieterly has joined #openstack-infra | 07:16 | |
| *** sshnaidm has joined #openstack-infra | 07:20 | |
| *** ddieterly has quit IRC | 07:20 | |
| *** coolsvap_ has joined #openstack-infra | 07:21 | |
| *** achanda has joined #openstack-infra | 07:24 | |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Add constraints jobs to Jenkins https://review.openstack.org/207262 | 07:25 |
| *** shardy has joined #openstack-infra | 07:25 | |
| *** coolsvap_ has quit IRC | 07:26 | |
| *** ifarkas has joined #openstack-infra | 07:26 | |
| *** zeih has quit IRC | 07:28 | |
| *** vivekd has quit IRC | 07:28 | |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Add non-voting constraints jobs to nova https://review.openstack.org/219582 | 07:31 |
| yuanying | hi | 07:32 |
| yuanying | Anyone know, how to merge this https://review.openstack.org/#/c/216933/6 | 07:32 |
| yuanying | ? | 07:32 |
| *** jerryz has quit IRC | 07:33 | |
| *** armax has quit IRC | 07:34 | |
| *** BharatK has quit IRC | 07:34 | |
| *** eantyshev has joined #openstack-infra | 07:35 | |
| *** shardy_ has joined #openstack-infra | 07:36 | |
| *** jerryz has joined #openstack-infra | 07:37 | |
| *** shardy has quit IRC | 07:37 | |
| *** jyuso has quit IRC | 07:38 | |
| *** arxcruz has joined #openstack-infra | 07:40 | |
| *** fhubik has joined #openstack-infra | 07:41 | |
| *** shardy_ has quit IRC | 07:41 | |
| *** jlanoux has joined #openstack-infra | 07:41 | |
| *** shardy has joined #openstack-infra | 07:42 | |
| *** zeih_ has joined #openstack-infra | 07:42 | |
| *** dims has joined #openstack-infra | 07:44 | |
| *** jlanoux has quit IRC | 07:45 | |
| *** esker has joined #openstack-infra | 07:48 | |
| *** BharatK has joined #openstack-infra | 07:48 | |
| *** puranamr_ has joined #openstack-infra | 07:48 | |
| *** mpopow__ has joined #openstack-infra | 07:48 | |
| *** puranamr has quit IRC | 07:48 | |
| *** dims has quit IRC | 07:49 | |
| *** k4n0 has joined #openstack-infra | 07:49 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Refactor Ambassador pages layout https://review.openstack.org/219588 | 07:50 |
| *** achanda has quit IRC | 07:51 | |
| *** mpopow__ has quit IRC | 07:52 | |
| *** esker has quit IRC | 07:52 | |
| *** BobBall-AWOL is now known as BobBall | 07:54 | |
| *** yfried_ is now known as yfried|afk | 07:54 | |
| *** jyuso1 has joined #openstack-infra | 07:55 | |
| openstackgerrit | Merged openstack-infra/groups: Refactor Ambassador pages layout https://review.openstack.org/219588 | 07:56 |
| *** yfried|afk is now known as yfried_ | 07:56 | |
| *** vinayyadhav has joined #openstack-infra | 07:56 | |
| *** jlanoux has joined #openstack-infra | 07:56 | |
| *** fhubik has quit IRC | 07:57 | |
| *** larainema has quit IRC | 07:57 | |
| *** b10n1k_ has quit IRC | 07:57 | |
| *** larainema has joined #openstack-infra | 07:58 | |
| *** e0ne has joined #openstack-infra | 08:01 | |
| *** shashank_hegde has quit IRC | 08:01 | |
| *** ildikov has quit IRC | 08:08 | |
| *** hashar has joined #openstack-infra | 08:09 | |
| *** jistr has joined #openstack-infra | 08:11 | |
| *** lucas-dinner has joined #openstack-infra | 08:12 | |
| *** lucas-dinner has quit IRC | 08:12 | |
| openstackgerrit | Merged openstack/requirements: Add tosca-parser to global requirements https://review.openstack.org/217991 | 08:12 |
| *** e0ne has quit IRC | 08:13 | |
| *** mquin_ is now known as mquin | 08:14 | |
| *** ddieterly has joined #openstack-infra | 08:17 | |
| *** zeih_ has quit IRC | 08:19 | |
| *** zeih has joined #openstack-infra | 08:19 | |
| *** ddieterly has quit IRC | 08:21 | |
| *** ildikov has joined #openstack-infra | 08:21 | |
| *** hichihara has quit IRC | 08:21 | |
| *** vivekd has joined #openstack-infra | 08:24 | |
| Kennan | hi any infra cores here? | 08:24 |
| Kennan | one project hit one issue | 08:24 |
| Kennan | about projects docs publish | 08:25 |
| Kennan | we added jenkins zuul jobs about docs publish | 08:25 |
| Kennan | but we did not find docs published in openstack.org | 08:25 |
| Kennan | does infra cores know about it ? | 08:25 |
| *** pelix has joined #openstack-infra | 08:25 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 08:26 |
| *** fhubik has joined #openstack-infra | 08:27 | |
| *** earlephilhower has quit IRC | 08:28 | |
| *** cdent has joined #openstack-infra | 08:29 | |
| *** vinayyadhav has quit IRC | 08:31 | |
| *** ihrachys has joined #openstack-infra | 08:33 | |
| *** HeOS has joined #openstack-infra | 08:35 | |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Add non-voting constraints jobs to nova https://review.openstack.org/219582 | 08:36 |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Add constraints jobs to Jenkins https://review.openstack.org/207262 | 08:36 |
| *** ParsectiX has quit IRC | 08:37 | |
| *** dizquierdo has joined #openstack-infra | 08:37 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip https://review.openstack.org/211016 | 08:37 |
| *** ihrachys has quit IRC | 08:37 | |
| *** liam__ has joined #openstack-infra | 08:38 | |
| *** liam__ has quit IRC | 08:38 | |
| *** dims has joined #openstack-infra | 08:38 | |
| Kennan | lifeless: there? | 08:42 |
| Kennan | or EmilientM ? | 08:42 |
| *** marzif has joined #openstack-infra | 08:43 | |
| *** puranamr_ has quit IRC | 08:43 | |
| *** dims has quit IRC | 08:43 | |
| *** zeih has quit IRC | 08:43 | |
| *** puranamr has joined #openstack-infra | 08:45 | |
| *** zeih has joined #openstack-infra | 08:47 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack/requirements: Add kolla to requirements family https://review.openstack.org/218144 | 08:47 |
| *** derekh has joined #openstack-infra | 08:50 | |
| *** mrmartin has quit IRC | 08:50 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip https://review.openstack.org/211016 | 08:52 |
| *** markvoelker has joined #openstack-infra | 08:57 | |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Add non-voting constraints jobs to neutron https://review.openstack.org/219610 | 08:58 |
| *** amotoki_ has quit IRC | 08:58 | |
| *** peristeri has quit IRC | 08:58 | |
| *** markvoelker has quit IRC | 09:02 | |
| openstackgerrit | Ye-hong Fang proposed openstack/diskimage-builder: Fix grub-efi-amd64-signed install failure https://review.openstack.org/219612 | 09:05 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip https://review.openstack.org/211016 | 09:05 |
| lifeless | Kennan: just ask the question... | 09:06 |
| *** rossella_s has quit IRC | 09:06 | |
| *** rossella_s has joined #openstack-infra | 09:06 | |
| *** john-davidge has joined #openstack-infra | 09:07 | |
| *** salv-orlando has quit IRC | 09:07 | |
| *** bexelbie_ has joined #openstack-infra | 09:08 | |
| *** mrmartin has joined #openstack-infra | 09:09 | |
| *** jerryz has quit IRC | 09:10 | |
| *** john-davidge_ has joined #openstack-infra | 09:10 | |
| *** john-davidge has quit IRC | 09:12 | |
| *** john-davidge_ is now known as john-davidge | 09:12 | |
| *** ParsectiX has joined #openstack-infra | 09:15 | |
| *** kaisers1 has joined #openstack-infra | 09:16 | |
| *** sanjayu has joined #openstack-infra | 09:17 | |
| *** ddieterly has joined #openstack-infra | 09:18 | |
| *** flepied1 has joined #openstack-infra | 09:19 | |
| *** amotoki has joined #openstack-infra | 09:19 | |
| *** kiran-r has joined #openstack-infra | 09:20 | |
| *** ParsectiX has quit IRC | 09:20 | |
| *** sharikapongubala has quit IRC | 09:21 | |
| *** peristeri has joined #openstack-infra | 09:21 | |
| *** flepied has quit IRC | 09:21 | |
| *** sanjayu_ has quit IRC | 09:21 | |
| *** marzif has quit IRC | 09:22 | |
| *** ddieterly has quit IRC | 09:22 | |
| *** peristeri has quit IRC | 09:23 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/diskimage-builder: Map ubuntu versions to match the SHA256SUMs https://review.openstack.org/219199 | 09:23 |
| Kennan | hi lifeless: | 09:24 |
| Kennan | I asked about docs publish questions | 09:24 |
| Kennan | could you help that ? | 09:24 |
| *** peristeri has joined #openstack-infra | 09:24 | |
| Kennan | lifeless: we added openstack-server-publish-jobs | 09:25 |
| Kennan | but seems the docs.openstack.org not publish our dcos | 09:25 |
| Kennan | do you know what's the issue ? lifeless: | 09:25 |
| *** dizquierdo has quit IRC | 09:26 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** puranamr has quit IRC | 09:28 | |
| Kennan | lifeless: | 09:28 |
| Kennan | here is the link https://review.openstack.org/#/c/218137/ | 09:28 |
| *** kushal has quit IRC | 09:30 | |
| *** amotoki has quit IRC | 09:30 | |
| *** e0ne has joined #openstack-infra | 09:32 | |
| EmilienM | good morning infra, I have an outstanding review for Puppet : https://review.openstack.org/#/c/219556/ | 09:32 |
| *** puranamr has joined #openstack-infra | 09:32 | |
| *** dims has joined #openstack-infra | 09:32 | |
| *** esker has joined #openstack-infra | 09:36 | |
| *** dims has quit IRC | 09:37 | |
| *** stevemar has joined #openstack-infra | 09:38 | |
| *** jhesketh has quit IRC | 09:39 | |
| *** andrew has joined #openstack-infra | 09:40 | |
| *** ociuhandu has quit IRC | 09:41 | |
| *** esker has quit IRC | 09:41 | |
| *** jhesketh has joined #openstack-infra | 09:41 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Decouple openstack_project::server from puppetdb class https://review.openstack.org/172871 | 09:42 |
| *** glauco_ has joined #openstack-infra | 09:42 | |
| *** stevemar has quit IRC | 09:42 | |
| *** andrew has quit IRC | 09:44 | |
| *** bigtaro has joined #openstack-infra | 09:46 | |
| *** e0ne has quit IRC | 09:46 | |
| *** glauco_ has quit IRC | 09:46 | |
| *** zeih has quit IRC | 09:50 | |
| *** salv-orlando has joined #openstack-infra | 09:50 | |
| *** yamamoto has joined #openstack-infra | 09:53 | |
| *** e0ne has joined #openstack-infra | 09:55 | |
| *** bigtaro has quit IRC | 09:59 | |
| *** hashar_ has joined #openstack-infra | 09:59 | |
| *** bigtaro has joined #openstack-infra | 10:00 | |
| *** fhubik is now known as fhubik_brb | 10:00 | |
| *** fhubik_brb is now known as fhubik | 10:00 | |
| *** fhubik is now known as fhubik_brb | 10:00 | |
| *** regXboi has joined #openstack-infra | 10:01 | |
| *** hashar has quit IRC | 10:02 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** yamamoto has joined #openstack-infra | 10:04 | |
| *** yamamoto has quit IRC | 10:04 | |
| *** yamamoto has joined #openstack-infra | 10:04 | |
| *** ociuhandu has joined #openstack-infra | 10:05 | |
| *** marzif has joined #openstack-infra | 10:06 | |
| *** yamamoto has quit IRC | 10:06 | |
| electrical | lifeless: finally implementing JJB further at Elastic :-) have to teach co-workers on how to use it and how the structure works :-) | 10:06 |
| lifeless | electrical: cool,gl | 10:07 |
| openstackgerrit | Merged openstack-infra/project-config: Add stable/liberty crosscheck jobs for reqs https://review.openstack.org/217300 | 10:07 |
| *** yamamoto has joined #openstack-infra | 10:12 | |
| *** ParsectiX has joined #openstack-infra | 10:13 | |
| *** esker has joined #openstack-infra | 10:14 | |
| *** bigtaro has quit IRC | 10:16 | |
| *** amotoki has joined #openstack-infra | 10:16 | |
| *** ddieterly has joined #openstack-infra | 10:18 | |
| *** esker has quit IRC | 10:18 | |
| *** kiran-r has quit IRC | 10:19 | |
| *** puranamr has quit IRC | 10:19 | |
| *** yamamoto has quit IRC | 10:20 | |
| *** sanjayu_ has joined #openstack-infra | 10:20 | |
| *** ihrachys has joined #openstack-infra | 10:21 | |
| *** puranamr has joined #openstack-infra | 10:21 | |
| *** ihrachys has quit IRC | 10:22 | |
| *** ddieterly has quit IRC | 10:23 | |
| *** hashar has joined #openstack-infra | 10:23 | |
| *** sanjayu has quit IRC | 10:24 | |
| *** hashar_ has quit IRC | 10:25 | |
| *** dims has joined #openstack-infra | 10:26 | |
| *** dims has quit IRC | 10:31 | |
| *** pc_m has joined #openstack-infra | 10:31 | |
| *** wenlock has quit IRC | 10:33 | |
| *** wenlock has joined #openstack-infra | 10:33 | |
| *** gsagie has joined #openstack-infra | 10:34 | |
| gsagie | Hello anyone, any infrastructure core can review the following patch: https://review.openstack.org/#/c/218575/ will appreciate it alot :) | 10:35 |
| gsagie | everyone :) not anyone | 10:35 |
| openstackgerrit | pangwa proposed openstack-infra/git-review: fix encoding issue on Windows https://review.openstack.org/218167 | 10:38 |
| *** chmouel has quit IRC | 10:40 | |
| *** akuznetsova has quit IRC | 10:41 | |
| *** chmouel has joined #openstack-infra | 10:41 | |
| *** akuznetsova has joined #openstack-infra | 10:41 | |
| *** yfried_ is now known as yfried|afk | 10:45 | |
| *** fhubik_brb is now known as fhubik | 10:46 | |
| *** ZZelle has quit IRC | 10:48 | |
| *** ZZelle has joined #openstack-infra | 10:48 | |
| *** daemontool_ has joined #openstack-infra | 10:50 | |
| pabelanger | morning | 10:50 |
| *** esker has joined #openstack-infra | 10:50 | |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| *** larainema has quit IRC | 10:54 | |
| *** larainema has joined #openstack-infra | 10:54 | |
| *** fhubik is now known as fhubik_brb | 10:56 | |
| *** fhubik_brb is now known as fhubik | 10:56 | |
| *** claudiub has joined #openstack-infra | 10:57 | |
| *** amotoki has quit IRC | 10:57 | |
| *** trown|outttypeww is now known as trown | 10:58 | |
| openstackgerrit | Gary Duan proposed openstack/diskimage-builder: Contains the directory name of /sys and /proc https://review.openstack.org/215060 | 10:59 |
| *** yfried has joined #openstack-infra | 11:01 | |
| *** yfried|afk has quit IRC | 11:01 | |
| *** glauco has joined #openstack-infra | 11:02 | |
| *** purp_2 has joined #openstack-infra | 11:02 | |
| *** purp has quit IRC | 11:02 | |
| *** jgriffith has quit IRC | 11:02 | |
| *** eharney has quit IRC | 11:02 | |
| *** yfried_ has joined #openstack-infra | 11:02 | |
| *** yfried_ is now known as yfried|afk | 11:02 | |
| *** jgriffith has joined #openstack-infra | 11:03 | |
| *** eharney has joined #openstack-infra | 11:04 | |
| *** jgriffith is now known as Guest91834 | 11:04 | |
| *** amotoki has joined #openstack-infra | 11:04 | |
| *** yfried has quit IRC | 11:06 | |
| *** glauco has quit IRC | 11:06 | |
| *** hashar_ has joined #openstack-infra | 11:08 | |
| *** kushal has joined #openstack-infra | 11:09 | |
| *** puranamr has quit IRC | 11:10 | |
| *** amotoki has quit IRC | 11:10 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 11:10 |
| *** hashar has quit IRC | 11:11 | |
| *** puranamr has joined #openstack-infra | 11:13 | |
| *** baoli_ has joined #openstack-infra | 11:13 | |
| *** kushal has quit IRC | 11:14 | |
| *** mwagner_lap has joined #openstack-infra | 11:15 | |
| *** mmedvede has joined #openstack-infra | 11:17 | |
| *** yfried|afk has quit IRC | 11:18 | |
| *** ddieterly has joined #openstack-infra | 11:19 | |
| *** fhubik has quit IRC | 11:20 | |
| *** fhubik has joined #openstack-infra | 11:20 | |
| *** dizquierdo has joined #openstack-infra | 11:20 | |
| *** dims has joined #openstack-infra | 11:21 | |
| *** wenlock has quit IRC | 11:21 | |
| *** kiran-r has joined #openstack-infra | 11:21 | |
| *** wenlock has joined #openstack-infra | 11:22 | |
| svasheka | hi guys could you please take a look at https://review.openstack.org/#/c/219254/1, it fails, but there's no appropriate message in logs | 11:23 |
| svasheka | is value too long? | 11:23 |
| *** kushal has joined #openstack-infra | 11:23 | |
| *** ddieterly has quit IRC | 11:24 | |
| *** esker has quit IRC | 11:26 | |
| *** dims has quit IRC | 11:26 | |
| *** esker has joined #openstack-infra | 11:26 | |
| *** yfried|afk has joined #openstack-infra | 11:27 | |
| *** amotoki has joined #openstack-infra | 11:28 | |
| *** zeih has joined #openstack-infra | 11:29 | |
| *** baoli has joined #openstack-infra | 11:31 | |
| *** yfried|afk has quit IRC | 11:31 | |
| *** yfried has joined #openstack-infra | 11:31 | |
| *** yfried is now known as yfried|afk | 11:31 | |
| *** baoli_ has quit IRC | 11:33 | |
| *** wenlock has quit IRC | 11:34 | |
| *** bharathm has quit IRC | 11:36 | |
| *** gordc has joined #openstack-infra | 11:38 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 11:42 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix exception lists in functional tests https://review.openstack.org/219655 | 11:42 |
| *** bapalm has quit IRC | 11:43 | |
| *** vinay_yadhav has joined #openstack-infra | 11:44 | |
| *** ldnunes has joined #openstack-infra | 11:46 | |
| *** bapalm has joined #openstack-infra | 11:49 | |
| *** esker has quit IRC | 11:52 | |
| *** thorst has joined #openstack-infra | 11:54 | |
| *** rfolco has joined #openstack-infra | 11:54 | |
| *** nadya has joined #openstack-infra | 11:55 | |
| *** abregman is now known as abregman|brb | 11:55 | |
| *** puranamr has quit IRC | 11:57 | |
| anteaya | morning EmilienM | 11:58 |
| openstackgerrit | Julien Danjou proposed openstack/requirements: tooz needs psycopg2>=2.5 https://review.openstack.org/219663 | 11:59 |
| *** puranamr has joined #openstack-infra | 12:00 | |
| *** nadya has quit IRC | 12:01 | |
| *** mmedvede has quit IRC | 12:01 | |
| *** zeih has quit IRC | 12:02 | |
| *** amuller has joined #openstack-infra | 12:02 | |
| anteaya | morning pabelanger | 12:03 |
| anteaya | pabelanger: when do you expect to be in Montreal? | 12:03 |
| *** abregman|brb is now known as abregman | 12:03 | |
| *** mmedvede has joined #openstack-infra | 12:05 | |
| *** stevemar has joined #openstack-infra | 12:05 | |
| *** jaypipes has joined #openstack-infra | 12:08 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 12:08 | |
| *** stevemar has quit IRC | 12:09 | |
| *** yfried|afk is now known as yfried | 12:10 | |
| *** mwagner_lap has quit IRC | 12:10 | |
| *** mwagner_lap has joined #openstack-infra | 12:10 | |
| *** jcoufal has joined #openstack-infra | 12:11 | |
| *** bswartz has joined #openstack-infra | 12:12 | |
| *** spzala has joined #openstack-infra | 12:12 | |
| *** dprince has joined #openstack-infra | 12:13 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add constraints jobs to Jenkins https://review.openstack.org/207262 | 12:14 |
| *** vinay_yadhav has quit IRC | 12:15 | |
| *** vinay_yadhav has joined #openstack-infra | 12:16 | |
| *** Ala has quit IRC | 12:16 | |
| *** ajmiller has quit IRC | 12:16 | |
| *** aysyd has joined #openstack-infra | 12:18 | |
| *** Ala has joined #openstack-infra | 12:18 | |
| fungi | Kennan: your docs job isn't working because your tox.ini is incomplete. here's the failing job log for your most recent merge http://logs.openstack.org/34/34937175172e44ed67d271f74540b7f5e0de7c6d/post/kolla-docs/7a9e70c/console.html#_2015-09-02_10_42_45_022 you need to create a section for an env named "venv" in your tox.ini file. see other openstack projects for examples | 12:19 |
| *** ddieterly has joined #openstack-infra | 12:20 | |
| *** ddieterly has quit IRC | 12:21 | |
| *** ddieterly has joined #openstack-infra | 12:22 | |
| *** jcoufal has quit IRC | 12:22 | |
| *** madong_ has joined #openstack-infra | 12:23 | |
| *** larainema has quit IRC | 12:23 | |
| *** flepied1 has quit IRC | 12:24 | |
| fungi | svasheka: that "fatal: remote did not send all necessary objects" failure on our layout jobs is because we're having trouble with at least some shallow clones from our git servers (which that job does). recheck for now, we have a probable solution to try | 12:25 |
| *** hashar_ has quit IRC | 12:25 | |
| *** ivar-lazzaro has joined #openstack-infra | 12:26 | |
| svasheka | fungi, thanks | 12:26 |
| *** mcdonagm has joined #openstack-infra | 12:26 | |
| *** purp_2 is now known as purp | 12:27 | |
| *** david-lyle has quit IRC | 12:27 | |
| *** mordred has quit IRC | 12:27 | |
| *** mfmcdonagh has quit IRC | 12:28 | |
| *** dansmith has quit IRC | 12:28 | |
| *** Swanson has quit IRC | 12:28 | |
| *** timrc has quit IRC | 12:28 | |
| *** haleyb has quit IRC | 12:28 | |
| rakhmerov | fungi: not sure if it's your area but I need to land the patch https://review.openstack.org/#/c/194999/ into openstack-requirements | 12:28 |
| rakhmerov | it's been on review for a while | 12:28 |
| *** david-lyle has joined #openstack-infra | 12:28 | |
| rakhmerov | who would be the best person to talk to about it? | 12:29 |
| anteaya | rakhmerov: requirements core reviewers: https://review.openstack.org/#/admin/groups/131,members | 12:30 |
| fungi | rakhmerov: the regular requirements-core members are the ones who should be reviewing those https://review.openstack.org/#/admin/groups/131,members (release managers and infra-core are only included groups because sometimes we need to un-break things project-wide by quickly approving requirements changes) | 12:30 |
| *** hashar has joined #openstack-infra | 12:31 | |
| *** sharikapongubala has joined #openstack-infra | 12:31 | |
| rakhmerov | ok, thnx | 12:32 |
| *** dansmith has joined #openstack-infra | 12:33 | |
| *** timrc has joined #openstack-infra | 12:33 | |
| *** ivar-lazzaro has quit IRC | 12:33 | |
| *** dansmith is now known as Guest91339 | 12:33 | |
| *** dkranz has joined #openstack-infra | 12:36 | |
| *** bharathm has joined #openstack-infra | 12:37 | |
| *** ddieterly has quit IRC | 12:37 | |
| *** nicodemos has joined #openstack-infra | 12:38 | |
| *** glauco has joined #openstack-infra | 12:38 | |
| *** claudiub has quit IRC | 12:38 | |
| *** vinay_yadhav has quit IRC | 12:38 | |
| *** cburroughs has joined #openstack-infra | 12:38 | |
| *** gsagie has left #openstack-infra | 12:39 | |
| *** yfried is now known as yfried|afk | 12:40 | |
| *** _kiran_ has joined #openstack-infra | 12:41 | |
| *** Swanson has joined #openstack-infra | 12:41 | |
| *** haleyb has joined #openstack-infra | 12:41 | |
| *** bharathm has quit IRC | 12:41 | |
| *** glauco has quit IRC | 12:42 | |
| *** kiran-r has quit IRC | 12:42 | |
| *** _kiran_ is now known as kiran-r | 12:43 | |
| *** yfried|afk is now known as yfried | 12:44 | |
| anteaya | fungi: can you change the topic in openstack-sprint to reflect the current puppet sprint? | 12:44 |
| anteaya | please | 12:44 |
| anteaya | EmilienM: ^ | 12:44 |
| *** mpaolino_ has joined #openstack-infra | 12:44 | |
| fungi | anteaya: sure, what should the new topic be? | 12:44 |
| *** ivar-lazzaro has joined #openstack-infra | 12:45 | |
| *** mpaolino has quit IRC | 12:45 | |
| fungi | EmilienM: ^ ? | 12:45 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 12:45 | |
| *** dims has joined #openstack-infra | 12:48 | |
| *** bexelbie_ is now known as bexelbie | 12:49 | |
| anteaya | I'll let EmilienM say but probably something that includes a link to the etherpad: https://etherpad.openstack.org/p/puppet-liberty-mid-cycle | 12:50 |
| anteaya | fungi: he is probably still on the train and should be online again shortly | 12:50 |
| anteaya | thanks | 12:50 |
| * anteaya changes locations | 12:51 | |
| *** jyuso1 has quit IRC | 12:51 | |
| fungi | i'll put in a temporary guess, thanks | 12:51 |
| *** alivigni has joined #openstack-infra | 12:51 | |
| *** annegentle has joined #openstack-infra | 12:51 | |
| fungi | can always adjust it again later if needed | 12:52 |
| *** e0ne has quit IRC | 12:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove last vestige of glanceclient being different https://review.openstack.org/219291 | 12:54 |
| *** sanjayu has joined #openstack-infra | 12:55 | |
| *** mpaolino_ has quit IRC | 12:56 | |
| *** bswartz has quit IRC | 12:58 | |
| sdague | fungi: so today's level of fire .... | 12:58 |
| *** sanjayu_ has quit IRC | 12:58 | |
| sdague | the neutron grenade job is 100% failing because of a devstack-gate change, as how neutron extension tests work was not well understood. There is a tempest fix, but it's bouncing off of other neutron races. | 12:59 |
| *** mase_x200 has joined #openstack-infra | 12:59 | |
| sdague | also heat is 100% fail on their functional job | 12:59 |
| fungi | i need to get a fireman's helmet | 13:00 |
| fungi | and send it to you | 13:00 |
| *** mpaolino has joined #openstack-infra | 13:00 | |
| *** ildikov has quit IRC | 13:00 | |
| fungi | how else can i help? still tracking on the same etherpad, or a new one? | 13:01 |
| *** kushal has quit IRC | 13:01 | |
| *** mwagner_lap has quit IRC | 13:01 | |
| *** bcourt has quit IRC | 13:01 | |
| fungi | if nothing else i can try to keep an eye on those patches and try to make sure they get through | 13:01 |
| *** mpaolino has quit IRC | 13:01 | |
| *** jcoufal has joined #openstack-infra | 13:02 | |
| sdague | so, honestly, https://review.openstack.org/#/c/219632/ is so straight forward, it should just merge | 13:02 |
| sdague | my patience on playing recheck games on that is low | 13:02 |
| *** kiran-r has quit IRC | 13:02 | |
| *** mpaolino has joined #openstack-infra | 13:02 | |
| *** sanjayu_ has joined #openstack-infra | 13:04 | |
| *** Ala has quit IRC | 13:04 | |
| *** vivekd has quit IRC | 13:04 | |
| *** rossella_s has quit IRC | 13:05 | |
| *** rossella_s has joined #openstack-infra | 13:06 | |
| *** ddieterly has joined #openstack-infra | 13:06 | |
| mescanef | hello. please review: https://review.openstack.org/217589 https://review.openstack.org/218154 and https://review.openstack.org/218769 :-) | 13:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Add support for listing folder indexes https://review.openstack.org/214207 | 13:06 |
| *** vivekd has joined #openstack-infra | 13:06 | |
| *** sanjayu has quit IRC | 13:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface https://review.openstack.org/214243 | 13:09 |
| *** Ala has joined #openstack-infra | 13:10 | |
| sdague | fungi: I'm also writing some scripting to build graphite fail graphs faster than by hand - https://github.com/sdague/failopotamus | 13:11 |
| *** flepied has joined #openstack-infra | 13:11 | |
| *** mordred has joined #openstack-infra | 13:11 | |
| fungi | mad props for the name alone | 13:12 |
| *** rlandy has joined #openstack-infra | 13:12 | |
| *** ildikov has joined #openstack-infra | 13:12 | |
| *** zeih has joined #openstack-infra | 13:14 | |
| fungi | oh, wow. now that i'm at a computer that can render the zuul status page, i see what you mean | 13:16 |
| *** tiswanso has joined #openstack-infra | 13:16 | |
| *** bpoulos has joined #openstack-infra | 13:16 | |
| *** mpaolino has quit IRC | 13:16 | |
| *** bcourt has joined #openstack-infra | 13:17 | |
| *** links has quit IRC | 13:17 | |
| *** dims_ has joined #openstack-infra | 13:17 | |
| *** esker has joined #openstack-infra | 13:18 | |
| sdague | yeh, it's kind of been like this for a while | 13:19 |
| fungi | well, it had caught up well last night | 13:19 |
| sdague | the heat thing is just going to be a wrecking ball | 13:19 |
| sdague | anyway, you can failgraph.py gate-heat-dsvm-functional-orig-mysql | 13:19 |
| fungi | do we need to just stop running the heat functional tests in the gate? any details on how complicated the issue is there? | 13:20 |
| *** esker has quit IRC | 13:20 | |
| sdague | and it opens up a graphite view comparing both check and gate queues | 13:20 |
| *** dims has quit IRC | 13:21 | |
| sdague | honestly, I don't know, and I'm about to go for a bike ride, because having to dig into every project's tests to figure out why they've gone into left field is frustrating | 13:21 |
| sdague | honestly, I've continued to believe that you should only be able to run tests in the gate which affect multiple projects | 13:21 |
| *** kushal has joined #openstack-infra | 13:21 | |
| fungi | yep, go do that. i need to catch up on the ml to make sure i know what the current status is on people solving these | 13:21 |
| sdague | and that all the single project tests should never be there | 13:21 |
| fungi | before i start hounding them in irc | 13:22 |
| sdague | but I think I'm of minority opinion on that one | 13:22 |
| *** ivar-lazzaro has quit IRC | 13:22 | |
| *** vinay_yadhav has joined #openstack-infra | 13:23 | |
| fungi | it would likely result in projects continually breaking their own check jobs, though not convinced forcing them to deal with that afterwards is necessarily bad | 13:24 |
| *** vivekd_ has joined #openstack-infra | 13:25 | |
| *** mriedem_away is now known as mriedem | 13:25 | |
| *** mwagner_lap has joined #openstack-infra | 13:25 | |
| *** vivekd has quit IRC | 13:25 | |
| *** esker has joined #openstack-infra | 13:25 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Add support for listing folder indexes https://review.openstack.org/214207 | 13:25 |
| *** vivekd_ is now known as vivekd | 13:26 | |
| *** esker has quit IRC | 13:26 | |
| sdague | fungi: right, then it would be project isolated | 13:27 |
| *** ajmiller has joined #openstack-infra | 13:27 | |
| *** lucasagomes has joined #openstack-infra | 13:27 | |
| *** esker has joined #openstack-infra | 13:27 | |
| fungi | yep | 13:27 |
| *** ajmiller has quit IRC | 13:27 | |
| *** ajmiller has joined #openstack-infra | 13:28 | |
| shardy | sdague: Hey I think the heat gate is broken due to bug #1491199 | 13:28 |
| openstack | bug 1491199 in heat "glance: error: unrecognized arguments: --is-public True --location..." [Critical,In progress] https://launchpad.net/bugs/1491199 - Assigned to Angus Salkeld (asalkeld) | 13:28 |
| shardy | sdague: there's a fix posted but not yet merged | 13:28 |
| shardy | tl;dr glanceclient released and broke us | 13:28 |
| sdague | shardy: so, when you identify something like that, it would also be good to message to your core team to stop approving patches until that's fixed | 13:29 |
| *** zul has quit IRC | 13:29 | |
| fungi | shardy: thanks, i was digging into the -dev ml looking for signs of some status like that. i can push that change to the front of the gate now to keep all the other heat changes currently in the queue from making a mess | 13:29 |
| shardy | sdague: Ok, fair point, sorry I think I'm guilty of approving some things, I thought that was near landing earlier thus was at the head of the queue | 13:30 |
| shardy | apologies | 13:30 |
| shardy | fungi: it's a critical bug on the heat tracker FWIW, reported 12h ago | 13:30 |
| *** ajmiller_ has joined #openstack-infra | 13:30 | |
| *** glauco has joined #openstack-infra | 13:31 | |
| *** ajmiller has quit IRC | 13:31 | |
| *** kgiusti has joined #openstack-infra | 13:31 | |
| fungi | yeah, 6 other heat changes ahead of it which are all going to have to fail before it gets to the front unless i reprioritize it | 13:31 |
| *** zul has joined #openstack-infra | 13:32 | |
| shardy | fungi: Ok, if you can bump https://review.openstack.org/#/c/219533/ to the top of the queue that'd be great, thanks! | 13:33 |
| sdague | fungi: also, promote - https://review.openstack.org/#/c/219632 | 13:33 |
| sdague | it passed check | 13:33 |
| sdague | probably put the heat change in front of it | 13:33 |
| shardy | Out of interest, how hard would it be to give core approves a comment tag which bumps gate-fix patches to the top? | 13:33 |
| shardy | gatefix bug ### | 13:33 |
| fungi | sdague: yep, had them both queued up in the cli. i'd already pushed that one into the gate anyway in case it didn't pass check sooner | 13:33 |
| sdague | shardy: hard, plus it would be massively abused | 13:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions https://review.openstack.org/183176 | 13:34 |
| sdague | if it was going to be done it should be keyed off of something like the elastic recheck bug list | 13:34 |
| fungi | right now we only reprioritize changes in the gate based on conversations between infra and qa | 13:34 |
| shardy | sdague: that's sad, I think in our community it'd be effectively self-policed and all core reviewers could be trusted to use it sanely | 13:34 |
| shardy | just trying to think of ways to spread the load wrt fixing these things | 13:34 |
| sdague | shardy: really? because I see core team members blindly rechecking patches all the time which is part of what gets us into these high failure rates | 13:35 |
| fungi | shardy: if projects were quicker to revoke core reviewership from people who were sloppy, careless, didn't attend meetings, et cetera then maybe | 13:35 |
| *** ajmiller_ has quit IRC | 13:35 | |
| fungi | and if the tc were quicker to punt project-teams out of the tent for not policing themselves, perhaps | 13:35 |
| *** amotoki has quit IRC | 13:36 | |
| sdague | we've talked about keying off of the elastic-recheck list, as those are bugs that we know are impacting a number of jobs. However, it's still quite a bit off effort to build that kind of priorty reorganization system. | 13:36 |
| shardy | sdague: Ok, cool, I was just reflecting on how this might have been solved sooner, given that the patch was approved 10 hours ago | 13:37 |
| shardy | next time I'll try to reach out to you folks sooner, thanks for the help! :) | 13:38 |
| *** amitgandhinz has joined #openstack-infra | 13:38 | |
| sdague | shardy: yeh, well that assumes all teams are keeping their test failure rates low | 13:38 |
| *** kdas_ has joined #openstack-infra | 13:38 | |
| sdague | which... as you can see, isn't the case, which is why it's still in the queue :) | 13:38 |
| shardy | sdague: well, in this case, we were, but then we got broken by a client release which we couldn't have predicted | 13:39 |
| sdague | shardy: right, I mean all teams | 13:39 |
| shardy | unfortunately that does happen semi-regularly because we use all the clients :( | 13:39 |
| *** tlian has joined #openstack-infra | 13:39 | |
| *** fhubik has quit IRC | 13:39 | |
| sdague | so, you could have sped things up by going and plucking all the heat changes above your fix out of the gate | 13:39 |
| *** fhubik has joined #openstack-infra | 13:40 | |
| shardy | sdague: Do we do that just by -2ing them? | 13:40 |
| sdague | that was a huge reason this was delayed | 13:40 |
| sdague | no, you have to update the commit | 13:40 |
| fungi | easiest way is to adjust their commit message or rebase them | 13:40 |
| fungi | anything that creates a new patchset | 13:40 |
| *** edmondsw has joined #openstack-infra | 13:41 | |
| *** bcourt has quit IRC | 13:41 | |
| fungi | because then they're immediately ejected from the gate pipeline and have to go back through check again | 13:41 |
| sdague | or, come into infra and ask. Because if you know you've got patches in the gate that can't pass, it's the team's responsibility to raise that | 13:41 |
| shardy | Ok, that just seems like a much harder way to do what I imagined re the gatefix thing, but I'll remember that next time, thanks! | 13:41 |
| *** kushal has quit IRC | 13:41 | |
| EmilienM | fungi, anteaya current topic for #openstack-sprint is perfect - thx | 13:42 |
| fungi | most of the time it's not really needed, but here at the milestone 3 rush when everyone has waited until the last minute to merge their random (and often conflicting) feature implementations, things get really bad really fast | 13:42 |
| mordred | sdague: did something that you know of change in nova? we've got a functional test failing on deleting a floating ip - and it says "invalid id" to the IP we passed - do we need to interact with floating ips by ID now? | 13:42 |
| *** madong_ is now known as larainema | 13:43 | |
| sdague | mordred: not that I know of, but I'm actually going for that bike ride now. If you write up the issue I'll look this afternoon. | 13:43 |
| mordred | sdague: ok. enjoy the bike | 13:43 |
| mordred | sdague: and I can figure it out - was just asking if you happened to know | 13:43 |
| *** ajmiller has joined #openstack-infra | 13:43 | |
| *** annegentle has quit IRC | 13:43 | |
| *** glauco has quit IRC | 13:44 | |
| shardy | fungi: Ok, thanks, I think I/we just misunderstood the expense of waiting for the fix to merge in this case | 13:45 |
| shardy | fungi, sdague: thanks for the assistance/patience! :) | 13:45 |
| *** dbite has quit IRC | 13:45 | |
| *** puranamr has quit IRC | 13:46 | |
| *** ajmiller has quit IRC | 13:46 | |
| *** glauco has joined #openstack-infra | 13:46 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 13:47 |
| *** prad- has joined #openstack-infra | 13:48 | |
| *** puranamr has joined #openstack-infra | 13:49 | |
| *** dguitarbite has joined #openstack-infra | 13:49 | |
| *** BharatK has quit IRC | 13:49 | |
| *** Guest91339 is now known as dansmith | 13:51 | |
| *** dansmith is now known as Guest36431 | 13:51 | |
| *** spzala has quit IRC | 13:51 | |
| *** spzala_ has joined #openstack-infra | 13:51 | |
| *** spzala_ is now known as spzala | 13:52 | |
| *** spzala has quit IRC | 13:54 | |
| *** bcourt has joined #openstack-infra | 13:54 | |
| *** kdas_ is now known as kushal | 13:54 | |
| *** kushal is now known as Guest56188 | 13:55 | |
| *** Guest36431 is now known as dansmith | 13:56 | |
| *** mriedem is now known as mriedem_meeting | 13:59 | |
| *** mrmartin has quit IRC | 14:00 | |
| *** vinay_yadhav has quit IRC | 14:00 | |
| *** vinay_yadhav has joined #openstack-infra | 14:01 | |
| *** markvoelker has joined #openstack-infra | 14:01 | |
| *** bswartz has joined #openstack-infra | 14:01 | |
| *** Guest56188 is now known as kushal | 14:03 | |
| *** sridhar_ram has joined #openstack-infra | 14:03 | |
| *** kushal has quit IRC | 14:03 | |
| *** kushal has joined #openstack-infra | 14:03 | |
| *** burgerk has joined #openstack-infra | 14:04 | |
| *** julim has joined #openstack-infra | 14:04 | |
| *** BharatK has joined #openstack-infra | 14:04 | |
| *** marzif has quit IRC | 14:05 | |
| *** bharathm has joined #openstack-infra | 14:06 | |
| *** markvoelker has quit IRC | 14:06 | |
| bswartz | does anyone know how to change the channel topic in IRC for channels managed by the bot? | 14:06 |
| *** marzif has joined #openstack-infra | 14:06 | |
| *** zz_dimtruck is now known as dimtruck | 14:06 | |
| *** xnox has quit IRC | 14:07 | |
| *** mrmartin has joined #openstack-infra | 14:09 | |
| *** kiran-r has joined #openstack-infra | 14:09 | |
| *** _kiran_ has joined #openstack-infra | 14:09 | |
| *** vivekd_ has joined #openstack-infra | 14:09 | |
| *** xnox has joined #openstack-infra | 14:09 | |
| *** zeih has quit IRC | 14:09 | |
| fungi | bswartz: what channel? | 14:10 |
| bswartz | fungi: #openstack-manila | 14:10 |
| *** bharathm has quit IRC | 14:10 | |
| *** _kiran_ has quit IRC | 14:10 | |
| *** mugsie has quit IRC | 14:10 | |
| *** _kiran_ has joined #openstack-infra | 14:11 | |
| bswartz | fungi: I'm hoping that I have the power to change the topic since I'm the PTL | 14:11 |
| fungi | bswartz: anyone with +t for that channel can just "/topic some topic" or channel administrators can "/msg chanserv topic #openstack-manila some topic" | 14:11 |
| *** ParsectiX has quit IRC | 14:11 | |
| * fungi checs the channel permissions | 14:11 | |
| *** mugsie has joined #openstack-infra | 14:11 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 14:12 |
| bswartz | fungi: I'm not authorized | 14:12 |
| *** vivekd has quit IRC | 14:12 | |
| *** vivekd_ is now known as vivekd | 14:12 | |
| lucasagomes | clarkb, if you have some time, mind taking a look at https://review.openstack.org/#/c/219351/3/jenkins/jobs/devstack-gate.yaml ? | 14:12 |
| fungi | bswartz: yeah, it looks like that channel never got any project-team founders | 14:12 |
| bswartz | fungi: how can we fix that? | 14:12 |
| fungi | bswartz: anyway, just a sec and i'll add you | 14:12 |
| bswartz | fungi: ty | 14:13 |
| lucasagomes | clarkb, me and the rest are a bit confused about how things works there | 14:13 |
| *** Piet has quit IRC | 14:13 | |
| *** kiran-r has quit IRC | 14:13 | |
| fungi | bswartz: try again | 14:14 |
| mordred | fungi, clarkb: do I remember some issue with devstack as part of the insanity with floating ips? | 14:14 |
| bswartz | fungi: works, ty | 14:14 |
| clarkb | mordred no just that you have to open security groups | 14:14 |
| *** annegentle has joined #openstack-infra | 14:14 | |
| fungi | mordred: no recollection | 14:15 |
| mordred | ok | 14:15 |
| mordred | clarkb: so nothing about not being able to delete a floating ip by ip address | 14:15 |
| *** zeih has joined #openstack-infra | 14:15 | |
| clarkb | no | 14:16 |
| fungi | mordred: fwiw i've only ever manipulated floating ips via id, not address | 14:16 |
| clarkb | that worked as of the last time I did multinode things | 14:16 |
| fungi | didn't know you could use the address too | 14:16 |
| *** zeih has quit IRC | 14:16 | |
| mordred | heh. well, maybe I should just rework the code to always use id | 14:16 |
| *** zeih has joined #openstack-infra | 14:17 | |
| *** _kiran_ has quit IRC | 14:18 | |
| *** kiran-r has joined #openstack-infra | 14:18 | |
| clarkb | lucasagomes: I think I understand the question there and yes you need to do a bit extra | 14:18 |
| clarkb | lucasagomes: that change has defined a jenkins job template but never instantiates that template with any concrete parameters, this means the job cannot be created in jenkins | 14:18 |
| clarkb | lucasagomes: so you need to instantiate the job that usually happens in projects.yaml under a projects jobs: list | 14:19 |
| clarkb | lucasagomes: then you haveto tell zuul when you wantto run that job in zuul/layout.yaml | 14:19 |
| *** kdas_ has joined #openstack-infra | 14:19 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source https://review.openstack.org/219351 | 14:19 |
| *** kdas_ has quit IRC | 14:21 | |
| lucasagomes | clarkb, gotcha! updated the patch | 14:21 |
| lucasagomes | thanks | 14:21 |
| *** kushal has quit IRC | 14:22 | |
| *** zeih has quit IRC | 14:22 | |
| *** thiagop has joined #openstack-infra | 14:22 | |
| clarkb | fungi: we still need to fix haproxy for the layout job ya? | 14:23 |
| *** wenlock has joined #openstack-infra | 14:24 | |
| *** zhenguo has quit IRC | 14:24 | |
| *** ajmiller has joined #openstack-infra | 14:25 | |
| *** wenlock has quit IRC | 14:26 | |
| fungi | clarkb: yep, was hoping to get around to pushing up a patch for that shortly, unless you want to | 14:26 |
| clarkb | fungi: I am writing it now | 14:26 |
| *** Guest91834 is now known as jgriffith | 14:26 | |
| mordred | Shrews, SpamapS: actually, in looking at it - I tihnk our floating ip cleanup in the shade functional tests is wrong | 14:26 |
| *** wenlock has joined #openstack-infra | 14:26 | |
| mordred | we're passing ips in to cleanup that we do not actually know to be floating ips | 14:27 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use source balancing to transition to centos7 https://review.openstack.org/219709 | 14:27 |
| mordred | although it's mostly likely that they will be in the envs we've been running in | 14:27 |
| anteaya | do we have a gerrit topic for zuul3 yet? http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 14:27 |
| * anteaya can't find one | 14:27 | |
| *** esker has quit IRC | 14:27 | |
| anteaya | jeblair: ^ | 14:27 |
| *** zeih has joined #openstack-infra | 14:27 | |
| *** sdake has quit IRC | 14:28 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Make gate-tempest-dsvm-ironic-pxe_ssh vote again https://review.openstack.org/219712 | 14:28 |
| jroll | sdague: curious to hear your thoughts on this ^ :) | 14:29 |
| jroll | lucasagomes: fyi ^ | 14:29 |
| mordred | SpamapS, Shrews: I believe instead we should have cleanup_ips get a list of floating ips for the server in question, and then delete those - not just try to delete the floating ip associated with the external ip of the vm | 14:29 |
| *** fhubik has quit IRC | 14:29 | |
| *** liam__ has joined #openstack-infra | 14:29 | |
| lucasagomes | jroll, cheers! | 14:29 |
| *** jsavak has joined #openstack-infra | 14:30 | |
| *** mtanino has joined #openstack-infra | 14:30 | |
| *** kiran-r has quit IRC | 14:30 | |
| *** afazekas__ has quit IRC | 14:31 | |
| *** sdake has joined #openstack-infra | 14:32 | |
| anteaya | perhaps we don't have an inital patch for zuul3 yet so perhaps it hasn't come up | 14:32 |
| anteaya | is there a zuul3 etherpad? | 14:32 |
| clarkb | anteaya: I think there is just the spec which should list a topic | 14:32 |
| anteaya | clarkb: I don't see a topic in the spec | 14:32 |
| * anteaya looks again | 14:32 | |
| dhellmann | anteaya, clarkb : if you have a minute, could you help fix up the adoh permissions so we can do a release? https://review.openstack.org/#/c/218254/ | 14:33 |
| *** jsavak has quit IRC | 14:34 | |
| *** mpopow__ has joined #openstack-infra | 14:34 | |
| anteaya | dhellmann: it is alarming | 14:34 |
| anteaya | ha ha ha | 14:34 |
| * anteaya shows herself the door | 14:34 | |
| clarkb | dhellmann: does the global all-projects acl list allow release managers to push tags? | 14:34 |
| * clarkb looks it up | 14:35 | |
| dhellmann | clarkb: that's my understanding | 14:35 |
| *** rainya has quit IRC | 14:35 | |
| dhellmann | clarkb: I feel like I looked at a view in gerrit that showed that, but couldn't actually find the configuration file | 14:35 |
| *** mase_x200 has quit IRC | 14:35 | |
| clarkb | ya I see it, refs/* has a rule for releaes managers to push signed tags | 14:35 |
| * dhellmann wonders what was in anteaya's tea this morning | 14:35 | |
| anteaya | :) | 14:36 |
| anteaya | montreal | 14:36 |
| *** zeih has quit IRC | 14:36 | |
| dhellmann | clarkb: cool, thanks for double checking | 14:36 |
| *** eharney has quit IRC | 14:36 | |
| dhellmann | anteaya: that must be a large tea mug | 14:36 |
| anteaya | ha ha ha | 14:36 |
| *** liam__ has quit IRC | 14:36 | |
| *** masayukig has quit IRC | 14:36 | |
| *** kushal has joined #openstack-infra | 14:37 | |
| anteaya | thanks clarkb | 14:37 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/diskimage-builder: Deprecate the deploy-ironic element https://review.openstack.org/209079 | 14:37 |
| anteaya | away it goes | 14:37 |
| clarkb | fungi: jenkins04 still has a healthy thread count, I am going to check the others too but so far everything is good | 14:38 |
| *** mpopow__ has quit IRC | 14:38 | |
| *** e0ne has joined #openstack-infra | 14:39 | |
| *** esker has joined #openstack-infra | 14:39 | |
| *** glauco has quit IRC | 14:41 | |
| *** dkranz has quit IRC | 14:41 | |
| clarkb | fungi: 01-07 all look good | 14:41 |
| *** jsavak has joined #openstack-infra | 14:42 | |
| fungi | awesome. i have the replacement etherpad-dev building now (finally) and will swap dns over once puppet runs clean on it | 14:42 |
| *** puranamr has quit IRC | 14:42 | |
| *** alexpilotti has joined #openstack-infra | 14:42 | |
| *** markvoelker has joined #openstack-infra | 14:42 | |
| *** markvoelker has quit IRC | 14:42 | |
| clarkb | woot, I never left that comment about updating the sha1 for the prod server to what the dev server ends up installing and running with | 14:43 |
| clarkb | will do that now | 14:43 |
| *** markvoelker has joined #openstack-infra | 14:43 | |
| *** masayukig has joined #openstack-infra | 14:43 | |
| *** puranamr has joined #openstack-infra | 14:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating ip by ID instead of name https://review.openstack.org/219725 | 14:44 |
| fungi | i was going to anyway, just hadn't gotten around to updating the change | 14:44 |
| *** wshao has joined #openstack-infra | 14:44 | |
| *** wenlock_ has joined #openstack-infra | 14:44 | |
| clarkb | np I just want to make sure that the comment is recorded so we don't end up forgetting and I left it just now | 14:45 |
| *** spzala has joined #openstack-infra | 14:45 | |
| *** ildikov has quit IRC | 14:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix Aodh permissions to let the release team do its job https://review.openstack.org/218254 | 14:46 |
| *** sputnik13 has joined #openstack-infra | 14:47 | |
| *** hdd has joined #openstack-infra | 14:48 | |
| *** hdd has quit IRC | 14:50 | |
| *** hdd has joined #openstack-infra | 14:50 | |
| *** vinay_yadhav has quit IRC | 14:51 | |
| *** glauco has joined #openstack-infra | 14:52 | |
| *** asselin__ has quit IRC | 14:52 | |
| *** yamahata has joined #openstack-infra | 14:52 | |
| *** glauco has quit IRC | 14:53 | |
| *** esker has quit IRC | 14:53 | |
| *** dkranz has joined #openstack-infra | 14:54 | |
| *** k4n0 has quit IRC | 14:54 | |
| *** mriedem_meeting is now known as mriedem | 14:54 | |
| *** glauco has joined #openstack-infra | 14:55 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting puppet-neutron beaker on trusty https://review.openstack.org/219556 | 14:55 |
| *** sabeen has joined #openstack-infra | 14:57 | |
| *** maishsk has quit IRC | 14:59 | |
| *** sabeen2 has joined #openstack-infra | 14:59 | |
| *** sridhar_ram has quit IRC | 14:59 | |
| *** krotscheck is now known as kro_afk | 14:59 | |
| *** glauco has quit IRC | 15:00 | |
| *** vivekd has quit IRC | 15:00 | |
| *** MarkAtwood has joined #openstack-infra | 15:00 | |
| *** tiswanso has quit IRC | 15:00 | |
| *** mwagner_lap has quit IRC | 15:00 | |
| *** tiswanso has joined #openstack-infra | 15:01 | |
| *** vinayyadhav has joined #openstack-infra | 15:01 | |
| *** sabeen has quit IRC | 15:01 | |
| *** julim_ has joined #openstack-infra | 15:02 | |
| anteaya | so this patch failed twice on the same issue that hit zm02 yesterday, today it is zm08 same failure both times: https://review.openstack.org/#/c/219556/ | 15:02 |
| *** bhunter71 has joined #openstack-infra | 15:02 | |
| dhellmann | clarkb, anteaya : the adoh acl changes merged; is there a delay before those take effect in gerrit? | 15:02 |
| *** yamahata has quit IRC | 15:02 | |
| *** yamahata has joined #openstack-infra | 15:03 | |
| clarkb | dhellmann: yes have to wait for the next puppet run to kick off the gerrit config update | 15:03 |
| *** dizquierdo has quit IRC | 15:04 | |
| anteaya | dhellmann: https://review.openstack.org/#/admin/projects/openstack/aodh,access | 15:04 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source https://review.openstack.org/219351 | 15:04 |
| *** bcourt has quit IRC | 15:04 | |
| *** kro_afk is now known as krotscheck | 15:04 | |
| *** unicell1 has joined #openstack-infra | 15:04 | |
| dhellmann | clarkb, anteaya : ok, I'll check back in later today. Thanks for the assist! | 15:04 |
| *** julim has quit IRC | 15:05 | |
| anteaya | dhellmann: sure | 15:05 |
| *** unicell has quit IRC | 15:06 | |
| anteaya | so should I just keep rechecking 219556 to get a different zm than 08? | 15:06 |
| anteaya | that feels wrong | 15:07 |
| *** ildikov has joined #openstack-infra | 15:07 | |
| fungi | anteaya: see my comment on your story, that error is nothing to do with the zuul mergers after all and we suspect it has to do with splitting related git requests across different versions of git servers | 15:08 |
| anteaya | ah | 15:08 |
| * anteaya goes back to her story | 15:08 | |
| *** gildub has quit IRC | 15:08 | |
| fungi | anteaya: hopefully 219709 solves it while we finish the git server upgrades | 15:08 |
| anteaya | wonderful thank you | 15:10 |
| *** vinayyadhav has quit IRC | 15:10 | |
| anteaya | so within about 40ish minutes or so, after it merges and a puppet run picks it up | 15:10 |
| timrc | fungi, clarkb, jeblair Looking for some guidance. We hit a fairly weird edge case in zuul just now -- http://paste.openstack.org/show/441456/ | 15:11 |
| openstackgerrit | Merged openstack-infra/system-config: Use source balancing to transition to centos7 https://review.openstack.org/219709 | 15:11 |
| timrc | fungi, clarkb, jeblair What happened is a user git review'ed two commits and made the child a draft. | 15:11 |
| *** ddieterly is now known as deklan | 15:11 | |
| timrc | The check jobs ran on patchset-created for 82592,1 but fell over on 'recheck' | 15:12 |
| jeblair | timrc: drafts are unworkable; we disabled them | 15:12 |
| *** dims_ has quit IRC | 15:12 | |
| *** jistr is now known as jistr|call | 15:12 | |
| fungi | timrc: yep, that's a known behavior (zuul can't enqueue if it can't read all changes in the chain, both child and parent) | 15:12 |
| timrc | So why would tests start on upload but not on recheck? | 15:12 |
| timrc | That seems inconsistent to me. | 15:13 |
| fungi | timrc: probably because when zuul saw the parent the first time, the child hadn't hit gerrit yet | 15:13 |
| timrc | FWIW he added zuul as reviewer to give it the necessary visibility to unblock him. I'm happy to document that as what you need to do should you do something like that. | 15:13 |
| fungi | for a split second | 15:13 |
| fungi | it's one of the reasons we disabled drafts in our gerrit | 15:14 |
| timrc | fungi, Good to know. Maybe something we should consider as well. | 15:14 |
| *** dannywilson has joined #openstack-infra | 15:14 | |
| clarkb | they dont really provide any vlaue over a WIP state since they are very leaky and shouldn't be treated as a secure way to hide information | 15:14 |
| *** bexelbie has quit IRC | 15:14 | |
| clarkb | and they break gerrit in all sorts of spectacular ways | 15:14 |
| anteaya | fungi: I updated the story 2000341 to include a link to clarkb's patch and the git backend postulation | 15:15 |
| clarkb | one of my favorites is pusing a draft patchset after a non draft patchset, confuses reviewers when they cannot vote but are looking at what they see is the latest patchset | 15:15 |
| timrc | clarkb, Yeah. | 15:15 |
| clarkb | another good one is two changes where there parent is made a draft | 15:15 |
| clarkb | now the child can't merge and no one knows why | 15:15 |
| clarkb | (this is all independent of zuul too fwiw, gerrit's draft implementation is just confusing) | 15:16 |
| *** dannywil_ has joined #openstack-infra | 15:16 | |
| *** dannywilson has quit IRC | 15:16 | |
| rcarrillocruz | s/draft// | 15:16 |
| fungi | that too | 15:17 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use os-brick 0.4.0 https://review.openstack.org/219736 | 15:18 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use python-novaclient 2.27.0 https://review.openstack.org/219737 | 15:19 |
| *** pc_m is now known as pc_m_AWAY | 15:19 | |
| clarkb | fungi: let me know when new etherpad-dev is ready for some testing | 15:19 |
| fungi | clarkb: it's up and dns is changed, but ttl may still lag | 15:20 |
| clarkb | fungi: and when we rebuild the prod server don't forget to make it bigger | 15:20 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use python-heatclient 0.7.0 https://review.openstack.org/219738 | 15:20 |
| clarkb | fungi: 104.130.252.212 looks like the new one ya? | 15:20 |
| *** bcourt has joined #openstack-infra | 15:20 | |
| fungi | clarkb: 2001:4800:7818:103:be76:4eff:fe04:9718 104.130.252.212 is the new one, yeah | 15:20 |
| fungi | right, yep | 15:21 |
| jeblair | clarkb: make what bigger? | 15:21 |
| fungi | etherpad.o.o replacement | 15:21 |
| jeblair | why? | 15:21 |
| clarkb | jeblair: beacuse it ran out of memory during the ops summit | 15:21 |
| *** Sukhdev_ has joined #openstack-infra | 15:21 | |
| clarkb | jeblair: we couldn't even run puppet on it without invoking OOMkiller | 15:21 |
| jeblair | i thought that was because of runaway abiword | 15:21 |
| clarkb | jeblair: no, nodejs and puppet wre the biggest ocnsumers | 15:22 |
| *** eharney has joined #openstack-infra | 15:22 | |
| clarkb | abiword was eating cpu not memory | 15:22 |
| jeblair | ah ok. 2 -> 4G then? | 15:22 |
| jeblair | or jumping to 8? | 15:22 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use python-ironicclient 0.8.0 https://review.openstack.org/219740 | 15:22 |
| fungi | and bup was eating disk i/o, so it's possible the memory balloon was related to other performance degredation, but still 4g seems reasonable | 15:22 |
| clarkb | ya I don't think we need to go huge, 4GB is probably a good step up | 15:23 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=117&rra_id=4&view_type=&graph_start=1408154147&graph_end=1441207331 | 15:23 |
| kfox1111 | Any more reviewers for: https://review.openstack.org/#/c/218898/ ? | 15:23 |
| jeblair | that's an interesting graph | 15:23 |
| jeblair | clarkb: i don't think the memory usage was related to etherpad usage | 15:24 |
| jeblair | it looks more like a slow leak over time | 15:24 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=117&rra_id=4&view_type=&graph_start=1435922638&graph_end=1441121005&graph_height=120&graph_width=500&title_font_size=10 | 15:24 |
| mordred | that just happened to hit its peak at the wrong time | 15:24 |
| jeblair | yeah | 15:24 |
| jeblair | in the second link, you can see that it was oom during the ops summit, but after that first restart, it didn't jump back up | 15:25 |
| fungi | s/its peak/capacity/ | 15:25 |
| *** peristeri has quit IRC | 15:25 | |
| *** jswarren_ is now known as jswarren | 15:25 | |
| clarkb | https://etherpad-dev.openstack.org/p/woo-trusty-woo | 15:25 |
| fungi | it still seems to be able to effectively fill all available memory with cache, so it might see some performance benefit from a ram increase regardless | 15:26 |
| openstackgerrit | Arun Kant proposed openstack/requirements: Updating pycadf to use recent taxonomy changes https://review.openstack.org/219742 | 15:26 |
| jeblair | anyway, i don't think we _need_ an increase, but it won't hurt. it will mean we only have to restart it every 2 months instead of every month. :) | 15:26 |
| clarkb | looks like it started back in april ish which is when we upgraded to more modern etherpad | 15:27 |
| clarkb | to address security concerns | 15:27 |
| fungi | moar security==moar leaky, apparently | 15:27 |
| anteaya | just moar | 15:27 |
| mordred | jeblair: so - reading some articles on node.js heap profiling and management | 15:27 |
| mordred | a case could be made that with a leaky app more memory will actually make it worse | 15:28 |
| mordred | because of how the v8 GC system works | 15:28 |
| anteaya | awesome | 15:28 |
| *** mwagner_lap has joined #openstack-infra | 15:28 | |
| fungi | puppetmaster hasn't spotted the new dns entry for etherpad-dev yet, so i've not updated its cached host key but i'll do that after lunch | 15:28 |
| fungi | gotta scoot, back in a little while | 15:29 |
| mordred | as it fills memory the GC gets more aggressive, which means it runs more and takes more CPU - but also if there is more traverse while GCing, then the cost of the rampant GC cycles is going to be higher | 15:29 |
| jeblair | mordred: ugh | 15:29 |
| clarkb | mordred: when you say fills memory the GC gets more aggressive is that absolute "fill" or relative to available memory? | 15:29 |
| *** wshao has quit IRC | 15:30 | |
| jeblair | so let's strike "won't hurt" from my summary. :) | 15:30 |
| mordred | jeblair: things I've read so far: http://stackoverflow.com/questions/16586705/configuring-v8s-memory-management-to-be-smart-for-a-node-js-process https://strongloop.com/strongblog/node-js-performance-garbage-collection/ https://strongloop.com/strongblog/node-js-performance-heap-profiling-tip/ | 15:30 |
| *** wshao has joined #openstack-infra | 15:30 | |
| mordred | clarkb: the one that talked about aggressive GC did not indicate which | 15:30 |
| jeblair | clarkb: in our case don't those numbers approach the same value? | 15:31 |
| *** puranamr has quit IRC | 15:31 | |
| mordred | the strongloop articles are annoying becuase they're trying to sell their enterprise node stuff | 15:31 |
| mordred | but there's still good internals info in there | 15:31 |
| clarkb | jeblair: they do but adding 2GB won't be an immediate issue if its relative | 15:31 |
| clarkb | mordred: the one article says it sa 1.7GB heap cap anyways | 15:32 |
| clarkb | mordred: so adding memory won't hurt unless I misunderstand something | 15:32 |
| jeblair | i'm going to putz with websockets on dev | 15:33 |
| clarkb | mordred: but may give puppet breathing room | 15:33 |
| *** puranamr has joined #openstack-infra | 15:34 | |
| jeblair | ReferenceError: io is not defined in https://etherpad-dev.openstack.org/javascripts/lib/ep_etherpad-lite/static/js/pad.js?callback=require.define at line 232' | 15:34 |
| jeblair | neat, eplite is broken with websockets? | 15:34 |
| mordred | awesome | 15:34 |
| mordred | https://hacks.mozilla.org/2012/11/tracking-down-memory-leaks-in-node-js-a-node-js-holiday-season/ <-- node-memwatch might be worth playing with | 15:35 |
| j^2 | I’m starting to build tho OpenStack Operators launchpad, i accidently gave OpenStack Administrators Maintainer rights and removed my own for the set up, can someone help me out here? | 15:35 |
| clarkb | mordred: so ya if I read this correctly we are capped at 1GB by default or 1.7GB if we fiddle with settings, in either case bumping the system memory shouldn't affect node but will give puppet memory to run | 15:35 |
| *** markus_z has quit IRC | 15:35 | |
| jeblair | clarkb: that sounds good to me | 15:35 |
| mordred | clarkb: ++ | 15:36 |
| *** jsavak has quit IRC | 15:36 | |
| *** MarkAtwood has quit IRC | 15:36 | |
| *** geoffarnold has joined #openstack-infra | 15:36 | |
| jeblair | clarkb: are we running latest sha of eplite? | 15:36 |
| clarkb | mordred: I think any memory management for node.js ahs to be independent of the host memory space, so good thing to investigate further but I dont expect bumping host size to factor in much other than allowing more processes to be happy | 15:36 |
| clarkb | jeblair: on dev we are | 15:36 |
| clarkb | jeblair: thats their presribed deployment method | 15:36 |
| *** ayoung_ZZzz is now known as ayoung | 15:36 | |
| *** tonytan4ever has joined #openstack-infra | 15:37 | |
| *** geoffarnold has quit IRC | 15:38 | |
| *** jistr|call is now known as jistr | 15:38 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 15:39 |
| *** geoffarnold has joined #openstack-infra | 15:39 | |
| clarkb | jeblair: that error looks like its just missing a dependency? maybe we need to add something ot the npm install list? | 15:39 |
| *** glauco has joined #openstack-infra | 15:39 | |
| clarkb | or maybe it failed to get the files over websockets and websockets in general dont work | 15:39 |
| *** armax has joined #openstack-infra | 15:40 | |
| jeblair | clarkb: both of those seem plausible | 15:40 |
| jeblair | rcarrillocruz: did you run into this? | 15:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating ip by ID instead of name https://review.openstack.org/219725 | 15:42 |
| *** sshnaidm has quit IRC | 15:42 | |
| clarkb | mordred: out of curiousity why not continue to delete by ip? | 15:42 |
| clarkb | mordred: since you discovered the ip was probably unrelated | 15:42 |
| openstackgerrit | Glauco Oliveira proposed openstack-infra/puppet-zuul: Add acceptance tests for puppet-zuul https://review.openstack.org/217732 | 15:43 |
| mordred | clarkb: well, I could - but everyone else seemed to work with ids - and with that new code I _have_ the ids, so why not? | 15:43 |
| *** Ala has quit IRC | 15:43 | |
| mordred | clarkb: I don't have as strong a feeling about that as I do about not assuming that the public IP is a floating IP | 15:44 |
| clarkb | mordred: oh I work with IPs | 15:44 |
| clarkb | because in theory they are unique (because globally routable) and its an extra lookup to find the id | 15:44 |
| mordred | yah. I tend to work with IPs too | 15:44 |
| mordred | but in this particular loop, I have the fip object | 15:45 |
| *** wshao has quit IRC | 15:45 | |
| *** cdent has quit IRC | 15:45 | |
| jeblair | clarkb: i think it's the second thing (failed to get file) | 15:45 |
| mordred | (and it's a cleanup, not a test of functionality - I think if we want to test that we can delete a floating ip by address, there should be a test that creates a floating ip and then deletes it) | 15:46 |
| *** dprince has quit IRC | 15:47 | |
| clarkb | jeblair: I agree if you dig under *_modules far enough you find that there is a socket.io present | 15:47 |
| *** hodos has joined #openstack-infra | 15:47 | |
| clarkb | jeblair: which I believe is the require happening in the error you pasted | 15:48 |
| clarkb | mordred: +1 | 15:48 |
| *** jsavak has joined #openstack-infra | 15:48 | |
| *** mpopow__ has joined #openstack-infra | 15:50 | |
| jeblair | the proxy config was wrong | 15:51 |
| jeblair | rcarrillocruz: ^ | 15:51 |
| *** amitgandhinz has quit IRC | 15:51 | |
| clarkb | mordred: are the two fixed_ip and fixed_ip_address keys from the neutron/nova apis? that seems funny but I know better than to blame shade for it :) | 15:51 |
| jeblair | disabling puppet on epdev | 15:51 |
| *** dims has joined #openstack-infra | 15:51 | |
| *** doude has quit IRC | 15:52 | |
| jeblair | "101 Switching Protocols" :) | 15:52 |
| *** kiran-r has joined #openstack-infra | 15:53 | |
| clarkb | take one down pass it around | 15:53 |
| *** bharathm has joined #openstack-infra | 15:54 | |
| *** mpopow__ has quit IRC | 15:54 | |
| *** regXboi has quit IRC | 15:55 | |
| *** amuller is now known as amuller_afk | 15:55 | |
| *** yfried has quit IRC | 15:55 | |
| *** esker has joined #openstack-infra | 15:57 | |
| *** mtanino has quit IRC | 15:57 | |
| *** gyee has joined #openstack-infra | 15:57 | |
| *** ivar-lazzaro has joined #openstack-infra | 15:57 | |
| SpamapS | mordred: OpenStack doesn't automatically deallocate floating ips when you delete a server? or am I missing something? | 15:58 |
| *** bharathm has quit IRC | 15:59 | |
| *** esker has quit IRC | 15:59 | |
| clarkb | SpamapS it doesnt | 16:00 |
| jeblair | SpamapS: when you say deallocate, do you mean "remove from server" or do you mean "remove from user's account"? | 16:00 |
| *** jsavak has quit IRC | 16:00 | |
| clarkb | it disassociates but does not delete | 16:00 |
| *** jsavak has joined #openstack-infra | 16:00 | |
| openstackgerrit | Glauco Oliveira proposed openstack-infra/puppet-zuul: Add acceptance tests for puppet-zuul https://review.openstack.org/217732 | 16:00 |
| rcarrillocruz | jeblair: what's up | 16:00 |
| jeblair | SpamapS: the first one should happen, the second should not (so that floating ips can be re-used as permanent service addresses) | 16:01 |
| rcarrillocruz | may i please get some eyes on https://review.openstack.org/#/c/219181/ and https://review.openstack.org/#/c/219185/ | 16:01 |
| jeblair | rcarrillocruz: we're trying websockets on etherpad-dev and the apache config for them didn't work; i had to modify it. i was wondering if you are using websockets with etherpad-lite? | 16:01 |
| rcarrillocruz | dependant changes | 16:02 |
| rcarrillocruz | what error you have? | 16:02 |
| *** unicell1 has quit IRC | 16:02 | |
| *** dims has quit IRC | 16:02 | |
| *** kiran-r has quit IRC | 16:02 | |
| *** dims has joined #openstack-infra | 16:03 | |
| rcarrillocruz | i still may have the vagrant vm i spinned for the etherpad websockets thing | 16:03 |
| *** markvoelker has quit IRC | 16:03 | |
| jeblair | rcarrillocruz: it was a 404 on sockte.io.js; i had to change "ProxyPass /socket.io http://localhost:9001" to "ProxyPass /socket.io http://localhost:9001/socket.io" | 16:03 |
| jeblair | socket.io.js | 16:04 |
| *** cburroughs has quit IRC | 16:04 | |
| *** jlanoux has quit IRC | 16:04 | |
| *** dims_ has joined #openstack-infra | 16:04 | |
| rcarrillocruz | that's on trusty, right? | 16:05 |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s) https://review.openstack.org/219760 | 16:05 |
| rcarrillocruz | -dev i mean | 16:05 |
| jeblair | rcarrillocruz: yes | 16:05 |
| rcarrillocruz | k, cos i still have the two vagrant vms i tested this | 16:05 |
| rcarrillocruz | precise and trusty | 16:05 |
| rcarrillocruz | i'll check it out and let you know | 16:05 |
| *** bknudson has joined #openstack-infra | 16:06 | |
| jeblair | rcarrillocruz: also, the manifest does not install mod_proxy_wstunnel by default, i had to manually do that | 16:06 |
| *** dims has quit IRC | 16:07 | |
| *** spzala has quit IRC | 16:07 | |
| rcarrillocruz | that's a bug, i'll push up something | 16:07 |
| *** mtanino has joined #openstack-infra | 16:08 | |
| *** andersonvom has joined #openstack-infra | 16:10 | |
| *** stevemar has joined #openstack-infra | 16:11 | |
| *** bharathm has joined #openstack-infra | 16:12 | |
| mordred | clarkb: yes - it's called one thing in nova and one in neutron | 16:13 |
| mordred | clarkb: clearly that particularly chunk of cleanup logic code should probably become a shade API at some point | 16:13 |
| mordred | "get_floating_ips_for_server" or somethin | 16:13 |
| mordred | SpamapS: the number of things that don't happen that should boggle the mind | 16:14 |
| clarkb | mordred: thanks | 16:14 |
| *** esker has joined #openstack-infra | 16:14 | |
| *** angela-s has joined #openstack-infra | 16:14 | |
| jeblair | hrm, firebug has no visibility into websocket traffic :( | 16:14 |
| mordred | SpamapS, jeblair: I'd argue that the floating-ip interface should be similar to the cinder volume one - you can create a volume on server boot and mark it as "delete when server goes away" | 16:14 |
| jeblair | mordred: i would not dispute that (as an option) | 16:15 |
| mordred | which is what we'd do with nodepool vms - since we know that we do not, in fact, want the ips to live | 16:15 |
| jeblair | ya | 16:15 |
| mordred | but then, for things where we DO want the floating-ip to be an object we manage | 16:15 |
| mordred | we do not want openstack to delete them for us | 16:15 |
| mordred | for those reasons | 16:15 |
| jeblair | apparently chromium devtools can show them, but first i need to figure out how to get chromium to permanently accept the ssl cert | 16:16 |
| *** afazekas__ has joined #openstack-infra | 16:16 | |
| *** devlaps has joined #openstack-infra | 16:16 | |
| clarkb | jeblair: you have to accept it system wide because chromium | 16:16 |
| clarkb | or in your user settings for whatever desktop system you use | 16:16 |
| *** john-davidge has quit IRC | 16:17 | |
| *** julim_ has quit IRC | 16:17 | |
| *** alexpilotti has quit IRC | 16:17 | |
| *** deklan has quit IRC | 16:20 | |
| *** deklan has joined #openstack-infra | 16:21 | |
| clarkb | we are using the source balance method now, if anyone sees the layout job fail due to git clones please let us know here | 16:21 |
| zaro | morning | 16:22 |
| clarkb | jeblair: mordred now that we are balancing based on source https://review.openstack.org/#/c/219427/ should be safe to go in | 16:22 |
| clarkb | zaro: morning | 16:22 |
| mordred | clarkb: k. +3 | 16:22 |
| *** esker has quit IRC | 16:23 | |
| *** jtomasek has quit IRC | 16:23 | |
| *** jsavak has quit IRC | 16:23 | |
| *** FallenPegasus has joined #openstack-infra | 16:24 | |
| *** jsavak has joined #openstack-infra | 16:24 | |
| *** burgerk_ has joined #openstack-infra | 16:24 | |
| *** abregman has quit IRC | 16:25 | |
| *** spzala has joined #openstack-infra | 16:26 | |
| sdague | jroll: is it working ? | 16:27 |
| *** andersonvom has quit IRC | 16:28 | |
| zaro | clarkb: yo! getting closer to finding the root of the jgit issue. looks like the problem occurs when jgit attmepts removePack when there are pack errors https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java#L562 | 16:28 |
| nibalizer | mordred: i agree about fips | 16:28 |
| *** Piet has joined #openstack-infra | 16:28 | |
| sdague | jroll: so the prefered direction is the ipa job right? I'd say change all the non ironic / dib bits over to ipa as well | 16:28 |
| nibalizer | cleaning up floating ips forom servers that were deleted is annoying | 16:28 |
| nibalizer | plus I always forget to do it | 16:28 |
| *** doude_ has joined #openstack-infra | 16:28 | |
| zaro | clarkb: that causes the repo to be inaccessable. | 16:28 |
| clarkb | zaro: that sounds right, basically there is an error due to a diff timeout and that error is interpreted as a corrupt pack file | 16:29 |
| sdague | mordred: do you have logs for where this nova thing went wrong? | 16:29 |
| jeblair | clarkb: export and then: certutil -d sql:$HOME/.pki/nssdb -A -t P -n <certificate nickname> -i <certificate filename> | 16:29 |
| jeblair | clarkb: i love google user interfaces! | 16:29 |
| SpamapS | jeblair: I meant deallocate from the server, not delete from the user. :) | 16:29 |
| clarkb | nibalizer: the trick is to delete it first but you do have to remember | 16:29 |
| mordred | sdague: yes - although I'm now not sure it was a server error | 16:30 |
| akuznetsova | hi guys! | 16:30 |
| sdague | if those logs are up somewhere, I can look | 16:30 |
| *** sergmelikyan has joined #openstack-infra | 16:30 | |
| *** sergmelikyan has quit IRC | 16:30 | |
| zaro | clarkb: so yeah, i think the error should not be handled as a corrupted pack. i'm thinking gerrit should do something else in that case, or maybe nothing at all? | 16:30 |
| clarkb | zaro probably just log the timeout and send error to http cliemt? | 16:31 |
| mordred | sdague: http://logs.openstack.org/55/219655/1/check/gate-shade-dsvm-functional-nova/1154770/console.html#_2015-09-02_12_10_56_113 | 16:31 |
| clarkb | but definitely dont mark thr packfile as corrupt | 16:32 |
| nibalizer | clarkb: this is hwat I do these days https://github.com/nibalizer/dotfiles/blob/master/.bashrc#L280 | 16:32 |
| nibalizer | annoying and dumb but functional | 16:32 |
| zaro | clarkb: that makes sense. | 16:32 |
| *** sergmelikyan has joined #openstack-infra | 16:32 | |
| *** mpopow__ has joined #openstack-infra | 16:32 | |
| akuznetsova | am I correct that if some project is specified in devstack-vm-gate-wrap.sh like PROJECTS="openstack/project $PROJECTS" then I need to add some fixes in project congig project.yaml and export export BRANCH_OVERRIDE={branch-override}if [ "$BRANCH_OVERRIDE" != "default" ] ; then | 16:32 |
| akuznetsova | export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE | 16:32 |
| akuznetsova | fi ? | 16:32 |
| *** hashar_ has joined #openstack-infra | 16:33 | |
| akuznetsova | oops, sorry for ugly formatting | 16:33 |
| *** sergmelikyan has quit IRC | 16:34 | |
| *** afazekas__ has quit IRC | 16:35 | |
| *** hashar has quit IRC | 16:35 | |
| jeblair | clarkb: http://i.imgur.com/QhOJ5JV.png | 16:35 |
| jeblair | websocket frames! | 16:35 |
| clarkb | neat | 16:37 |
| jeblair | there is a heartbeat in there -- client sends "2" server responds "3". so at least client side, it can still do disconnection detection | 16:37 |
| jeblair | hopefully there's a sever-side timeout so we don't end up with lots of stuck-open connections when conference wifi reboots | 16:38 |
| *** earlephilhower has joined #openstack-infra | 16:38 | |
| sdague | mordred: yeh, so the REST resource should have always been by id | 16:38 |
| sdague | I'm curious when that worked | 16:39 |
| openstackgerrit | Merged openstack-infra/system-config: Swap out git07 to new centos7 node https://review.openstack.org/219427 | 16:39 |
| mordred | sdague: it's pretty much worked constantly until just now | 16:39 |
| nibalizer | jeblair: thats pretty slick | 16:39 |
| sdague | https://github.com/openstack/nova/blob/7a1ef0b53874d3a596583aa5ad451ae3c72637f2/nova/network/api.py#L128 - that line dates back to feb | 16:40 |
| *** julim has joined #openstack-infra | 16:40 | |
| *** dprince has joined #openstack-infra | 16:40 | |
| sdague | but it dates back further than that | 16:40 |
| *** julim has quit IRC | 16:40 | |
| sdague | mordred: nova-net or neutron backend? | 16:41 |
| *** julim has joined #openstack-infra | 16:41 | |
| *** hashar_ has quit IRC | 16:41 | |
| mordred | sdague: nova-net - all of our nova-net functional tests started failing this morning with that error | 16:41 |
| mordred | sdague: they'd al been passing up until then | 16:41 |
| *** cdent has joined #openstack-infra | 16:42 | |
| *** FallenPegasus has quit IRC | 16:42 | |
| *** ashleighfarnham has joined #openstack-infra | 16:42 | |
| *** ashleighfarnham has quit IRC | 16:42 | |
| *** sergmelikyan has joined #openstack-infra | 16:42 | |
| sdague | hmm | 16:43 |
| jeblair | mordred: wait, did shade just catch a nova api break? | 16:43 |
| *** marzif has quit IRC | 16:43 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues https://review.openstack.org/219514 | 16:43 |
| *** marzif has joined #openstack-infra | 16:43 | |
| *** jistr has quit IRC | 16:44 | |
| *** amuller_afk is now known as amuller | 16:44 | |
| *** angdraug has joined #openstack-infra | 16:44 | |
| mordred | jeblair: it's possible. although the docs do not indicate that it should have worked before | 16:45 |
| *** larainema has quit IRC | 16:45 | |
| *** sergmelikyan has quit IRC | 16:45 | |
| jeblair | mordred: which may explain the lack of a test for that. | 16:45 |
| jeblair | so, more like it caught an undefined api break :) | 16:46 |
| *** larainema has joined #openstack-infra | 16:46 | |
| *** sergmelikyan has joined #openstack-infra | 16:46 | |
| *** unicell has joined #openstack-infra | 16:47 | |
| *** ifarkas has quit IRC | 16:47 | |
| *** pc_m_AWAY is now known as pc_m | 16:48 | |
| *** esmute_ has joined #openstack-infra | 16:48 | |
| sdague | yeh, it definitely wasn't meant to work the way shade was using it | 16:48 |
| jeblair | clarkb: do we need to re-tune apache to increase the number of simultaneous connections it can support, since websockets will maintain long-running connections? | 16:48 |
| *** afazekas__ has joined #openstack-infra | 16:49 | |
| sdague | however, I'm still really curious to figure out what changed | 16:49 |
| sdague | because, from all the code that I'm looking at, this should have thrown an error for at least a year | 16:49 |
| clarkb | jeblair possibly, and also file descriptor limits | 16:49 |
| sdague | mordred: you happen to have the first test run where this failed to narrow it down? | 16:50 |
| *** jtomasek has joined #openstack-infra | 16:50 | |
| *** MIDENN_ has quit IRC | 16:50 | |
| *** mdenny has quit IRC | 16:50 | |
| *** lucasagomes is now known as lucas-dinner | 16:51 | |
| *** MIDENN_ has joined #openstack-infra | 16:52 | |
| *** mdenny has joined #openstack-infra | 16:52 | |
| sdague | oh, so I actually know why this failed | 16:52 |
| mordred | oh! yay! | 16:52 |
| sdague | or I have a good guess | 16:52 |
| sdague | so can you file a bug on this | 16:52 |
| mordred | (I mean, even if it was a behavior change, it was a buggy cleanup and the new version I like better anyway) | 16:53 |
| mordred | sdague: sure! | 16:53 |
| mordred | sdague: on nova? | 16:53 |
| sdague | yeh | 16:53 |
| *** mpopow__ has quit IRC | 16:53 | |
| sdague | we should evaluate | 16:53 |
| greghaynes | SpamapS: I've added a property to some of our inventory/mac.yaml files of reserved: true for the nodes that gh2bf should skip over | 16:53 |
| sdague | this started failing about 12 hours ago? | 16:53 |
| greghaynes | SpamapS: for crinkle's nodes and for the new bastion im spinning up | 16:53 |
| mordred | sdague: yeah - give of take | 16:53 |
| mordred | sdague: my tests yesterday were passing nova but failing neutorn - but that was due to a bug in my code | 16:54 |
| sdague | https://review.openstack.org/#/c/214085/ - we changed the default paste pipelines. The new one was supposed to work correctly in this scenario, however we apparently missed a thing | 16:54 |
| jeblair | clarkb: i think our existing tuning is probably a sufficient starting place; it looks like you sized it assuming everyone's making GET requests at the same time anyway | 16:54 |
| mordred | then, today, I fixed my bug and started hitting the nova things | 16:54 |
| *** jerryz has joined #openstack-infra | 16:54 | |
| mordred | sdague: ah - that would explain it | 16:54 |
| sdague | yeh, so, I don't think the old code actually worked | 16:54 |
| sdague | I think it just failed silently | 16:54 |
| mordred | AH. LOVELY | 16:55 |
| mordred | so, really, fixing it in our code is more better | 16:55 |
| sdague | yeh, but we should evaluate the fact that those old requests didn't explode before, and if we care in that case. | 16:55 |
| sdague | so, you definitely want to fix shade to use ids, because that's actually the API | 16:56 |
| *** earlephilhower has quit IRC | 16:56 | |
| clarkb | jeblair 2k iirc | 16:56 |
| jeblair | clarkb: ypp | 16:56 |
| *** mcdonagm has quit IRC | 16:56 | |
| *** mcdonagm__ has quit IRC | 16:56 | |
| *** shashank_hegde has joined #openstack-infra | 16:56 | |
| sdague | and I'll take the bug into the nova team and evaluate if we want to care about this particular break | 16:56 |
| *** tonytan4ever has quit IRC | 16:57 | |
| greghaynes | SpamapS: I am also assigning the 15.x ip's to the nodes starting at 15.184.52.100, we have less than 100bm nodes so that should make us within the bm /24 | 16:57 |
| mordred | sdague: https://bugs.launchpad.net/nova/+bug/1491512 | 16:57 |
| openstack | Launchpad bug 1491512 in OpenStack Compute (nova) "Behavior change with latest nova paste config" [Undecided,New] | 16:57 |
| *** salv-orlando has quit IRC | 16:57 | |
| sdague | mordred: thanks | 16:57 |
| *** puranamr has quit IRC | 16:57 | |
| *** derekh has quit IRC | 16:57 | |
| mordred | sdague: thanks for looking at it! | 16:57 |
| *** earlephilhower has joined #openstack-infra | 16:58 | |
| mordred | woot! the new test worked | 16:58 |
| *** julim has quit IRC | 16:58 | |
| mordred | jeblair, SpamapS: https://review.openstack.org/#/c/219725/ +3 me? | 16:58 |
| Shrews | mordred: got it. thx sdague | 16:59 |
| *** puranamr has joined #openstack-infra | 16:59 | |
| sdague | no prob, thanks for raising the issue | 16:59 |
| *** sharikapongubala has quit IRC | 17:00 | |
| sdague | mordred: do you have logs of runs that are a day old, so I can see what this looked like in the service logs back then | 17:00 |
| *** SotK has quit IRC | 17:01 | |
| *** deklan has quit IRC | 17:01 | |
| *** julim has joined #openstack-infra | 17:01 | |
| *** annegentle has quit IRC | 17:01 | |
| *** Zara has quit IRC | 17:01 | |
| jeblair | clarkb, fungi: the thing with the proxypass retry i was worried about should not be a problem -- because the proxypass line is now no longer exactly the same as the the mod_rewrite proxy line, the retry=0 should take effect | 17:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface https://review.openstack.org/214243 | 17:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move keystone to common identity client interface https://review.openstack.org/214244 | 17:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass timeout to session, not constructors https://review.openstack.org/219290 | 17:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove last vestige of glanceclient being different https://review.openstack.org/219291 | 17:02 |
| mordred | sdague: yup. one sec | 17:02 |
| jeblair | clarkb, fungi: apache is not complaining about it, and i have performed a simulation of the 500 error propogation that we saw during the ops summit and verified that it works as expected | 17:02 |
| *** ashleighfarnham has joined #openstack-infra | 17:02 | |
| sdague | mordred: also, I would add a check at the end of - https://github.com/openstack-infra/shade/blob/ceb6a1d36ea197d6489d981c112ec187cf29a961/shade/tests/functional/test_floating_ip.py#L232 | 17:03 |
| jeblair | clarkb, fungi: (i was also able to reproduce the 500 error propogation by removing the retry=0 line; so i'm fairly confident in my test) | 17:03 |
| mordred | sdague: this is about 2 weeks old: http://logs.openstack.org/44/214244/2/check/gate-shade-dsvm-functional-nova/2009d01/ | 17:03 |
| sdague | that actually makes sure the floating ip is removed | 17:03 |
| jeblair | clarkb, fungi: i will write up my local changes now | 17:03 |
| mordred | sdague: yeah - I was thinking the same thing | 17:03 |
| sdague | because, I'm betting it just never was | 17:03 |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s) https://review.openstack.org/219760 | 17:03 |
| *** woodster_ has joined #openstack-infra | 17:03 | |
| mordred | sdague: here's from yesterday http://logs.openstack.org/43/214243/3/check/gate-shade-dsvm-functional-nova/bd70567/ | 17:04 |
| sdague | mordred: cool | 17:04 |
| *** cburroughs has joined #openstack-infra | 17:04 | |
| *** rossella_s has quit IRC | 17:05 | |
| *** Sukhdev_ has quit IRC | 17:06 | |
| *** rossella_s has joined #openstack-infra | 17:06 | |
| sdague | mordred: are you ignore 404? | 17:06 |
| sdague | http://logs.openstack.org/43/214243/3/check/gate-shade-dsvm-functional-nova/bd70567/logs/screen-n-api.txt.gz#_2015-09-01_19_18_35_379 | 17:06 |
| *** arxcruz has quit IRC | 17:07 | |
| pc_m | Can infra cores please review updates to https://review.openstack.org/#/c/209887/ for me? Neutron team decided to have two jobs, one for core neutron-client tests and one for advanced services based tests (which will enable the plugins in hook script). | 17:07 |
| sdague | mordred: yeh, you were being handed a 404 100% of the time before | 17:07 |
| sdague | http://logs.openstack.org/43/214243/3/check/gate-shade-dsvm-functional-nova/bd70567/logs/screen-n-api.txt.gz#_2015-09-01_19_18_47_066 | 17:07 |
| *** sridhar_ram has joined #openstack-infra | 17:08 | |
| sdague | I guess now you get a 400 | 17:08 |
| sdague | because we validate first | 17:08 |
| sdague | but in that run from yesterday, it did not return success | 17:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-etherpad_lite: Fix websocket configuration https://review.openstack.org/219790 | 17:08 |
| clarkb | its nova client receiving it so possible client bug | 17:09 |
| *** skylerberg has joined #openstack-infra | 17:09 | |
| jeblair | clarkb, fungi, rcarrillocruz: ^ that's on-system on etherpad-dev and working | 17:09 |
| mordred | sdague: sweet! | 17:09 |
| sdague | same 2 weeks ago http://logs.openstack.org/44/214244/2/check/gate-shade-dsvm-functional-nova/2009d01/logs/screen-n-api.txt.gz | 17:09 |
| sdague | search for os-floating-ips/172 | 17:09 |
| sdague | and you'll see they were 404s | 17:09 |
| sdague | so I'm surprised that test passed | 17:09 |
| *** sridhar_ram has quit IRC | 17:09 | |
| jeblair | this is cool; etherpad is going to use websockets at the next summit! :) | 17:09 |
| sdague | well, at least that's the whole story, previously this made invalid API call, it was generating a 404 on the server, which isn't getting bubbled up in shade as an error. With new code we validate before letting it go hit the db, and just tell you it's not possible that this is a valid request via a 400 | 17:11 |
| jeblair | sdague: because 404 in a rest api means "this object does not exist" which is the desired end state anyway. | 17:12 |
| *** e0ne has quit IRC | 17:12 | |
| mordred | right. so a 404 would have been pleasing | 17:13 |
| sdague | right, except it's a 404 for the wrong reasons. The thing does not exist, because it's not possible for that thing to ever exist | 17:13 |
| mordred | right | 17:13 |
| sdague | because you tried to delete the wrong resource | 17:13 |
| jeblair | (and of course it's perfectly natural to get a 404 on an object you thought existed a moment ago because it's an _asynchronous_ api) | 17:13 |
| openstackgerrit | Emma Barber proposed openstack-infra/puppet-hound: Configure and run apache on top of hound https://review.openstack.org/211640 | 17:13 |
| sdague | jeblair: right, except, the 404 happened before the delete | 17:14 |
| *** afazekas__ has quit IRC | 17:14 | |
| sdague | so, unless you have a time machine :) | 17:14 |
| jeblair | sdague: yep. i like the validation. i think it's an improvement; helps clarify this situation | 17:14 |
| mordred | totally | 17:14 |
| mordred | it's good stuff | 17:14 |
| *** glauco has quit IRC | 17:14 | |
| mordred | I MUCH prefer getting an error so I can decide if, in my workflow, an error on the thing is something I expect or something I do not expect | 17:15 |
| *** jtomasek has quit IRC | 17:15 | |
| mordred | sdague: so, just for the record, I'm fine if that bug gets closed EWONTFIX | 17:15 |
| sdague | mordred: ok, cool | 17:16 |
| sdague | anyway, at least we understand now. I just don't like things changing without understanding :) | 17:16 |
| mordred | I do not believe I was counting on the broken behavior anywhere real | 17:16 |
| mordred | and, even if I was, I would prefer to get the new errors and break for a second | 17:16 |
| sdague | yeh | 17:16 |
| *** sridhar_ram has joined #openstack-infra | 17:17 | |
| sdague | it may be a while before you get that new behavior in production clouds. They actually need to make a recommended paste change to get there | 17:17 |
| *** wshao_ has joined #openstack-infra | 17:17 | |
| sdague | though, if you start using the /v2.1 nova endpoint if available, you'll always get the new behavior | 17:17 |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s) https://review.openstack.org/219760 | 17:18 |
| *** trown is now known as trown|lunch | 17:19 | |
| *** marzif has quit IRC | 17:20 | |
| *** shashank_hegde has quit IRC | 17:20 | |
| *** pvaneck has joined #openstack-infra | 17:20 | |
| jeblair | oh, i think i hear the radiation detecting helicopter. | 17:21 |
| openstackgerrit | Anastasia Kuznetsova proposed openstack-infra/project-config: Remove export of 'openstack/rally' to in jobs' configs https://review.openstack.org/219795 | 17:21 |
| *** maishsk has joined #openstack-infra | 17:23 | |
| *** cdent has quit IRC | 17:24 | |
| jroll | sdague: yes, pxe_ssh is working now, pxe_ipa is the future... need to track down the rest and change them over | 17:24 |
| *** achanda has joined #openstack-infra | 17:24 | |
| openstackgerrit | Anastasia Kuznetsova proposed openstack-infra/project-config: Remove export of 'openstack/rally' to PROJECTS list in jobs' configs https://review.openstack.org/219795 | 17:26 |
| *** wshao_ has quit IRC | 17:26 | |
| *** glauco has joined #openstack-infra | 17:27 | |
| dhellmann | I'm seeing a lot of requirements job failures with "ERROR: branch not allowed by features matrix: liberty", does anyone have any idea what that means? I think we turned on some new jobs for the requirements repo today? | 17:28 |
| dhellmann | lifeless, ttx: ^^ | 17:28 |
| *** sharikapongubala has joined #openstack-infra | 17:28 | |
| dhellmann | clarkb: ^^ | 17:29 |
| clarkb | dhellmann: gah yes I know what that means | 17:30 |
| clarkb | dhellmann: patch inconing | 17:30 |
| dhellmann | clarkb: thanks | 17:30 |
| clarkb | we should write this patch as soon as the release name is solid | 17:30 |
| clarkb | hrm looks like we did have it but it was incomplete | 17:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions https://review.openstack.org/183176 | 17:31 |
| mordred | morgan: ^^ I think that'll pass this time | 17:31 |
| morgan | Looking. We did land some changes yesterday re: loading. | 17:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Make liberty actually allowed in features.yaml https://review.openstack.org/219796 | 17:32 |
| sdague | dhellmann: things started branching today? | 17:32 |
| clarkb | dhellmann: ^ you can depends on that to fix or wiat for stuff to merge | 17:32 |
| morgan | mordred: i think we are all but done for 1.0 though. I think the changes shouldnt impact you since you use auth_type | 17:32 |
| clarkb | sdague: no, we added cross branch tests to requirements to prevent master requirements from breaking stable/liberty on other things | 17:32 |
| *** sharikapongubala has quit IRC | 17:32 | |
| clarkb | sdague: and this was a missed piece of d-g config | 17:32 |
| dhellmann | sdague: no, no branches yet | 17:33 |
| *** dims_ has quit IRC | 17:33 | |
| dhellmann | sdague: the new jobs are supposed to ensure we test changes to master requirements against any new stable/liberty branches when they are created | 17:33 |
| sdague | clarkb: ah gotcha | 17:33 |
| sdague | anyway +2 on the d-g patch | 17:33 |
| dhellmann | clarkb, sdague : why do we have a hard-coded list of branch names in there? | 17:33 |
| dhellmann | clarkb: want to go ahead and add mitaka now? | 17:34 |
| jeblair | clarkb, sdague: why does q-vpn get added? | 17:34 |
| mordred | morgan: yeah. I agree | 17:35 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use python-novaclient 2.27.0 https://review.openstack.org/219737 | 17:35 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use os-brick 0.4.0 https://review.openstack.org/219736 | 17:35 |
| *** markvoelker has joined #openstack-infra | 17:35 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use python-heatclient 0.7.0 https://review.openstack.org/219738 | 17:35 |
| *** markvoelker has quit IRC | 17:35 | |
| *** dims has joined #openstack-infra | 17:35 | |
| *** abdelwas has joined #openstack-infra | 17:36 | |
| clarkb | jeblair: because it is part of kilo so maintaining forward compat lists | 17:36 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: use python-ironicclient 0.8.0 https://review.openstack.org/219740 | 17:36 |
| clarkb | dhellmann: because the testing changes depending on branch | 17:36 |
| *** rlandy has quit IRC | 17:36 | |
| *** markvoelker has joined #openstack-infra | 17:36 | |
| dhellmann | clarkb: and I guess we don't want branches of devstack-gate itself? | 17:36 |
| dhellmann | clarkb: anyway, thanks for fixing it! | 17:36 |
| jeblair | clarkb: i guess i don't understand why it's not based on master | 17:36 |
| *** rlandy has joined #openstack-infra | 17:36 | |
| clarkb | jeblair: oh hrm | 17:37 |
| jeblair | dhellmann: since it drives cross-branch testing, having branches itself would be exceedingly complicated | 17:37 |
| armax | hi folks, it seems that change https://review.openstack.org/#/c/219477/ failed because of a switf error, is tehre any chance it can be promoted over all the other neutron changes in the gate? It addresses a race condition | 17:37 |
| clarkb | wait thats so weird | 17:37 |
| clarkb | so we default have vpn in kilo but not liberty or juni | 17:37 |
| dhellmann | jeblair: right, makes sense | 17:37 |
| clarkb | *juno | 17:37 |
| clarkb | jeblair: but I Think you are correct I will update | 17:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Make liberty actually allowed in features.yaml https://review.openstack.org/219796 | 17:38 |
| sdague | jeblair: because it's not in juno | 17:38 |
| clarkb | sdague: but its not part of master either | 17:38 |
| clarkb | sdague: so right now kilo is the only branch doing vpn which is fine I guess, just odd | 17:38 |
| sdague | clarkb: hmmmm ok | 17:39 |
| *** puranamr has quit IRC | 17:39 | |
| *** wshao_ has joined #openstack-infra | 17:39 | |
| sdague | yeh, right, it's a plugin now so configured outside of devstack | 17:39 |
| *** puranamr has joined #openstack-infra | 17:39 | |
| armax | I would be grateful, and so would be the gate :) | 17:40 |
| *** angdraug has quit IRC | 17:40 | |
| sdague | clarkb: +A | 17:40 |
| *** ashleighfarnham has quit IRC | 17:40 | |
| *** jsavak has quit IRC | 17:41 | |
| *** yamahata has quit IRC | 17:44 | |
| *** ashleighfarnham has joined #openstack-infra | 17:45 | |
| *** pelix has quit IRC | 17:45 | |
| *** shashank_hegde has joined #openstack-infra | 17:45 | |
| openstackgerrit | Merged openstack-infra/shade: Delete floating ip by ID instead of name https://review.openstack.org/219725 | 17:45 |
| asselin | are infra nodepool images updating ok? | 17:46 |
| asselin | ours are 5 days old | 17:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Don't run swift on ironic jobs https://review.openstack.org/219801 | 17:46 |
| *** Swami_ has joined #openstack-infra | 17:48 | |
| *** Swami has joined #openstack-infra | 17:48 | |
| asselin | ours keep failing here: http://paste.openstack.org/show/441848/ | 17:48 |
| greghaynes | crinkle: SpamapS Question about the new bm deploy host im spinning up - did we decide that we arent going to use puppet to install bifrost for now because of the need to install from source? (I think I remember that convo but not 100%) | 17:48 |
| *** ybathia has joined #openstack-infra | 17:48 | |
| SpamapS | greghaynes: no we didn't decide that | 17:48 |
| SpamapS | greghaynes: puppet pulls it fromg it, no? | 17:49 |
| * greghaynes looks @ crinkle | 17:49 | |
| *** rockyg has joined #openstack-infra | 17:49 | |
| greghaynes | I thought it was pretty well hard coded to only use packages | 17:49 |
| *** sharikapongubala has joined #openstack-infra | 17:49 | |
| greghaynes | but crinkle im sure will school us ;) | 17:49 |
| *** dkranz has quit IRC | 17:50 | |
| *** jsavak has joined #openstack-infra | 17:50 | |
| clarkb | asselin: thats a known issue with dib/ubuntu repos iirc | 17:50 |
| asselin | clarkb, is there a workaround? | 17:51 |
| clarkb | asselin: the dib devs may be able to confirm but it was being worked last I saw | 17:51 |
| mordred | asselin: greghaynes ^^ | 17:51 |
| * greghaynes looks | 17:51 | |
| greghaynes | asselin: the fix for that merged I think | 17:52 |
| greghaynes | asselin: it was up yesterdy | 17:52 |
| asselin | greghaynes, ok cool. do you have a link? | 17:52 |
| crinkle | greghaynes: puppet pulls it from source | 17:53 |
| greghaynes | asselin: looking | 17:53 |
| fungi | okay, back and reading the ~300 lines of lunchtime scrollback | 17:53 |
| clarkb | sdague: http://logs.openstack.org/77/219477/5/check/gate-grenade-dsvm-neutron/742745e/logs/grenade.sh.txt.gz#_2015-09-02_16_44_59_479 is the thing armax's change ran into above. I would be surprised if a neutron change affected that file being used as a block device | 17:53 |
| crinkle | greghaynes: http://git.openstack.org/cgit/openstack/puppet-ironic/tree/manifests/bifrost.pp#n193 is how it does it | 17:53 |
| clarkb | armax: have you filed a bug for that at least? | 17:53 |
| armax | clarkb: not yet...I will | 17:54 |
| sdague | man, haven't seen that one in a while | 17:54 |
| armax | clarkb: I have other fires to put out | 17:54 |
| clarkb | armax: also the gate has a large series of things that appear to be passing currently so I will hold off on promoting that change until we get to a reset | 17:54 |
| greghaynes | where did that dib patch go... | 17:54 |
| armax | clarkb: ok sure | 17:54 |
| sdague | clarkb: which change is that ? | 17:55 |
| greghaynes | asselin: oh, the patch was abandoned because apparently upstream (ubuntu) fixed it | 17:55 |
| clarkb | sdague: 219477 | 17:55 |
| greghaynes | asselin: https://review.openstack.org/#/c/219199/ | 17:55 |
| greghaynes | crinkle: awesome, do you have any time to help me get set up with that module? | 17:56 |
| crinkle | greghaynes: sure, we can do that today | 17:56 |
| greghaynes | asselin: If that isnt the case please let me know and we can resurrect that patch | 17:56 |
| greghaynes | crinkle: w00t | 17:56 |
| sdague | clarkb: ok, gotcha | 17:56 |
| *** armax has quit IRC | 17:57 | |
| *** dzamboni has joined #openstack-infra | 17:57 | |
| asselin | greghaynes, so what do I need to update in my system? or just rerun the image build? | 17:58 |
| *** HeOS has quit IRC | 17:58 | |
| greghaynes | asselin: just rebuild | 17:58 |
| asselin | greghaynes, ok thanks | 17:58 |
| greghaynes | asselin: assuming you arent caching that file somewhere | 17:58 |
| *** mpopow__ has joined #openstack-infra | 17:58 | |
| greghaynes | (the sha256sums file) | 17:58 |
| asselin | greghaynes, I'm caching whater dib's caching | 17:59 |
| sdague | clarkb: it is interesting that - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIndyb25nIGZzIHR5cGUsIGJhZCBvcHRpb24sIGJhZCBzdXBlcmJsb2NrIG9uIC9kZXYvXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDEyMTY3MzM3NjR9 | 17:59 |
| greghaynes | asselin: ok, dib should invalidate that fine | 17:59 |
| asselin | cool | 17:59 |
| sdague | superblock fail is only on neutron grenade jobs | 17:59 |
| *** maishsk has quit IRC | 17:59 | |
| clarkb | sdague: that is odd | 18:00 |
| sdague | Sep 2 16:44:59 devstack-trusty-hpcloud-b4-4720563 kernel: [ 2466.205514] XFS (loop2): Filesystem has duplicate UUID c30d9726-bab6-4c4e-9952-16577f49d514 - can't mount | 18:00 |
| *** kmartin_ has joined #openstack-infra | 18:02 | |
| *** madong_ has joined #openstack-infra | 18:02 | |
| *** Swami__ has joined #openstack-infra | 18:02 | |
| *** kmartin has quit IRC | 18:03 | |
| *** mdenny has quit IRC | 18:03 | |
| *** MIDENN_ has quit IRC | 18:03 | |
| *** Swami_ has quit IRC | 18:03 | |
| *** Swami has quit IRC | 18:03 | |
| *** larainema has quit IRC | 18:03 | |
| *** mdenny has joined #openstack-infra | 18:03 | |
| *** MIDENN_ has joined #openstack-infra | 18:03 | |
| *** Swami has joined #openstack-infra | 18:03 | |
| *** sridhar_ram has quit IRC | 18:04 | |
| *** jsavak has quit IRC | 18:04 | |
| *** dkranz has joined #openstack-infra | 18:05 | |
| *** jsavak has joined #openstack-infra | 18:05 | |
| *** maishsk has joined #openstack-infra | 18:05 | |
| *** markvoelker has quit IRC | 18:05 | |
| sdague | clarkb: apparently it's just a thing that xfs does from time to time - https://bugs.launchpad.net/diskimage-builder/+bug/1443706 | 18:06 |
| openstack | Launchpad bug 1443706 in diskimage-builder "Failed to mount: Filesystem has duplicate UUID" [Medium,Fix released] - Assigned to Steve Baker (steve-stevebaker) | 18:06 |
| sdague | disk image builder even put in a work around | 18:07 |
| sdague | so once someone files a bug on the issue, I'll put the work around into grenade and devstack | 18:07 |
| fungi | weird. wonder if there's some birthday paradox problem with the uuid generation there | 18:08 |
| anteaya | too much cake? | 18:08 |
| fungi | like maybe they're not doing a great job of picking an initial seed | 18:08 |
| fungi | seed cake! | 18:08 |
| fungi | mmm | 18:08 |
| clarkb | pickle the initial seed cakes | 18:09 |
| anteaya | pickled cakes? | 18:09 |
| *** marzif has joined #openstack-infra | 18:09 | |
| anteaya | I keep kicking pabelanger | 18:09 |
| anteaya | I have shoes and he has flipflops | 18:09 |
| slagle | when we saw that in dib it was b/c we were running an instance of the cloud image which was trying to build a dib image using that same cloud image | 18:10 |
| *** Sukhdev_ has joined #openstack-infra | 18:11 | |
| anteaya | sdague: did someone say they were going to file a bug on that? | 18:12 |
| anteaya | armax has left irc | 18:12 |
| anteaya | I mean I can if we want me to | 18:12 |
| *** sthillma has joined #openstack-infra | 18:12 | |
| *** mpopow__ has quit IRC | 18:13 | |
| anteaya | oh armax said he will | 18:13 |
| anteaya | after he deals with other fires | 18:13 |
| *** sdake_ has joined #openstack-infra | 18:13 | |
| *** rockyg has quit IRC | 18:16 | |
| *** sdake has quit IRC | 18:17 | |
| *** maishsk has quit IRC | 18:17 | |
| *** sthillma_ has joined #openstack-infra | 18:18 | |
| *** amitgandhinz has joined #openstack-infra | 18:18 | |
| *** sthillma has quit IRC | 18:18 | |
| *** sthillma_ is now known as sthillma | 18:18 | |
| *** annegentle has joined #openstack-infra | 18:20 | |
| clarkb | I am going to enqueue 219477 into the gate now then promote it when the current head of the gate list is processed, we actually did merge a few thngs | 18:20 |
| *** sdake has joined #openstack-infra | 18:21 | |
| anteaya | yay merged things | 18:21 |
| *** maishsk has joined #openstack-infra | 18:22 | |
| fungi | thanks clarkb | 18:22 |
| *** julim has quit IRC | 18:22 | |
| anteaya | I thought I saw about 8 or so pass at the head of the gate | 18:22 |
| *** marzif has quit IRC | 18:22 | |
| *** tonytan4ever has joined #openstack-infra | 18:22 | |
| *** marzif has joined #openstack-infra | 18:22 | |
| clarkb | anteaya: yup and there wee a few before this batch | 18:22 |
| anteaya | yay for things working | 18:22 |
| clarkb | this is why I didn't immediately promote it wanted to give them a chance since they were all passing | 18:22 |
| anteaya | good call | 18:23 |
| clarkb | when thats done my next thing will be deleting old git07 and making a new git06 | 18:24 |
| clarkb | gonna try to pick up the pace on that so we can go back to leastconn balancing | 18:25 |
| *** sdake_ has quit IRC | 18:25 | |
| *** sridhar_ram has joined #openstack-infra | 18:25 | |
| *** ybathia has quit IRC | 18:26 | |
| *** sharikapongubala has quit IRC | 18:27 | |
| *** puranamr has quit IRC | 18:27 | |
| clarkb | fungi: did you see that jeblair got websockets owrking on etherpad-dev? | 18:27 |
| clarkb | fungi: so we can enable them on prod server when we rebuild too :) | 18:27 |
| fungi | clarkb: yep, commented on the review | 18:27 |
| mordred | yay! the shade changes are merging again | 18:28 |
| * fungi is excited | 18:28 | |
| *** puranamr has joined #openstack-infra | 18:28 | |
| clarkb | jeblair: fungi has a good point about precise, I am not sure that mod_wstunnel exists on precise so it may just explode | 18:28 |
| *** sharikapongubala has joined #openstack-infra | 18:28 | |
| *** marzif has quit IRC | 18:28 | |
| clarkb | derp its already wrapped | 18:29 |
| fungi | ahh, yep | 18:29 |
| clarkb | fungi: so that should be fine on precise | 18:29 |
| clarkb | +2 from me | 18:29 |
| fungi | i missed it the first time through as well, forgot we branched on ubuntu release | 18:29 |
| *** yfried has joined #openstack-infra | 18:29 | |
| clarkb | ya the nested block isn't super clear in the diff | 18:30 |
| fungi | and the vhost config is conditional on the module existing | 18:30 |
| fungi | so lgtm | 18:30 |
| *** trown|lunch is now known as trown | 18:30 | |
| anteaya | clarkb: how many more git servers to upgrade to centos7? | 18:30 |
| docaedo | yolanda: have you had a chance to look at the changes to https://review.openstack.org/#/c/218898/ yet? | 18:31 |
| fungi | anteaya: 6 | 18:31 |
| anteaya | fungi: thanks | 18:31 |
| *** armax has joined #openstack-infra | 18:31 | |
| anteaya | docaedo: I'm guessing she is probably offline until tomorrow | 18:31 |
| fungi | clarkb: any objection to me blowing away the old etherpad-dev now? | 18:31 |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s) https://review.openstack.org/219760 | 18:31 |
| docaedo | anteaya: ah thank you | 18:32 |
| clarkb | fungi nope | 18:32 |
| fungi | clarkb: you may need the extra bit of headroom to build replacement git servers since we're close to quota | 18:32 |
| clarkb | anteaya 6 | 18:32 |
| anteaya | docaedo: welcome, she is in Europe | 18:32 |
| anteaya | clarkb: thanks | 18:32 |
| clarkb | fungi +1 | 18:32 |
| *** kdas_ has joined #openstack-infra | 18:32 | |
| *** sthillma_ has joined #openstack-infra | 18:33 | |
| docaedo | anteaya: perhaps you would like to take a look at https://review.openstack.org/#/c/218898/ for us? we would sure like it merged if it's good to go :) | 18:33 |
| fungi | also puppetmaster is ssh'ing into new etherpad-dev successfully as of a little while ago | 18:33 |
| anteaya | clarkb fungi speed for taking a git server out of rotation, upgrading and putthing it back in is about one per day? | 18:33 |
| fungi | anteaya: can be much faster if we want | 18:33 |
| anteaya | docaedo: I don't have +2 on that repo | 18:33 |
| *** ybathia has joined #openstack-infra | 18:33 | |
| clarkb | anteaya no I think I can do the whole set today if I dont get too distracted | 18:33 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Switch Rally gates to new DevStack plugin style https://review.openstack.org/219823 | 18:33 |
| fungi | anteaya: we were just starting slow in case we had to roll back, but once we're in for a penny... | 18:34 |
| * anteaya clears space for clarkb | 18:34 | |
| anteaya | fungi: /me nods | 18:34 |
| boris-42 | fungi: hi could you please take a look at that patch ^ | 18:34 |
| *** esker has joined #openstack-infra | 18:34 | |
| boris-42 | fungi: rally gates are terrible broken (they just test master) | 18:34 |
| clarkb | ok that stack of gate changes merged, I am promoting now | 18:34 |
| anteaya | clarkb: yay | 18:34 |
| docaedo | anteaya: no problem - if you happen to know anyone who might feel like reviewing it though, please let me know (hint hint fungi clarkb crinkle ) | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| *** esker has quit IRC | 18:35 | |
| *** sthillma has quit IRC | 18:35 | |
| *** sthillma_ is now known as sthillma | 18:35 | |
| anteaya | docaedo: clarkb is upgrading git servers today | 18:35 |
| anteaya | and crinkle is in a puppet sprint | 18:35 |
| anteaya | but hopefully someone will review soon | 18:35 |
| clarkb | I had forgotten how this works | 18:35 |
| docaedo | anteaya: well, I think all things considered, my app catalog patch is probably much more important than that, right? | 18:35 |
| clarkb | it actually clears out the queue then redoes it right? | 18:35 |
| clarkb | fungi: ^ | 18:35 |
| *** esker has joined #openstack-infra | 18:35 | |
| anteaya | docaedo: I have no idea how important your app catalog patch is | 18:36 |
| fungi | clarkb: it cancels running jobs and reorders the queue | 18:36 |
| docaedo | hahaha | 18:36 |
| anteaya | and won't speak for others and how they manage my time | 18:36 |
| fungi | clarkb: so in a manner of speaking, i guess | 18:36 |
| clarkb | fungi: and status.json shows them go out of queue but will redo it ya? | 18:36 |
| fungi | clarkb: correct | 18:36 |
| anteaya | except that I will be clearing space for clarkb | 18:36 |
| clarkb | ah yup everything is back again | 18:36 |
| docaedo | anteaya: I'm just kidding, the stuff clarkb is working on is more important by a long shot at the moment | 18:36 |
| anteaya | as having git servers running different oss has problems so it is best if we complete the work as soon as we can | 18:36 |
| anteaya | docaedo: oh okay | 18:37 |
| anteaya | thanks for understanding | 18:37 |
| fungi | docaedo: anteaya: yep, with puppet mid-cycle sprint going on, we're down a lot of our usual puppet-centric backup teammates | 18:37 |
| anteaya | how they manage their time | 18:37 |
| rcarrillocruz | jeblair: sweet, was working that on my old vm, so probably a new thing on latest socketio versions. In regards to the wstunnel module, i *think* i didn't put cos by the time the servers of system-config were on precise/2.2, which does not come with that module | 18:38 |
| rcarrillocruz | so i left it for a later commit when it was upgraded | 18:38 |
| rcarrillocruz | but now that i'm thinking i could've probably installed it conditionally per distrelease | 18:38 |
| fungi | rcarrillocruz: which is what the manifest is doing now | 18:39 |
| rcarrillocruz | happy to see etherpad on websockets btw, it performed better for my on my tests with websockets enabled | 18:39 |
| fungi | at least as soon as the change merges in a few moments | 18:39 |
| rcarrillocruz | yeah, just saw the comment from jeblalir and the change, is already approved | 18:40 |
| *** burgerk has quit IRC | 18:40 | |
| fungi | and as soon as we upgrade our production server we'll be using it there | 18:40 |
| j^2 | I’m starting to build tho OpenStack Operators launchpad, i accidently gave OpenStack Administrators Maintainer rights and removed my own for the set up, can someone help me out here? | 18:40 |
| rcarrillocruz | ++ | 18:40 |
| fungi | j^2: probably. what's your lp username? | 18:41 |
| j^2 | fungi: d-jj | 18:41 |
| fungi | j^2: and what's the project shortname (the part in the url)? | 18:41 |
| j^2 | osops | 18:41 |
| fungi | thanks, looking now | 18:41 |
| *** sdake_ has joined #openstack-infra | 18:41 | |
| j^2 | rockon thanks fungi | 18:41 |
| slagle | if anyone has a spare moment, tripleo is looking for reviews on https://review.openstack.org/#/c/215186/ | 18:41 |
| *** ihrachys has joined #openstack-infra | 18:42 | |
| *** jsavak has quit IRC | 18:42 | |
| *** e0ne has joined #openstack-infra | 18:42 | |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: Fix websocket configuration https://review.openstack.org/219790 | 18:42 |
| fungi | j^2: i've switched it back to you for now so you can finish setting what you need | 18:43 |
| j^2 | fungi: perfect, thanks again. | 18:43 |
| * clarkb updates schedule and eats food before digging into server replacements | 18:43 | |
| *** jsavak has joined #openstack-infra | 18:43 | |
| openstackgerrit | Merged openstack-infra/shade: Pass timeout to session, not constructors https://review.openstack.org/219290 | 18:43 |
| *** sdake has quit IRC | 18:45 | |
| fungi | clarkb: huh... nova list in our openstackci tenant on rax is suddenly giving me an unauthorized error from the puppetmaster | 18:45 |
| clarkb | fungi: let me check | 18:45 |
| rcarrillocruz | fungi: when you get a sec, could you have a look at https://review.openstack.org/#/c/219181/ and https://review.openstack.org/#/c/219185/ ? infra-ansible is not passing tests cos I set it up with the ansible job template, but that's only meant for ansible roles | 18:45 |
| fungi | creds file hasn't been touched since may | 18:46 |
| clarkb | can confirm | 18:46 |
| clarkb | maybe it really is time for lunch :) | 18:46 |
| fungi | yep, get to it | 18:47 |
| *** austin81 has joined #openstack-infra | 18:47 | |
| fungi | i can log in via the webui at least | 18:47 |
| fungi | oh yeah, looks like our node count has gone to pot too. same problem with openstackjenkins | 18:48 |
| fungi | i suspect rackspace just tanked their caystone api | 18:49 |
| anteaya | :( | 18:50 |
| *** btully has joined #openstack-infra | 18:52 | |
| nibalizer | https://status.rackspace.com/ seem that they don't think they have an issue | 18:52 |
| fungi | yeah, and while they've got a migration maintenance going on today for dfw and a (supposedly solved) issue for ord, i'm even seeing it in iad | 18:52 |
| *** olaph has quit IRC | 18:53 | |
| *** kdas_ is now known as kushal | 18:53 | |
| *** kushal has quit IRC | 18:53 | |
| *** kushal has joined #openstack-infra | 18:53 | |
| *** harlowja has quit IRC | 18:54 | |
| *** olaph has joined #openstack-infra | 18:55 | |
| *** julim has joined #openstack-infra | 18:56 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 18:58 |
| *** harlowja has joined #openstack-infra | 18:58 | |
| *** Rockyg has joined #openstack-infra | 18:58 | |
| fungi | huh, i don't find myself poking around the rax dashboard except when their api goes kaput... looks like they started doing ping monitoring by default? any of the instances we've launched in the past ~month have it turned on, but the rest do not | 18:59 |
| *** cdent has joined #openstack-infra | 19:00 | |
| *** markvoelker has joined #openstack-infra | 19:00 | |
| *** dkranz has quit IRC | 19:00 | |
| fungi | oh! i see what it is | 19:00 |
| * anteaya wants to know what it is | 19:00 | |
| *** HeOS has joined #openstack-infra | 19:01 | |
| fungi | no, wait, i don't | 19:01 |
| anteaya | :( | 19:01 |
| mordred | anteaya: it is what it is | 19:01 |
| fungi | it's what i thought at first. they really do have ping checks going on the newest instances we've booted | 19:02 |
| anteaya | so I'm told | 19:02 |
| *** bcourt has quit IRC | 19:02 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Add announcements for keystoneauth to #openstack-sdks https://review.openstack.org/219842 | 19:03 |
| *** mrunge has quit IRC | 19:03 | |
| *** julim_ has joined #openstack-infra | 19:03 | |
| fungi | jroll: you seem to have your finger on the pulse on all things rackspace. any idea why all of our tenants suddenly (in the past 20-30 minutes) started getting authorization errors from the api for all regions? | 19:04 |
| fungi | nothing on the status page seems to correspond | 19:04 |
| *** julim has quit IRC | 19:06 | |
| clarkb | fungi: nodepool must be affected? | 19:06 |
| fungi | clarkb: i expect so, the graph looked pretty wonky | 19:06 |
| *** coolsvap_ has joined #openstack-infra | 19:09 | |
| *** hyakuhei has joined #openstack-infra | 19:10 | |
| *** SotK has joined #openstack-infra | 19:10 | |
| jeblair | fungi: nodepool _seems_ to be talking to rax still | 19:11 |
| fungi | yeah, maybe it's already authenticated | 19:11 |
| jeblair | that could be due to an existing session | 19:11 |
| *** beboj has joined #openstack-infra | 19:11 | |
| fungi | i agree tailing the debug log it seems working | 19:11 |
| hyakuhei | Any docs people around? | 19:11 |
| jeblair | fungi: 'nodepool alien-list rax-dfw' works | 19:11 |
| jeblair | fungi: that's suspicious | 19:12 |
| fungi | huh, weird. i wonder if something's different about the puppetmaster | 19:12 |
| jeblair | that should be entirely new sessions, etc. | 19:12 |
| *** e0ne has quit IRC | 19:12 | |
| fungi | maybe one of the new *client releases is tripping us up? | 19:12 |
| fungi | there were several while i was at lunch, apparently | 19:12 |
| *** mrmartin has quit IRC | 19:13 | |
| *** dkranz has joined #openstack-infra | 19:13 | |
| mordred | fungi: did we restart nodepool? | 19:13 |
| fungi | mordred: nope | 19:13 |
| fungi | though the cli should be authenticating fresh anyway | 19:13 |
| mordred | ok. so none of the most recent shade changes should be to blame | 19:13 |
| mordred | yeah | 19:13 |
| jeblair | mordred: nodepool is the thing working | 19:14 |
| mordred | ah | 19:14 |
| jeblair | mordred: python-novaclient on puppetmaster is not | 19:14 |
| fungi | yeah, where we're seeing it is novaclient on the puppetmaster | 19:14 |
| mordred | AH | 19:14 |
| mordred | sorry | 19:14 |
| mordred | hadn't read the entire scrollback | 19:14 |
| fungi | looks like we got 2.27.0 | 19:14 |
| mordred | first panic is "I just landed things to how we call client constructors in shade" | 19:14 |
| *** shashank_hegde has quit IRC | 19:14 | |
| fungi | that's today's release | 19:14 |
| fungi | smart money's on latest novaclient wanting different envvars | 19:15 |
| fungi | (again!) | 19:15 |
| jeblair | fungi: 0.26 works 0.27 fails | 19:15 |
| fungi | i just confirmed the same | 19:15 |
| anteaya | skylerberg: you don't need a spec to put your code in openstack/ | 19:15 |
| mordred | jeblair: I concur | 19:15 |
| mordred | this is the same behavior locally on my laptop | 19:15 |
| anteaya | skylerberg: just create a patch to project-config to create the repo and add your code to it | 19:15 |
| jeblair | but that's unpossible with semver! | 19:16 |
| fungi | checking release notes now | 19:16 |
| *** bcourt has joined #openstack-infra | 19:17 | |
| * mordred is comparing http debug | 19:17 | |
| *** annegentle has quit IRC | 19:17 | |
| *** sharikapongubala has quit IRC | 19:18 | |
| mordred | the HTTP transactions for "list" are definitely different | 19:19 |
| mordred | so - in the working version, novaclient gets the token, then does a POST to /v2.0/tokens | 19:20 |
| mordred | then does some iso8601 processing | 19:20 |
| mordred | then does a GET of /v2/{project_id}/servers/detail | 19:21 |
| *** beboj has left #openstack-infra | 19:21 | |
| mordred | in 2.27.0 it does the post to /v2.0/tokens | 19:21 |
| mordred | it then does a get of /v2 | 19:21 |
| mordred | which gets a 401 | 19:21 |
| mordred | it then does another post and another get | 19:22 |
| mordred | which then propagates the 401 | 19:22 |
| mordred | so -I'm not sure what it's trying to accomplish with the GET of /v2 | 19:22 |
| mordred | but it clearly is not working | 19:22 |
| mordred | sdague, jroll: ^^ | 19:22 |
| *** bharathm has quit IRC | 19:23 | |
| sdague | mordred: you got example code and the failure? | 19:23 |
| mordred | sdague: yeah | 19:23 |
| mordred | it's "nova list" on the command line with 2.27.0 | 19:23 |
| mordred | sdague: http://paste.openstack.org/show/442203/ <-- 2.26.0 against Rackspace | 19:24 |
| mordred | sdague: http://paste.openstack.org/show/442204/ <-- 2.27.0 against Rackspace | 19:25 |
| clarkb | iirc rax does more precision on timestamps than keystone does | 19:25 |
| mordred | same creds | 19:25 |
| clarkb | if iso8601 is related that may be part of it | 19:25 |
| mordred | 2.27.0 doesn't seem to be doing the timestamp processing | 19:25 |
| mordred | at all | 19:25 |
| mordred | or - it might be - and we might have stopped logging iso8601 | 19:25 |
| *** BharatK has quit IRC | 19:26 | |
| sdague | probably stopped logging it | 19:26 |
| fungi | i'm testing against hpcloud for comparison, but there it seems to probably be timing out | 19:27 |
| sdague | yeh, it's the version discovery going arry | 19:27 |
| sdague | mordred: what's the env (minus secrets) that's set? | 19:28 |
| mordred | sdague: one sec | 19:28 |
| fungi | i bet this is the best novaclient yet for use with public clouds that run devstack ;) | 19:28 |
| mordred | sdague: http://paste.openstack.org/show/442205/ | 19:29 |
| mordred | sdague: also OS_PASSWORD is set | 19:29 |
| sdague | man paste.o.o is so slow | 19:29 |
| *** puranamr has quit IRC | 19:29 | |
| sdague | mordred: what's the service catalog look like for rax? | 19:30 |
| mtreinish | mordred: without OS_PASSWORD in that paste, how am I supposed to know whether you used the right creds? :) | 19:30 |
| mordred | sdague: one sec | 19:30 |
| mordred | mtreinish: :) | 19:30 |
| *** wshao_ has quit IRC | 19:30 | |
| *** puranamr has joined #openstack-infra | 19:33 | |
| mordred | sdague: http://paste.openstack.org/show/442206/ | 19:34 |
| *** annegentle has joined #openstack-infra | 19:35 | |
| sdague | yeh, well their service catalog is wrong .... | 19:35 |
| sdague | sigh, ok, pondering the right work around | 19:35 |
| sdague | they've explicitly said "versionInfo | https://dfw.servers.api.rackspacecloud.com/v2 " | 19:36 |
| sdague | is where you should get the versionInfo | 19:36 |
| mordred | yes. they have said that. | 19:36 |
| mordred | it does not work | 19:36 |
| fungi | huh, i think there may be a mass mailer bouncing their execution through http://paste.openstack.org/show/437782/ to avoid locally installing their code. i see hundreds of get requests a second from a single ip address for that page | 19:36 |
| *** austin81 has left #openstack-infra | 19:37 | |
| mordred | fungi: we should edit the code on that paste | 19:37 |
| fungi | that would be hilarious. set the recipient to the fbi crime reporting address | 19:38 |
| mordred | ++ | 19:38 |
| sdague | mordred: can you do a favor for me and live hack a thing | 19:38 |
| lifeless | Kennan: I don't know, sorry. | 19:38 |
| *** austin81 has joined #openstack-infra | 19:38 | |
| sdague | mordred: just print out the value of url there - https://github.com/openstack/python-novaclient/blob/893bf9313c2efb78d30fc5df3b67313eec154caa/novaclient/v2/versions.py#L43 | 19:38 |
| lifeless | dhellmann: I see clarkb was on it. clarkb for my edification, where does that patch go ? | 19:38 |
| sdague | that's where things go wrong for you | 19:38 |
| *** amitgandhinz has quit IRC | 19:38 | |
| sdague | I no longer have rax creds, so I can't test this thing | 19:39 |
| fungi | lifeless: i answered Kennan's question back around 12:19 utc | 19:39 |
| fungi | bad tox.ini | 19:39 |
| mordred | sdague: sure. one sec | 19:39 |
| jeblair | fungi: https://review.openstack.org/gitweb?p=openstack%2Fzaqar.git;a=commitdiff;h=e9dd687983dfc8dcab0f8413dcedd3294c9143f9 | 19:39 |
| sdague | mordred: I guess I'd also love the value of get_endpoint raw before the splitting | 19:40 |
| fungi | jeblair: hah, did you use hound to find that? | 19:40 |
| jeblair | fungi: no :( google | 19:40 |
| fungi | and yeah, the offending source address seems to be on telstra broadband in nz, so fbi probably isn't the best option to crank whoever that is | 19:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add db migration to align column size to 255 https://review.openstack.org/203251 | 19:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Compact existing schema and start 1.x.x release branch https://review.openstack.org/203252 | 19:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: SUPER WIP: Stop using oslo anything https://review.openstack.org/203308 | 19:42 |
| mordred | sdague: ('get_current:endpoint:', u'https://dfw.servers.api.rackspacecloud.com/v2/956868') | 19:42 |
| fungi | but i am going to drop that ip address in our iptables penalty box for now | 19:42 |
| mordred | sdague: and I get the error on line 43 | 19:42 |
| clarkb | lifeless context? | 19:44 |
| *** gyee has quit IRC | 19:44 | |
| *** coolsvap_ has quit IRC | 19:45 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:45 | |
| *** shashank_hegde has joined #openstack-infra | 19:47 | |
| *** esker has quit IRC | 19:48 | |
| *** bpoulos has left #openstack-infra | 19:49 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: SUPER WIP: Stop using oslo anything https://review.openstack.org/203308 | 19:50 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Compact existing schema and start 1.x.x release branch https://review.openstack.org/203252 | 19:50 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add db migration to align column size to 255 https://review.openstack.org/203251 | 19:50 |
| *** ildikov has quit IRC | 19:51 | |
| SumitNaiksatam | hi, we are seeing an issue with our gerrit patches from a stackforge project not entering the merge queue | 19:52 |
| fungi | SumitNaiksatam: let's see an example and i should be able to give you the cause | 19:52 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: First stab at making things bifrosty https://review.openstack.org/208751 | 19:52 |
| SumitNaiksatam | fungi: https://review.openstack.org/#/c/219397/ | 19:52 |
| SumitNaiksatam | fungi: thanks, the last patch before the merged roughly a day back | 19:53 |
| fungi | SumitNaiksatam: its parent commit isn't approved, and isn't even passing tests | 19:53 |
| *** b10n1k_ has joined #openstack-infra | 19:53 | |
| *** hdd has quit IRC | 19:53 | |
| fungi | SumitNaiksatam: see the linked change in the dependencies box there in that page | 19:53 |
| SumitNaiksatam | fungi: ah bummer, did notice the dependency issue, sorry about the false alarm! | 19:53 |
| sdague | mordred: so on actual nova you can get the top of the version URL and get information about the service | 19:53 |
| SumitNaiksatam | *didnt | 19:53 |
| SumitNaiksatam | fungi: thanks! | 19:54 |
| fungi | SumitNaiksatam: you're welcome | 19:54 |
| mordred | sdague: yah. which is cool | 19:54 |
| sdague | http://paste.openstack.org/show/442210/ | 19:54 |
| sdague | we clearly did not account for RAX's not actually nova | 19:54 |
| mordred | sdague: how silly of you | 19:54 |
| *** hdd has joined #openstack-infra | 19:54 | |
| mordred | sdague: perhaps trapping for a 401 on the version list and defaulting to something old and dumb? | 19:55 |
| sdague | yeh, so clearly we need a work around | 19:55 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add bifrost inventory management procedures https://review.openstack.org/214279 | 19:55 |
| sdague | there is a nova meeting in the morning, we'll huddle up there, if you can pin back temporarily | 19:56 |
| *** pc_m has quit IRC | 19:56 | |
| *** julim_ has quit IRC | 19:56 | |
| mordred | sdague: we can - we're fine - just mainly wanted to make sure that nova was aware | 19:56 |
| sdague | yep, thanks much | 19:56 |
| *** ZZelle_ has joined #openstack-infra | 19:57 | |
| lifeless | clarkb: the matrix thing | 19:57 |
| clarkb | lifeless: oh, thats a file in devstack-gate that determines what services we enable when testing specific branches | 19:58 |
| clarkb | lifeless: so I updated features.yaml in devstack-gate to allow liberty as a valid branch | 19:58 |
| clarkb | sdague: would this affect any nova too old to have microversions or too old to have that functionality? | 19:59 |
| lifeless | clarkb: thanks | 19:59 |
| clarkb | sdague: or is rax extra special here ? | 19:59 |
| sdague | no, this is part of v2.0 for ever | 19:59 |
| *** puranamr_ has joined #openstack-infra | 20:00 | |
| *** puranamr has quit IRC | 20:00 | |
| *** erhudy has joined #openstack-infra | 20:00 | |
| *** shardy has quit IRC | 20:00 | |
| *** bharathm has joined #openstack-infra | 20:01 | |
| jroll | mordred: fungi so tl;dr latest novaclient is broken with our keystone? or did I skim too quick | 20:01 |
| clarkb | old git07 is deleting now | 20:02 |
| clarkb | going to boot two new ones one each for 06 and 05 if quota will let me | 20:02 |
| fungi | jroll: possibly broken with all non-devstack clouds. hard to tell just yet | 20:02 |
| sdague | fungi: no, just with rax | 20:02 |
| sdague | it's not even the nova code, it's something in their repose config probably | 20:03 |
| fungi | sdague: oh, interesting that it's timing out with hpcloud then | 20:03 |
| *** geoffarnold is now known as geoffarnoldX | 20:03 | |
| *** bharathm has quit IRC | 20:03 | |
| fungi | ERROR (ConnectionRefused): Unable to establish connection to http://region-b.geo-1.compute-internal.hpcloudsvc.com/v2/ | 20:03 |
| *** bharathm has joined #openstack-infra | 20:03 | |
| fungi | i guess that could be unrelated | 20:03 |
| sdague | yeh, I hope so. This really should exist everywhere | 20:04 |
| *** amitgandhinz has joined #openstack-infra | 20:04 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: SUPER WIP: Stop using oslo anything https://review.openstack.org/203308 | 20:04 |
| fungi | that's what i get after several minutes of nova list hanging at the console | 20:04 |
| *** ildikov has joined #openstack-infra | 20:04 | |
| jroll | fungi: oh fun, let me know if I can help :) | 20:05 |
| mordred | jroll: yes | 20:05 |
| mordred | sdague: I believe HP also does not allow access to that endpoint | 20:06 |
| *** Sree has quit IRC | 20:06 | |
| jeblair | fungi, sdague: i agree, 2.26 works with hpcloud and 2.27 does not. | 20:06 |
| *** baoli has quit IRC | 20:06 | |
| sdague | o_O | 20:06 |
| mordred | so - it's broken with RAX and HP | 20:06 |
| *** glauco has quit IRC | 20:06 | |
| *** gyee has joined #openstack-infra | 20:06 | |
| mordred | sdague: /v2 raw hadn't been used by any tools for anything - so I imagine it's not opt-in'd to the apigee or lbs or whatever | 20:06 |
| sdague | ffs people, we put code into the project, and then you block it by proxy ? | 20:06 |
| mordred | yeah | 20:06 |
| fungi | what are the odds of it being broken _only_ on rackspace and hpcloud but working for other major service providers? | 20:07 |
| mordred | that is the predominant mode of deployment | 20:07 |
| mordred | fungi: wnat me to check? | 20:07 |
| sdague | yes please | 20:07 |
| * mordred has all the accounts | 20:07 | |
| fungi | mordred: i don't have an easy way to check, so it was mostly rhetorical | 20:07 |
| fungi | but if you can make it non-rhetorical, all the better | 20:07 |
| mordred | dreamhost is hanging | 20:07 |
| anteaya | :( | 20:07 |
| jeblair | i feel like having a non-theoretical answer will help in future keynote rants. | 20:07 |
| mordred | vexxhost works, as expected | 20:07 |
| mordred | auro I get 401 | 20:08 |
| sdague | jeblair: I am all for non theoretical answers | 20:08 |
| fungi | four... four clouds! hahaha | 20:08 |
| fungi | </the_count> | 20:08 |
| mordred | actually - let me make a quick venv so that I can verify that 26 works on each of these | 20:08 |
| clarkb | mordred: what about runabove? | 20:09 |
| mordred | clarkb: they were hanging before the thing which is why I'm checking 26 | 20:09 |
| mordred | yeah. runabove is not responding. I'm guessing htey're working on transitioning to ovh cloud | 20:10 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add openstack-modules.env https://review.openstack.org/209617 | 20:10 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: (WIP) Create playbook to puppet InfraCloud nodes https://review.openstack.org/209632 | 20:10 |
| *** Sukhdev_ has quit IRC | 20:11 | |
| openstackgerrit | Robert Clark proposed openstack-infra/project-config: Changes to allow publishing of Anchor developer documentation https://review.openstack.org/219865 | 20:11 |
| EmilienM | crinkle: why don't you target stable/kilo for ironic module on https://review.openstack.org/#/c/209617/14/openstack-modules.env,cm ? because of bifrost ? | 20:12 |
| sdague | mordred: so for these clouds "nova version-list" works on 2.26? | 20:13 |
| mordred | sdague: unitedstack works with 26 and hangs with 27 | 20:13 |
| mordred | sdague: checking | 20:13 |
| *** jtomasek has joined #openstack-infra | 20:13 | |
| mordred | | v2.0 | CURRENT | 2011-01-21T11:33:21Z | | 20:13 |
| mordred | oh! | 20:13 |
| mordred | it works on unitedstack both ones | 20:13 |
| mordred | let me try a few more things | 20:13 |
| mordred | so - vexxhost and unitedstack both just wor | 20:14 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: First stab at making things bifrosty https://review.openstack.org/208751 | 20:14 |
| *** Swami has quit IRC | 20:14 | |
| *** cburroughs has quit IRC | 20:14 | |
| *** glauco has joined #openstack-infra | 20:14 | |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: First stab at making things bifrosty https://review.openstack.org/208751 | 20:14 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add bifrost inventory management procedures https://review.openstack.org/214279 | 20:15 |
| mordred | sdague: on hp, version-list works, but nova list hangs | 20:15 |
| clarkb | I have git05 and git06 replacements building now | 20:15 |
| *** Swami__ has quit IRC | 20:15 | |
| *** austin81 has left #openstack-infra | 20:15 | |
| mordred | | v2.0 | CURRENT | 2011-01-21T11:33:21Z | | 20:15 |
| mordred | that's hp version list | 20:15 |
| sdague | yeh, they should all be the same | 20:15 |
| sdague | rax? | 20:15 |
| mordred | rax version-list fails 401 | 20:15 |
| *** sdake_ is now known as sdake | 20:15 | |
| mordred | on both 26 and 27 | 20:15 |
| sdague | are they the only one? | 20:15 |
| EmilienM | crinkle: if yes, I would suggest to backport the patches to stable/kilo | 20:16 |
| mordred | sdague: yes | 20:16 |
| sdague | ok | 20:16 |
| mordred | dreamhost and hp work with version-list with 2.6 and fail with 2.7 | 20:16 |
| mordred | auro: | 20:17 |
| mordred | ('get_current:endpoint:', u'https://api.auro.io:8774/v2/0776b1156efa42b4b3e14fca96cabf00') | 20:17 |
| sdague | ok, so that gives me a path. We can get the same info off the resource that version-list calls | 20:17 |
| mordred | ERROR (ConnectionRefused): Unable to establish connection to http://api.auro.io:8774/v2/ | 20:17 |
| mordred | but also works with version-list in 26 | 20:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add experimental puppet-unit job for single node https://review.openstack.org/219866 | 20:17 |
| sdague | yeh, so version list is actually doing a GET of http://api.auro.io:8774 | 20:17 |
| *** geoffarnoldX is now known as geoffarnold | 20:19 | |
| *** glauco has quit IRC | 20:19 | |
| sdague | mordred: thanks for all of that | 20:21 |
| *** markvoelker has quit IRC | 20:21 | |
| *** mwagner_lap has quit IRC | 20:23 | |
| clarkb | I have updated ssh rsa host keys to match the old hosts and will flip DNS now then do git repo replication | 20:23 |
| clarkb | I probably should puppet the host keys but it will likely take longer than just doing it | 20:23 |
| *** ildikov has quit IRC | 20:23 | |
| nibalizer | ya probably | 20:23 |
| nibalizer | also there is a bootstrap problem there likely because of known-hosts on the puppet master and ansible ya? | 20:24 |
| mordred | clarkb: just do it | 20:24 |
| clarkb | zaro: ^ maybe there is a way to teach gerrit to reload its known hosts file if it hits X number of host key mismatches? | 20:27 |
| clarkb | zaro: or maybe make it a admin command? | 20:27 |
| *** sharikapongubala has joined #openstack-infra | 20:27 | |
| mordred | clarkb, sdague: runabove list works with 2.26, hangs with 2.27. version-info hangs in both | 20:27 |
| greghaynes | SpamapS: How did you make that inventory erb on the bm deploy host? | 20:28 |
| greghaynes | SpamapS: I see, you commited it | 20:29 |
| *** jsavak has quit IRC | 20:29 | |
| *** sharikapongubala has quit IRC | 20:29 | |
| clarkb | ok DNS is done now just waiting for it to resolve properly on review.o.o | 20:29 |
| *** bhunter71 has quit IRC | 20:30 | |
| asselin | is this the proper way to restart zuul? we always use sudo service zuul stop/start/restart https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart | 20:30 |
| *** jsavak has joined #openstack-infra | 20:30 | |
| clarkb | asselin: it depends on your init system, systemd for example would do something like systemctl restart zuul | 20:30 |
| sdake | could a core reviewer ack this review for kolla infra so I can fix up the project rename review scheduled for the 11th? | 20:30 |
| sdake | https://review.openstack.org/#/c/217926/ | 20:30 |
| clarkb | asselin: with upstart `restart zuul` is the coorect incantation iirc | 20:31 |
| sdake | already has a +2 from Ian | 20:31 |
| *** bswartz has quit IRC | 20:31 | |
| sdake | fwiw, this gate is failing all the time (its a removal patch) | 20:31 |
| asselin | ok thanks | 20:31 |
| sdake | so it would also free up more resources | 20:31 |
| sdake | as an aded plus ;) | 20:31 |
| clarkb | I have about 45 minutes until DNS is correct on review.o.o | 20:31 |
| clarkb | sdake: there is a disconnect between removing the f21 job and the rest of the commit message. How are they related? | 20:32 |
| amuller | quick question - I recall that the neutron functional job had at some point a mechanism that would fail a test if it used sudo directly without neutron-rootwrap, that was controlled by setting the sudo settings on the stack user correct? It seems like that's gone, and the stack user that is executing the tests can actually just call random sudo commands now | 20:33 |
| clarkb | amuller: it was done by the run-tox.sh script | 20:34 |
| clarkb | amuller: it noted when the job started in the auth log and checked for any instances of sudo after that | 20:34 |
| clarkb | if found the job exited 1 | 20:34 |
| dimtruck | good afternoon! i was wondering if there's a way to tie a node in a nodepool to a specific openstack project | 20:34 |
| clarkb | so could be a number of things not working there, have an example job run we can look at? | 20:35 |
| amuller | clarkb: Thanks I'll look in to it | 20:35 |
| amuller | yes | 20:35 |
| *** jsavak has quit IRC | 20:35 | |
| amuller | clarkb: http://logs.openstack.org/03/216603/4/check/gate-neutron-dsvm-functional/9ed19a4/console.html, ctrl-f for '['sudo | 20:35 |
| amuller | clarkb: I know exactly why it's using sudo directly in that specific test | 20:35 |
| dimtruck | e.g. run nova jenkins jobs on a specific instance ... i tried http://docs.openstack.org/infra/nodepool/configuration.html but couldn't find anything | 20:35 |
| amuller | and I'm going to fix that | 20:35 |
| *** e0ne has joined #openstack-infra | 20:35 | |
| amuller | clarkb: but I first want to re-enable the code that checks for direct sudo usage without rootwrap | 20:35 |
| *** jsavak has joined #openstack-infra | 20:36 | |
| *** sharikapongubala has joined #openstack-infra | 20:36 | |
| fungi | mordred: sdague: is there a novaclient bug report open for the currently observed behavior, or should i go ahead and create one? | 20:36 |
| clarkb | amuller: oh, so that actually runs in devstack-gate which uses a different mechanism, we remove the stack users sudo privs there | 20:36 |
| sdague | fungi: mordred is creating one right now | 20:36 |
| sdague | it will include george carlin level language | 20:37 |
| clarkb | amuller: and the job did fail, are we sure that this isn't working properly? | 20:37 |
| fungi | okay, i'll hold off following up to the release announcement in that case | 20:37 |
| amuller | clarkb: it failed for a different reason | 20:37 |
| amuller | clarkb: there's plenty of tests in that run that are actually using sudo without rootwrap | 20:37 |
| amuller | and passed | 20:37 |
| *** markvoelker has joined #openstack-infra | 20:37 | |
| *** Hal has joined #openstack-infra | 20:37 | |
| *** Hal has quit IRC | 20:37 | |
| sdague | fungi: and I'll look at fixes in the morning | 20:37 |
| clarkb | amuller: I see what it is, that test overrides the gate_hook which is where we remove sudo powers normally | 20:38 |
| *** rfolco has quit IRC | 20:38 | |
| clarkb | amuller: so the test has gone off and done its own thing without preserving that functionality | 20:38 |
| amuller | clarkb: ok, I'll add it back | 20:38 |
| mordred | fungi: https://bugs.launchpad.net/nova/+bug/1491579 | 20:38 |
| openstack | Launchpad bug 1491579 in OpenStack Compute (nova) "against all sanity, nova needs to work around broken public clouds" [Undecided,New] | 20:38 |
| clarkb | amuller: look for DEVSTACK_GATE_REMOVE_STACK_SUDO in devstack-gate to see how we do it typically | 20:39 |
| amuller | clarkb: Much appreciated :) | 20:39 |
| fungi | mordred: thanks, i'll reply on the -dev ml with a link to that unless you're already doing so | 20:39 |
| mordred | is there a thread? | 20:39 |
| *** Hal has joined #openstack-infra | 20:39 | |
| clarkb | mordred: sdague it is common to put layer 7 firewall type things in front of web services and they have to be trained to accept traffic, usually you do this by putting it in permissive type mode and hitting it with all the client functionality, then you tell it to enforce | 20:39 |
| clarkb | mordred: sdague so the client not exercising valid APIs there may have bitten us | 20:40 |
| *** vahidh has joined #openstack-infra | 20:40 | |
| mordred | clarkb: what does that gain? | 20:40 |
| clarkb | whether or not thats sane is another question but in my experience very common for deploying HTTP services | 20:40 |
| fungi | mordred: i replied to the novaclient release announcement | 20:40 |
| sdake | clarkb perhaps I busted the commit let me look second | 20:40 |
| *** sharikapongubala has quit IRC | 20:40 | |
| fungi | mordred: because that's a good way to start a -dev ml thread about breakage ensuing in new releases | 20:41 |
| sdague | clarkb: yeh, so my plan is, create a series of work arounds in novaclient so people can work again | 20:41 |
| *** sharikapongubala has joined #openstack-infra | 20:41 | |
| sdague | then add tempest tests to touch all these urls, and ensure they get into the defcore definition | 20:41 |
| sdague | so you just won't be openstack if you don't expose this | 20:41 |
| mordred | sdague: ++ | 20:41 |
| sdake | clarkb we are remvoing this gate : - name: gate-kolla-functional-f21 | 20:41 |
| clarkb | sdake: yes but what does that have to do with bash vs python? there is a giant paragraph of stuff that has nothing to do with f21 | 20:42 |
| *** dprince has quit IRC | 20:42 | |
| sdake | clarkb yes i understand now the commit log is confusing | 20:42 |
| sdake | clarkb probbly has something to do with the 36 hour days i've been pulling over the last 3 weeks ;) | 20:43 |
| sdake | would you prefer i udpate the commit log? | 20:43 |
| *** julim has joined #openstack-infra | 20:43 | |
| sdake | I just want to get er done before the 11th so i can get the renames done in one go | 20:43 |
| clarkb | sdake: well first just want to check that the change does what you want it to do and if it does then yes please update the commit message, otherwise updatethe change to do the thing that is described | 20:44 |
| clarkb | sdake: because as is I have no idea what the correct thing is | 20:44 |
| sdake | so basically functional-f21 uses a bash script tobulid | 20:44 |
| sdake | the other gates use a python tool | 20:44 |
| sdake | functional-f21 is really misnamed it only does bah script building tet | 20:44 |
| sdake | but the bash scripts have been removed from the repo | 20:44 |
| clarkb | I see, in that case a small update to the commit message that says "removing the functional-f21 job removes the only use of the old bashscript" | 20:44 |
| sdake | so its a dead gate | 20:44 |
| sdake | will do | 20:45 |
| *** Piet has quit IRC | 20:45 | |
| sdake | in a ocnversation in tripeo atm, but after that I'll udpate | 20:45 |
| sdake | thanks for the recommendation - hopefully can get a quck review after | 20:45 |
| crinkle | EmilienM: yes we should backport it | 20:45 |
| *** harlowja has quit IRC | 20:45 | |
| *** wshao has joined #openstack-infra | 20:46 | |
| *** signed8bit has joined #openstack-infra | 20:47 | |
| *** puranamr_ has quit IRC | 20:48 | |
| *** markvoelker has quit IRC | 20:48 | |
| jasondotstar | I've got a system-config qn | 20:50 |
| jasondotstar | qn is about running a local.pp file to setup a VM as a test node for review.o.o | 20:50 |
| mordred | jasondotstar: yes! it's definitely a thing to do | 20:51 |
| *** wshao has quit IRC | 20:51 | |
| jasondotstar | tried using this: http://paste.openstack.org/show/442387/ | 20:51 |
| *** dkranz has quit IRC | 20:52 | |
| openstackgerrit | Emma Barber proposed openstack-infra/puppet-hound: Configure and run apache on top of hound https://review.openstack.org/211640 | 20:52 |
| jasondotstar | and basically following this: http://docs.openstack.org/infra/system-config/sysadmin.html#sysadmin | 20:52 |
| jasondotstar | to kick it off, I created this Vagrantfile: http://paste.openstack.org/show/442388/ | 20:53 |
| jasondotstar | i get what it's *suppose* to do... but it's not running cleanly | 20:53 |
| jasondotstar | lemme see if I can grab my output... | 20:54 |
| boris-42 | mordred: hi | 20:54 |
| *** ociuhandu has quit IRC | 20:54 | |
| *** kushal has quit IRC | 20:56 | |
| *** adrian_otto has joined #openstack-infra | 20:56 | |
| rm_work | hey all, could we get this merged? https://review.openstack.org/#/c/217330/ | 20:57 |
| rm_work | the code it depends on is merged now in Barbican | 20:57 |
| anteaya | rm_work: we are having a hard time merging anything right now | 20:57 |
| *** julim has quit IRC | 20:57 | |
| rm_work | would like to get the jobs running for a bit | 20:57 |
| rm_work | ahhh ok | 20:57 |
| rm_work | problems? :( | 20:57 |
| anteaya | as both host clouds are having issues | 20:57 |
| rm_work | ok | 20:57 |
| anteaya | https://bugs.launchpad.net/nova/+bug/1491579 | 20:58 |
| openstack | Launchpad bug 1491579 in python-novaclient "against all sanity, nova needs to work around broken public clouds" [Critical,Confirmed] | 20:58 |
| anteaya | and the backscroll has details | 20:58 |
| rm_work | thanks, I will come back ... later :) give you some time to work that out | 20:58 |
| anteaya | rm_work: http://lists.openstack.org/pipermail/openstack-dev/2015-September/073535.html | 20:59 |
| anteaya | rm_work: thanks for understanding, its a fun one | 20:59 |
| boris-42 | anteaya: hi | 20:59 |
| anteaya | how can I help you boris | 21:00 |
| *** armax has quit IRC | 21:00 | |
| *** stevemar has quit IRC | 21:00 | |
| *** armax has joined #openstack-infra | 21:00 | |
| *** trown is now known as trown|outttypeww | 21:01 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: kolla: Remove kolla-functional-f21 job https://review.openstack.org/217926 | 21:01 |
| rm_work | anteaya: lol, I love that bug report -- doesn't seem very professional, but made me laugh none-the-less | 21:02 |
| annegentle | mordred: around? | 21:02 |
| anteaya | rm_work: the first time you have witnessed mordred's work I take it | 21:02 |
| anteaya | annegentle: hey | 21:02 |
| boris-42 | anteaya: so in Rally project we are facing issues | 21:02 |
| boris-42 | anteaya: currently all DSVM jobs are testing master (instead of patches) | 21:03 |
| boris-42 | anteaya: this is fix https://review.openstack.org/#/c/219823/ | 21:03 |
| anteaya | annegentle: if you could review dhellmann's patch both he and I would be very grateful: https://review.openstack.org/#/c/219420/ | 21:03 |
| boris-42 | anteaya: could you please take a look? | 21:03 |
| anteaya | annegentle: if you can work it on your stack | 21:03 |
| anteaya | boris-42: I can take a look | 21:03 |
| boris-42 | anteaya: thank you | 21:03 |
| anteaya | boris-42: be aware our host clouds are both in a bad way | 21:03 |
| anteaya | as I just told rm_work | 21:03 |
| rm_work | yes, i mean, we should have known everything would be broken -- it's right about code freeze time, so this is to be expected | 21:04 |
| *** btully has quit IRC | 21:04 | |
| rm_work | the gods enjoy taunting us right at code-freeze | 21:04 |
| zaro | clarkb: sorry, was afk. gerrit loading known hosts is in regards to replication correct? | 21:05 |
| *** rossella_s has quit IRC | 21:05 | |
| *** rossella_s has joined #openstack-infra | 21:06 | |
| anteaya | rm_work: now you've got it | 21:07 |
| *** baoli has joined #openstack-infra | 21:07 | |
| *** glauco has joined #openstack-infra | 21:07 | |
| anteaya | boris-42: you have two jobs that have two plugins enabled in the job, both sahara and rally and both manilla and rally | 21:07 |
| annegentle | anteaya: sure | 21:07 |
| fungi | anteaya: the novaclient issue isn't (yet) impacting nodepool and we can pin it on the server if needed as a workaround should it start to do so | 21:07 |
| anteaya | annegentle: thank you | 21:07 |
| *** Rockyg has quit IRC | 21:08 | |
| boris-42 | anteaya: as far as I remember we have more | 21:08 |
| *** baoli has quit IRC | 21:08 | |
| boris-42 | anteaya: e.g. zaqar & rally, designate & rally | 21:08 |
| anteaya | fungi: ah sorry, I thought you were seeing rax and hp issues and that the novaclient was the issue | 21:08 |
| anteaya | fungi: but yes I did wonder how it could affect us so quickly | 21:08 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Run beaker tests on Puppet 3.8 and 4.2 https://review.openstack.org/219879 | 21:08 |
| *** baoli has joined #openstack-infra | 21:08 | |
| *** Swami has joined #openstack-infra | 21:08 | |
| *** Swami_ has joined #openstack-infra | 21:08 | |
| *** achanda has quit IRC | 21:08 | |
| anteaya | boris-42: okay so I'm about to break for dinner and you can't believe how I hesitate to ask this question | 21:09 |
| anteaya | but if you have two devstack plugins enabled for the same job | 21:09 |
| anteaya | how can you know what you are testing? | 21:09 |
| clarkb | zaro yes | 21:09 |
| *** thorst has quit IRC | 21:09 | |
| jasondotstar | mordred: k sorry for the long wait but here's how I ran it: https://asciinema.org/a/2rdb51mh5lq7jm8xxr95ru9qa | 21:09 |
| *** stevemar has joined #openstack-infra | 21:09 | |
| boris-42 | anteaya: so like we are running rally agaisnt zaqar | 21:10 |
| boris-42 | anteaya: so you need to have rally & zaqar installed | 21:10 |
| boris-42 | anteaya: that is why we are adding 2 plugins.. | 21:10 |
| *** tiswanso has quit IRC | 21:10 | |
| anteaya | but the plugins pull in the tip of master, yes? | 21:11 |
| *** pfallenop has quit IRC | 21:11 | |
| anteaya | to get the patch in, yes? | 21:11 |
| anteaya | isn't that what your commit message states? | 21:11 |
| boris-42 | anteaya: sooo if we are trigerring rally job for rally patch | 21:12 |
| *** glauco has quit IRC | 21:12 | |
| * anteaya listens | 21:12 | |
| boris-42 | anteaya: it should use zaqar master and rally + patch | 21:12 |
| boris-42 | anteaya: if it is in zaqar gate it should run zaqar + patch and rally master | 21:12 |
| boris-42 | anteaya: that's the idea=) | 21:12 |
| boris-42 | it worked before | 21:13 |
| boris-42 | but now we are trying to switch to new plugin base | 21:13 |
| anteaya | you weren't testing the patch before | 21:13 |
| zaro | clarkb: does gerrit load known hosts? i thought ssh was the thing that loads it? i could be wrong, but seems to me that load/reload of known hosts is outside Gerrit's scope. | 21:13 |
| anteaya | that is what your commit message states | 21:13 |
| anteaya | the job ran before | 21:13 |
| *** Sukhdev_ has joined #openstack-infra | 21:13 | |
| anteaya | but it wasn't testing what you thought it was testing | 21:13 |
| anteaya | that is what you tell me is the point of this patch | 21:13 |
| boris-42 | anteaya: I know exactly that before the new devstack plugins we tested everything properly | 21:14 |
| *** pfallenop has joined #openstack-infra | 21:14 | |
| clarkb | zaro: gerrit uses a java ssh implementation (MINA iirc) and loads the file on start | 21:14 |
| clarkb | zaro: you hvae to restart gerrit to get it to reload the file | 21:14 |
| boris-42 | anteaya: e.g. if it is in rally (it is rally+patch against master) if it is in other project other project + rally maser | 21:14 |
| *** unicell has quit IRC | 21:14 | |
| anteaya | boris-42: can you update your commit message then please? | 21:14 |
| *** unicell has joined #openstack-infra | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| anteaya | because I am going by what your commit message says | 21:15 |
| pabelanger | clarkb: fungi: jeblair: When you have a moment, I wold be interested in feedback of: https://review.openstack.org/#/c/217346/ | 21:15 |
| boris-42 | anteaya: okay let me try to rephrase | 21:15 |
| anteaya | which you now tell me is not an accurate reflection of reality | 21:15 |
| pabelanger | interested in what you thing of the organization variable, or if you see another way | 21:15 |
| anteaya | I just want what you say you want to do to be reflected in the commit message | 21:15 |
| pabelanger | We are looking, EmilienM and team, to get that merged to fix some depends-on issue we have with beaker-jobs and our integration job | 21:15 |
| clarkb | pabelanger: we would prefer to not have organization vars back in jjb | 21:16 |
| clarkb | pabelanger: can you use ZUUL_PROJECT instead? | 21:16 |
| *** jtomasek has quit IRC | 21:16 | |
| *** bcourt has quit IRC | 21:17 | |
| pabelanger | clarkb: don't think I can. I would want ZUUL_PROJECT=foo to zuul clone project bar | 21:17 |
| pabelanger | and don't see how I could get that info | 21:17 |
| *** chutwig has joined #openstack-infra | 21:18 | |
| clarkb | pabelanger: ok well we need some way to not reencode the github thing back into all our config | 21:18 |
| *** tonytan4ever has quit IRC | 21:18 | |
| clarkb | in theory ZUUL_PROJECT is that thing but I may have misread what zuul cloner is doing | 21:18 |
| zaro | clarkb: ahh, ok. i'll write up a bug to remind myself to look into that. | 21:19 |
| *** dims has quit IRC | 21:19 | |
| pabelanger | clarkb: okay, let me look | 21:19 |
| asselin | greghaynes, fyi, my images built sucessfully now w/o any updates. thanks | 21:19 |
| *** dims has joined #openstack-infra | 21:19 | |
| pabelanger | I couldn't see a way to extract out the organization info from zuul | 21:20 |
| *** baoli has quit IRC | 21:20 | |
| clarkb | pabelanger: you are setting up org/name with zuul cloner, the thing that encodes that in the zuul parameters is ZUUL_PROJECT | 21:21 |
| clarkb | pabelanger: so I don't understand why you can't use that parameter | 21:21 |
| *** harlowja has joined #openstack-infra | 21:21 | |
| *** edmondsw has quit IRC | 21:21 | |
| *** chutwig is now known as erhudy1 | 21:22 | |
| clarkb | its the zuul cloner default too iirc | 21:22 |
| clarkb | so it should just work (tm) | 21:22 |
| * anteaya is off for dinner | 21:22 | |
| jeblair | clarkb, pabelanger: use the macro "zuul-git-prep" to clone $ZUUL_PROJECT; use the macro "zuul-clone" to clone a _specific_ project | 21:23 |
| *** dims has quit IRC | 21:24 | |
| jeblair | but that change is functionally equivalent to what's there right now | 21:24 |
| *** aysyd has quit IRC | 21:25 | |
| jeblair | as nibalizer points out | 21:25 |
| *** tonytan4ever has joined #openstack-infra | 21:26 | |
| *** stevemar has quit IRC | 21:26 | |
| *** ociuhandu has joined #openstack-infra | 21:29 | |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s) https://review.openstack.org/219760 | 21:29 |
| nibalizer | jeblair: paul corrected me a bit, if the gate-keystone-... test is added to another repo then this change does the right thing | 21:29 |
| *** ociuhandu has quit IRC | 21:30 | |
| clarkb | nibalizer: pabelanger if thats how you want to do it you should have a single job that just does the correct thing, similar to how devstack gate works | 21:30 |
| clarkb | you don't want many many integration tests, you just want one | 21:31 |
| *** ociuhandu has joined #openstack-infra | 21:31 | |
| *** e0ne has quit IRC | 21:32 | |
| nibalizer | or http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n122 | 21:32 |
| *** ociuhandu has quit IRC | 21:32 | |
| jeblair | nibalizer: if that job were instantiated for the "foo" jjb project, then it would create "gate-foo-puppet..." and checkout foo. it would do that regardless of using gerrit-git-prep or zuul-clone | 21:32 |
| *** kgiusti has left #openstack-infra | 21:33 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Switch Rally gates to new DevStack plugin style https://review.openstack.org/219823 | 21:33 |
| *** e0ne has joined #openstack-infra | 21:33 | |
| *** derekh has joined #openstack-infra | 21:33 | |
| boris-42 | anteaya: ^ better ? | 21:33 |
| *** cdent has quit IRC | 21:33 | |
| fungi | pabelanger: clarkb: jeblair: nibalizer: wow, 217346 is a blast from the past. what was old really is new again | 21:33 |
| *** ociuhandu has joined #openstack-infra | 21:33 | |
| jeblair | fungi: if only it had shown up as a revert? :) | 21:34 |
| fungi | srsly | 21:34 |
| nibalizer | oh did we used to attach a 'project' to each repo? | 21:34 |
| clarkb | replication is almost done to new 05 and 06 | 21:34 |
| nibalizer | er organization | 21:34 |
| clarkb | once done I will do some test clones and push up a change to put them in use | 21:34 |
| fungi | nibalizer: yep, even used the same parameter name | 21:35 |
| clarkb | nibalizer: yes | 21:35 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 21:35 |
| fungi | it was a pbd when we were finally able to rip it out | 21:35 |
| nibalizer | fungi: clarkb haha | 21:35 |
| fungi | well, actually it was called github-org for a while i think | 21:35 |
| fungi | i can't remember now if it got renamed to organization before being ripped out though... that was so many beers ago | 21:36 |
| openstackgerrit | Christopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip https://review.openstack.org/218898 | 21:36 |
| *** lucas-dinner has quit IRC | 21:36 | |
| nibalizer | fungi: more beers more commits :) | 21:36 |
| *** achanda has joined #openstack-infra | 21:37 | |
| *** dannywil_ has quit IRC | 21:40 | |
| *** pvaneck_ has joined #openstack-infra | 21:40 | |
| *** jsavak has quit IRC | 21:40 | |
| *** glauco has joined #openstack-infra | 21:41 | |
| *** e0ne has quit IRC | 21:41 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/system-config: Index logs/screen-m-shr.txt for debugging manila runs in the gate https://review.openstack.org/219888 | 21:42 |
| *** olaph has quit IRC | 21:43 | |
| *** e0ne has joined #openstack-infra | 21:43 | |
| *** pvaneck has quit IRC | 21:44 | |
| *** marzif has joined #openstack-infra | 21:44 | |
| clarkb | mriedem: should you do the other manila services too? | 21:45 |
| mriedem | clarkb: i thought about it | 21:45 |
| mriedem | i can | 21:45 |
| mriedem | mtreinish: seen this? http://logs.openstack.org/35/219835/2/check/gate-novaclient-dsvm-functional/a887156/console.html#_2015-09-02_20_48_59_361 | 21:45 |
| mriedem | melwitt: ^ | 21:45 |
| clarkb | mriedem: I think people will expect all of them if one is indexed | 21:45 |
| *** baoli has joined #openstack-infra | 21:46 | |
| *** alivigni has quit IRC | 21:46 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/system-config: Index logs/screen-m-shr.txt for debugging manila runs in the gate https://review.openstack.org/219888 | 21:46 |
| mriedem | let me rename that | 21:46 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/system-config: Index manila service logs for debugging manila runs in the gate https://review.openstack.org/219888 | 21:47 |
| *** dannywilson has joined #openstack-infra | 21:47 | |
| *** achanda has quit IRC | 21:47 | |
| *** FallenPegasus has joined #openstack-infra | 21:49 | |
| *** glauco has quit IRC | 21:49 | |
| mtreinish | mriedem: http://lists.openstack.org/pipermail/openstack-dev/2015-August/072982.html | 21:50 |
| *** amuller has quit IRC | 21:50 | |
| mriedem | mtreinish: ok, i'm going to e-r query that thing | 21:50 |
| mtreinish | mriedem: it shouldn't be breaking anything | 21:50 |
| mriedem | it is | 21:50 |
| mtreinish | does the hook err exit on that? | 21:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Switch to centos7 git05 and git06 hosts https://review.openstack.org/219892 | 21:51 |
| clarkb | anteaya: ^ tha will get us to half centos7 | 21:51 |
| melwitt | mriedem: I find 20 hits the past 24h in logstash, one of them is success the rest are failures | 21:51 |
| clarkb | once those are in I can delete the old centos6 hosts and do another batch (semi limited by rax quota here) | 21:51 |
| mriedem | mtreinish: melwitt: http://goo.gl/IQLHW1 | 21:52 |
| clarkb | mriedem: should nova consider releasing a 2.28 that is 2.26 rereleased? | 21:52 |
| clarkb | mriedem: then nova can discuss what to do with behavior change and maybe release a 3.0? | 21:52 |
| mriedem | clarkb: i was going to do 2.27.1 | 21:52 |
| mriedem | for this https://review.openstack.org/#/c/219835/ | 21:52 |
| *** dzamboni has quit IRC | 21:52 | |
| *** e0ne has quit IRC | 21:52 | |
| *** prad- has quit IRC | 21:53 | |
| mriedem | and then bump g-r min to that | 21:53 |
| clarkb | mriedem: right but that won't fix the break every public cloud behavior right? | 21:53 |
| mriedem | clarkb: no, we don't have a fix for that yet | 21:53 |
| clarkb | and in theory a behavior change like that requires a 3.0 if we are semvering | 21:53 |
| clarkb | mriedem: right so maybe we release 2.26 as 2.28 so people don't get hit with that? | 21:53 |
| mriedem | mordred: ^ thoughts? | 21:53 |
| mriedem | i thought it sounded like we kind of expect clouds to expose that | 21:53 |
| *** achanda has joined #openstack-infra | 21:54 | |
| mriedem | plus i need to leave RSN to pickup my kid from daycare | 21:54 |
| clarkb | I think we do but it won't happen any time soon if I know anyhting abut change control | 21:54 |
| mriedem | yeah | 21:54 |
| clarkb | and in the mean time not breaking everyone is nice | 21:54 |
| mriedem | let them eat cake! | 21:54 |
| *** glauco_ has joined #openstack-infra | 21:54 | |
| mriedem | so, | 21:54 |
| mriedem | can i double back on this in about an hour when i'm home? | 21:55 |
| mriedem | dansmith: ^ thoughts? | 21:55 |
| clarkb | sure | 21:55 |
| mriedem | not that having a 4 year old in my ear is going to help... | 21:55 |
| * dansmith reads | 21:55 | |
| mriedem | i'll check back in when i'm home | 21:56 |
| *** mriedem has quit IRC | 21:56 | |
| *** amitgandhinz has quit IRC | 21:56 | |
| dansmith | clarkb: I wasn't really part of the discussion of that | 21:57 |
| dansmith | clarkb: I just threw unhelpful thoughts in the middle | 21:57 |
| *** mtanino has quit IRC | 21:57 | |
| clarkb | ok, I was just throwing the idea out there since 2.27 provably doesn't work against the majority of public clouds that we (mordred) have access to | 21:58 |
| clarkb | and at least one major onsumer of these clouds is going to have to either pin novaclient everywhere or halt config managment or both to work around this until the clouds catch up | 21:58 |
| dansmith | I think that the proposed change fixes it for everyone, right? | 21:59 |
| *** Hal has quit IRC | 21:59 | |
| *** glauco_ has quit IRC | 21:59 | |
| dansmith | the correct behavior is to not pass it at all if it's not specified, which should satisfy the older and newer code, AFAIK | 21:59 |
| fungi | dansmith: no, there is no proposed change for bug 1491579 | 21:59 |
| openstack | bug 1491579 in python-novaclient "against all sanity, nova needs to work around broken public clouds" [Critical,Confirmed] https://launchpad.net/bugs/1491579 | 21:59 |
| *** burgerk_ has quit IRC | 21:59 | |
| fungi | dansmith: which is what clarkb is referring to | 22:00 |
| dansmith | fungi: not this? https://review.openstack.org/#/c/219835/ | 22:00 |
| clarkb | dansmith: two unrelated bugs aiui | 22:00 |
| fungi | nope. that one's small potatoes | 22:00 |
| dansmith | I thought they were related | 22:01 |
| fungi | dansmith: 1491579 is latest novaclient not working because most large service providers have decided to filter/break version discovery | 22:01 |
| fungi | dansmith: so the most likely fix is a variety of workarounds to accommodate the different ways they've broken their nova implementations in production | 22:02 |
| *** wshao has joined #openstack-infra | 22:02 | |
| *** wshao has quit IRC | 22:03 | |
| dansmith | I see, I thought this was related to the device thing in terms of being able to determine what the proper behavior is | 22:03 |
| mtreinish | fungi: my fix for the version discovery thing is to get hogepodge to fail them if they block it | 22:03 |
| mtreinish | I'm working on getting that test into tempest to enable that | 22:03 |
| dansmith | I was really only tangentially paying attention to both of those conversations | 22:03 |
| mtreinish | but it's sort of a long term fix | 22:03 |
| clarkb | mtreinish: I think we should do that too | 22:03 |
| dansmith | my feeling is that the most graceful thing we can/should do is assume that if you fail to return versions, then we assume 2.0, and you never get any new features until you provide that | 22:04 |
| dansmith | and if you hang, then whatever, you're stupid | 22:04 |
| clarkb | in the mean time it would be pragmatic to have novaclient work against clouds, file bugs against the clouds for their brokeness, and work on the long term fixes from there | 22:04 |
| dansmith | since this versioned API has been like three years in the making, I'm not sure I expect those bugs to go anywhere in a reasonable amount of time | 22:05 |
| *** mtanino has joined #openstack-infra | 22:05 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 22:05 |
| clarkb | dansmith: given how people setup http proxies I am assuming the fix is as simple as go update your rulesets and that will take however long change control for $cloud takes | 22:06 |
| clarkb | that may or may not be a reasonable amount of time | 22:06 |
| fungi | dansmith: yes, the alternative is to make a big stink in public and shame the offending providers into fixing their s...tuff | 22:06 |
| *** adrian_otto1 has joined #openstack-infra | 22:06 | |
| fungi | and let users suffer in the interim | 22:06 |
| *** ivar-lazzaro has quit IRC | 22:06 | |
| *** Sree has joined #openstack-infra | 22:07 | |
| dansmith | clarkb: I dunno, maybe | 22:07 |
| *** adrian_otto has quit IRC | 22:07 | |
| *** jsavak has joined #openstack-infra | 22:07 | |
| dansmith | fungi: yeah, I don't want to punch users in the face, but I also think that if we have to wait another six months to a year to get them all resolved, that will suck pretty hard | 22:07 |
| *** signed8bit has quit IRC | 22:08 | |
| fungi | yeah | 22:08 |
| clarkb | right so that is why I suggest a 3.0 release | 22:08 |
| *** annegentle has quit IRC | 22:08 | |
| melwitt | dansmith: yeah, sdague is going to work on implementing that tomorrow he said | 22:08 |
| clarkb | make it clear that this is not necessarily backward compatible with existing clouds and let users footgun if they want | 22:08 |
| fungi | dansmith: and the thing is, if you could name 3 or 4 of the largest openstack-based public providers off the top of your head, it's likely they're all not working with novaclient 2.27 | 22:08 |
| dansmith | clarkb: 3.0 is enough signaling for you? | 22:09 |
| *** btully has joined #openstack-infra | 22:09 | |
| *** ams__ has quit IRC | 22:09 | |
| dansmith | melwitt: cool | 22:09 |
| clarkb | dansmith: ya its not perfect but according to semver would be a more correct way to handle this | 22:09 |
| dansmith | fungi: yeah, I saw the paste | 22:09 |
| melwitt | dansmith: the fall back on v2.0 if getting versions fails or times out and use a short time out, that is | 22:09 |
| dansmith | clarkb: I don't understand at all how this is a semver-able thing, from the library side | 22:09 |
| dansmith | clarkb: are you saying "meh, a bunch of clouds are broken" is a semantic difference? | 22:10 |
| clarkb | dansmith: the client has made a change that is not backward compatible with existing clouds | 22:10 |
| *** jaypipes has quit IRC | 22:10 | |
| dansmith | clarkb: but only because those clouds are stupid, not because we're doing anything that is not legit | 22:10 |
| clarkb | dansmith: sure | 22:11 |
| clarkb | but still not backward compatible | 22:11 |
| clarkb | as a user that is what you will see | 22:11 |
| clarkb | the technicaly details don't matter much when novaclient stops working everywhere | 22:11 |
| melwitt | in the ideal case we could just do a new release with that workaround in it and not have to re-release an older client as the newest (essentially a revert) | 22:11 |
| clarkb | melwitt: that wont fix the problem | 22:11 |
| clarkb | it fixes a problem but not the big one | 22:12 |
| dansmith | clarkb: well, if we look for that behavior and dump a message with the phone number of the CEO of said cloud.... | 22:12 |
| clarkb | dansmith: hahaha | 22:12 |
| *** Sree has quit IRC | 22:12 | |
| dansmith | clarkb: I dunno, I feel like the client moving forward is the most leverage we'll have | 22:12 |
| clarkb | dansmith: right I say move forwad with a 3.0 release | 22:12 |
| clarkb | then users know hey upgrading to this may not work I have some testing to do | 22:12 |
| *** jsavak has quit IRC | 22:13 | |
| clarkb | rather than user upgrading point release and breaking everything grumping and never using openstack again | 22:13 |
| dansmith | I disagree with clouds being stupid being able to force us to make a versioning change like this, but if that's what people want as a compromise then that's fine I guess | 22:13 |
| *** jsavak has joined #openstack-infra | 22:13 | |
| *** btully has quit IRC | 22:13 | |
| dansmith | like, in a less all-clouds-are-apparently-stupid situation, what percentage of them being stupid warrants us doing this? | 22:14 |
| *** SumitNaiksatam has quit IRC | 22:14 | |
| dansmith | like, all the deployed redhat, mirantis, metacloud clouds (assuming) are fine... | 22:14 |
| *** jcoufal has quit IRC | 22:14 | |
| *** Rockyg has joined #openstack-infra | 22:14 | |
| clarkb | dansmith: its possible they are fine, depends on how they deploy the nova api | 22:15 |
| fungi | i could get behind recommending the non-broken providers to every user who complains about this problem ;) | 22:15 |
| dansmith | "4 out of 5 dentists agree" -> semantic version bump | 22:16 |
| clarkb | but clearly there is a majority of deployments that are publicly available doing the crazy thing so would not be surprised if private deployments fall into the same trap | 22:16 |
| dansmith | fungi: yeah, "sorry, your cloud is broken, but https:///foobar.com is not" | 22:16 |
| dansmith | clarkb: I'm not so sure | 22:16 |
| *** SumitNaiksatam has joined #openstack-infra | 22:17 | |
| fungi | i mean, ultimately that's refstack/marketplace... "here's the list of non-broken providers. when you start running a non-broken openstack we'll be happy to add you (back) to the list!" | 22:17 |
| rm_work | as someone peripherally involved only, I vote for whatever makes it so we can merge changes soonest <_< | 22:18 |
| *** marzif has quit IRC | 22:18 | |
| dansmith | is this blocking the gate? | 22:18 |
| rm_work | I was told we can't merge stuff really right now because of this by anteaya | 22:18 |
| clarkb | dansmith: no, but if nodepool upgrades we will no longer boot any test VMs | 22:18 |
| clarkb | which means to fix that I have to go fix nodepool and a half dozen other places we use novaclient | 22:18 |
| rm_work | or maybe it was just that "everyone is busy looking at this, so we don't have time to review" | 22:18 |
| rm_work | i could have misunderstood | 22:19 |
| dansmith | rm_work: yeah, doesn't sound like that's right | 22:19 |
| clarkb | rm_work: I think it was more we don't have time to review | 22:19 |
| *** hdd has quit IRC | 22:19 | |
| rm_work | alright | 22:19 |
| dansmith | clarkb: can't we cap whatever nodepool is doing to avoid that terrible situation? | 22:19 |
| dansmith | "if nodepool upgrades" makes it sound like it has a mind of its own.. :) | 22:19 |
| clarkb | dansmith: yes and shade, and the puppetmaster, and ansible (though in theory will be shade there?), | 22:19 |
| clarkb | and its not just us using rackspace and hpcloud and friends | 22:20 |
| dansmith | sure, but tomorrow is a deadline and if we know about it, then being proactive would be good | 22:21 |
| clarkb | I get that the public clouds are silly, but I also know that novaclient users won't care, they want a novaclient that works with clouds that exist | 22:21 |
| fungi | basically we can update our tools to only support python-novaclient<2.7.0 | 22:21 |
| fungi | but that seems like not a great long-term solution | 22:21 |
| dansmith | fungi: no, not a long-term solution | 22:22 |
| dansmith | fungi: a solution to avoid mutiny in the next 48 hours because we have a deadline and we knew about the iceberg early enough to make sure we didn't hit it | 22:22 |
| * dansmith just watched a titanic documentary | 22:22 | |
| clarkb | fungi: jenkins04 still happy with the thread counts | 22:22 |
| *** sabeen2 has quit IRC | 22:23 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 22:23 | |
| *** achanda has quit IRC | 22:24 | |
| *** achanda_ has joined #openstack-infra | 22:24 | |
| dansmith | clarkb: given that this is going to happen now or later, has anyone filed a bug with the two clouds we use? presumably that is the most useful if someone does it "as the client" | 22:24 |
| *** tonytan4ever has quit IRC | 22:24 | |
| clarkb | dansmith: I have not, I was hoping mordred would considering he had all the accounts (iirc filing bugs requires accounts) | 22:24 |
| dansmith | right, okay | 22:24 |
| clarkb | jroll: onmetal question, does nova client just work with it ? (assuming we use a client version that works with rax) | 22:25 |
| armax | clarkb: fwiw https://bugs.launchpad.net/grenade/+bug/1274539 | 22:25 |
| openstack | Launchpad bug 1274539 in grenade "Failure in /.../grenade/projects/30_swift/upgrade.sh" [Undecided,New] | 22:25 |
| clarkb | jroll: eg can I nova boot --flavor onmetal --image somethinginglance clarkbmachine? | 22:25 |
| *** eharney has quit IRC | 22:26 | |
| *** glauco has joined #openstack-infra | 22:26 | |
| *** thorst has joined #openstack-infra | 22:27 | |
| *** jsavak has quit IRC | 22:27 | |
| clarkb | jroll: I ask beacuse the docs tell me to use this supernova thing http://www.rackspace.com/knowledge_center/article/using-onmetal-cloud-servers-through-api | 22:29 |
| *** puranamr has joined #openstack-infra | 22:29 | |
| dansmith | isn't supernova generally recommended for rax stuff? | 22:29 |
| clarkb | dansmith: we only use the upstream libs and other than the current issue and their ssl cert being something requests doesn't like it just works | 22:30 |
| dansmith | right, I meant, | 22:30 |
| dansmith | I thought that they document using supernova is probably just their SOP, and would expect the nova client to work as much as it does with vms | 22:30 |
| clarkb | also we should totally rebug about the requests cert thing when we complain about the nova version api thing | 22:30 |
| dansmith | when I was playing with onmetal pre-ga, | 22:30 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 22:30 |
| *** btully has joined #openstack-infra | 22:31 | |
| dansmith | there were instructions to use the novaclient, IIRC | 22:31 |
| *** puranamr has quit IRC | 22:31 | |
| clarkb | dansmith: ok, so it should work or at least did work when you were testing it out? | 22:31 |
| *** thiagop has quit IRC | 22:31 | |
| dansmith | I didn't try it, but pretty sure there were instructions for it | 22:31 |
| j^2 | Is Craig Bryant still around? | 22:31 |
| dansmith | I just used the gui because it was the quickest way to get a box to watch things go fast on :) | 22:31 |
| clarkb | gotcha | 22:33 |
| fungi | j^2: seems to still be active on monasca as of last week https://review.openstack.org/#/q/owner:craig.bryant%2540hp.com,n,z | 22:33 |
| dansmith | clarkb: so basically, completely unhelpful response, but until jroll comes around, you're bored anyway, right? :) | 22:33 |
| clarkb | assuming supernova doens't do any fancy switching on the flavor type then novaclient would work | 22:33 |
| j^2 | fungi: ah he posted a question on one of my reviews, i wanted to make sure i understood it: https://review.openstack.org/#/c/219760/5 | 22:34 |
| dansmith | http://docs.rackspace.com/servers/api/v2/cs-gettingstarted/content/overview.html | 22:34 |
| dansmith | ^ mentions onmetal and nova client, makes no caveats, that I can see | 22:34 |
| clarkb | aha I needed out of the knowledge center to get the real docs | 22:35 |
| clarkb | google you failed me | 22:35 |
| *** glauco has quit IRC | 22:35 | |
| *** btully has quit IRC | 22:35 | |
| *** taron2 has quit IRC | 22:35 | |
| *** dannywilson has quit IRC | 22:35 | |
| clarkb | yup and onmetal flavors are in that list | 22:35 |
| dansmith | yeah | 22:35 |
| *** gordc has quit IRC | 22:36 | |
| *** dannywilson has joined #openstack-infra | 22:37 | |
| *** mwagner_lap has joined #openstack-infra | 22:37 | |
| clarkb | ok I have to go buy groceries for dinner, back in a bit to check up on git server backends | 22:38 |
| fungi | clarkb: i've approved that change | 22:40 |
| clarkb | thanks | 22:40 |
| fungi | confirmed the existing 5 and 6 are handling no requests any longer | 22:40 |
| clarkb | I did disable them in haproxy | 22:42 |
| *** unicell has quit IRC | 22:43 | |
| *** unicell has joined #openstack-infra | 22:43 | |
| armax | lifeless: ping | 22:43 |
| lifeless | armax: pong | 22:43 |
| armax | lifeless: I was wondering if you could help me understand how I can render testrepository.subunit.gz in a human readable format | 22:44 |
| *** unicell has quit IRC | 22:44 | |
| clarkb | changing their IPs reenables them so no explicit enable needed | 22:44 |
| *** unicell has joined #openstack-infra | 22:44 | |
| armax | lifeless: forgive the silly question | 22:44 |
| openstackgerrit | Merged openstack-infra/system-config: Switch to centos7 git05 and git06 hosts https://review.openstack.org/219892 | 22:45 |
| lifeless | armax: is this the file attached to a failed CI run in OpenStack ? | 22:45 |
| armax | yes | 22:45 |
| lifeless | ok, so there is a web render done automatically | 22:45 |
| armax | like this one | 22:45 |
| armax | http://logs.openstack.org/01/214601/13/check/gate-tempest-dsvm-full/677ee89/logs/testrepository.subunit.gz | 22:45 |
| reed | the fact that some people don't want to learn how to use IRC and pretend to force on others to learn how to use $ANY_OTHER_TOOL is getting more annoying by the day | 22:45 |
| *** thorst has quit IRC | 22:45 | |
| reed | bad channel to rant about that, sorry | 22:46 |
| lifeless | hmm, its not glued in for tempest, odd. | 22:46 |
| armax | but I notice that the subunit file has a lot more information than the html one | 22:46 |
| lifeless | armax: sure yes. So - | 22:46 |
| reed | I came here to ask about shade though :) | 22:46 |
| reed | mordred, does shade have its own graphic logo or something to identify it visually? | 22:46 |
| lifeless | armax: its a subunit V1 file currently. so: gunzip -c < testrepository.subunit.gz | subunit1to2 | subunit2html | 22:46 |
| lifeless | armax: or whatever filter you want to use | 22:47 |
| lifeless | armax: the subunit package has a number, or you can import it into your local testrepository:: | 22:47 |
| lifeless | gunzip -c < testrepository.subunit.gz | subunit1to2 | testr load | 22:47 |
| armax | lifeless: ok let me give it a shot | 22:47 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 22:47 |
| lifeless | armax: 'human readable' formats - there are so many - junitxml might be useful for you, or csv | 22:48 |
| lifeless | armax: depends what sort of querying you want to do really | 22:48 |
| armax | lifeless: plain text would be fine | 22:49 |
| armax | lifeless: I am mainly interested in the test logging output | 22:49 |
| armax | lifeless: and I see that the html version doesn’t have it all | 22:49 |
| *** Rockyg has quit IRC | 22:49 | |
| clarkb | armax I just open the subunitv1 files with my text editor | 22:50 |
| *** rlandy has quit IRC | 22:50 | |
| armax | clarkb: there’s binary chars in the ones I am looking at | 22:50 |
| *** derekh has quit IRC | 22:50 | |
| clarkb | thats v2 then run it through subunit2to1 | 22:51 |
| clarkb | the output should be all text with v1 | 22:51 |
| *** shashank_hegde has quit IRC | 22:51 | |
| jhesketh | Morning | 22:52 |
| armax | clarkb: that’ll do I think | 22:52 |
| armax | clarkb, lifeless: thanks | 22:52 |
| *** ihrachys has quit IRC | 22:53 | |
| *** shashank_hegde has joined #openstack-infra | 22:53 | |
| *** ayoung has quit IRC | 22:53 | |
| lifeless | clarkb: v1 isn't text | 22:54 |
| lifeless | clarkb: it only looks like it | 22:54 |
| *** dimtruck is now known as zz_dimtruck | 22:54 | |
| lifeless | clarkb: because its framing is somewhat human readable. Chunk delimiters and the embedded content are 8-bit | 22:54 |
| *** mriedem has joined #openstack-infra | 22:55 | |
| lifeless | armax: in subunit's git repo there is a new filter | 22:55 |
| lifeless | armax: subunit2disk | 22:55 |
| lifeless | armax: it will explode all the attachments onto disk so you can look at them with a text editor | 22:55 |
| armax | lifeless: that’s good to know I’ll give it a shot | 22:55 |
| mriedem | clarkb: dansmith: so what did you guys come up with for the novaclient thing? | 22:56 |
| dansmith | mriedem: when we detect the situation, | 22:56 |
| dansmith | mriedem: we print a banner to the console with the home phone number of the CEO for the cloud being contacted | 22:56 |
| mriedem | dansmith: can it flash? | 22:57 |
| *** asettle has joined #openstack-infra | 22:57 | |
| dansmith | it *must* flash | 22:58 |
| clarkb | and scroll | 22:58 |
| dansmith | flashing marquis | 22:58 |
| mtreinish | dansmith: shell out to sl and print it as a banner on the trains | 22:59 |
| dansmith | ....Your cloud is broken ...... let the CEO know ..... CALL NOW! ..... 777-555-1212 ..... | 22:59 |
| dansmith | heh | 22:59 |
| *** mtanino_ has joined #openstack-infra | 22:59 | |
| openstackgerrit | Merged openstack/requirements: Add django-formtools to global requirements https://review.openstack.org/219222 | 22:59 |
| rm_work | lol | 23:01 |
| rm_work | well, once again I will plug https://review.openstack.org/#/c/217330/ if anyone has a moment, would love to get those jobs running ASAP | 23:01 |
| *** mtanino has quit IRC | 23:02 | |
| rm_work | two new nonvoting jobs | 23:02 |
| rm_work | coming from one that was experimental and already passing | 23:02 |
| rm_work | since I guess people will be busy with this novaclient thing for ... an "indeterminate" amount of time, heh | 23:03 |
| *** mriedem has quit IRC | 23:03 | |
| *** mriedem has joined #openstack-infra | 23:04 | |
| *** SumitNaiksatam has quit IRC | 23:09 | |
| *** ayoung has joined #openstack-infra | 23:09 | |
| *** angela-s has quit IRC | 23:10 | |
| *** SumitNaiksatam has joined #openstack-infra | 23:10 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for missing subunit2html bug 1491646 https://review.openstack.org/219909 | 23:11 |
| openstack | bug 1491646 in OpenStack-Gate ".tox/functional/bin/python: can't open file '/usr/local/jenkins/slave_scripts/subunit2html.py': [Errno 2] No such file or directory" [Undecided,New] https://launchpad.net/bugs/1491646 | 23:11 |
| *** achanda_ has quit IRC | 23:12 | |
| *** wenlock_ has quit IRC | 23:13 | |
| *** Sukhdev_ has quit IRC | 23:13 | |
| *** baoli has quit IRC | 23:14 | |
| boris-42 | clarkb: please could you take a look as well at the patch: https://review.openstack.org/#/c/219823/ Rally gates are really badly broken | 23:14 |
| *** sdake has quit IRC | 23:15 | |
| *** wenlock has quit IRC | 23:15 | |
| *** gildub has joined #openstack-infra | 23:15 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add bifrost manifest https://review.openstack.org/208751 | 23:16 |
| boris-42 | clarkb: I mean all dsvm jobs test Rally master instead of propsed patch | 23:16 |
| *** sputnik1_ has joined #openstack-infra | 23:16 | |
| *** shashank_hegde has quit IRC | 23:17 | |
| *** Piet has joined #openstack-infra | 23:17 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: First stab at making things bifrosty https://review.openstack.org/208751 | 23:17 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add openstack-modules.env https://review.openstack.org/209617 | 23:17 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add bifrost inventory management procedures https://review.openstack.org/214279 | 23:17 |
| *** shashank_hegde has joined #openstack-infra | 23:18 | |
| *** b10n1k_ has quit IRC | 23:19 | |
| *** erhudy has quit IRC | 23:19 | |
| *** b10n1k_ has joined #openstack-infra | 23:19 | |
| *** bknudson has left #openstack-infra | 23:20 | |
| *** sputnik13 has quit IRC | 23:20 | |
| *** bknudson has joined #openstack-infra | 23:20 | |
| *** baoli has joined #openstack-infra | 23:20 | |
| *** achanda has joined #openstack-infra | 23:21 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add bifrost inventory management procedures https://review.openstack.org/214279 | 23:21 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add bifrost manifest https://review.openstack.org/208751 | 23:23 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add bifrost inventory management procedures https://review.openstack.org/214279 | 23:23 |
| crinkle | ^ painful rebase | 23:24 |
| *** sharikapongubala has quit IRC | 23:24 | |
| *** baoli has quit IRC | 23:25 | |
| *** pfallenop has quit IRC | 23:26 | |
| *** sdake has joined #openstack-infra | 23:27 | |
| *** pfallenop has joined #openstack-infra | 23:27 | |
| *** yamamoto has quit IRC | 23:31 | |
| *** dannywilson has quit IRC | 23:31 | |
| notmyname | where could I go to read about or learn how infra is doing the logstash and swift stuff? ie you're storing logs in swift and also indexing it with ELK. I'd like to learn more about that plubming | 23:31 |
| notmyname | (actually i want to hand those docs to someone else ;-) | 23:32 |
| notmyname | jhesketh: ^ | 23:32 |
| *** wshao has joined #openstack-infra | 23:34 | |
| *** krotscheck is now known as kro_paternity | 23:34 | |
| *** wshao has quit IRC | 23:35 | |
| *** bhunter71 has joined #openstack-infra | 23:35 | |
| nibalizer | notmyname: here is a start http://docs.openstack.org/infra/system-config/logstash.html | 23:38 |
| nibalizer | afaik swift and logstash don't really interact | 23:39 |
| notmyname | ah, ok | 23:39 |
| notmyname | thanks | 23:39 |
| *** baoli has joined #openstack-infra | 23:40 | |
| fungi | yeah, we have a "proxy" in front of swift (a nova instance running apache) and the logstash workers just download the files they want through that apache server | 23:40 |
| fungi | in theory we could probably find a way to hand off the direct swift urls to the logstash worker layer | 23:41 |
| fungi | except today we're in a split transitional situation where some logs are in swift and some are on the filesystem of the logserver (that apache server) | 23:42 |
| *** harlowja has quit IRC | 23:42 | |
| *** harlowja has joined #openstack-infra | 23:43 | |
| *** morgabra has quit IRC | 23:45 | |
| *** ZZelle_ has quit IRC | 23:45 | |
| *** morgabra has joined #openstack-infra | 23:45 | |
| *** markvoelker has joined #openstack-infra | 23:46 | |
| jroll | clarkb: supernova is just a wrapper around nova that deals with env variables | 23:46 |
| jroll | clarkb: so yes, novaclient works with onmetal, do a flavor-list / image-list | grep OnMetal for flavors and images | 23:47 |
| nibalizer | downstream we run without swift, and the architecture is the same | 23:47 |
| jhesketh | notmyname: sorry was in a meeting.. but yes, that's the link I would have sent | 23:47 |
| jhesketh | and yes, swift and logstash don't interact | 23:48 |
| fungi | you know, just like that loverboy song... "pig and elephant dna just won't splice" | 23:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix exception lists in functional tests https://review.openstack.org/219655 | 23:49 |
| Kennan | hi fungi: | 23:51 |
| Kennan | could you help one question? | 23:51 |
| Kennan | we added publish docs jobs in project-config | 23:52 |
| *** achanda has quit IRC | 23:52 | |
| Kennan | but we did notice openstack.org site have such documents | 23:52 |
| Kennan | did you know about it ? why | 23:52 |
| *** sergmelikyan has quit IRC | 23:52 | |
| *** wshao has joined #openstack-infra | 23:53 | |
| Kennan | clarkb: or lifeless: do you know about it ? | 23:53 |
| sc68cal | fungi: lol wow that's a reference that takes me back | 23:54 |
| *** ajmiller has quit IRC | 23:56 | |
| lifeless | Kennan: I believe fungi answered your question about that about 12 hours back | 23:56 |
| Kennan | lifeless: maybe IRC was flooded with many message, I did not notice, OK, let me check IRC logs on yesterday | 23:57 |
| *** dangers is now known as dangers_away | 23:57 | |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Bump puppetlabs-apt to 2.1.0 https://review.openstack.org/215779 | 23:57 |
| *** wshao has quit IRC | 23:57 | |
| *** ybathia has quit IRC | 23:57 | |
| *** markvoelker has quit IRC | 23:58 | |
| *** shashank_hegde has quit IRC | 23:58 | |
| *** olaph has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
