| *** erlon has quit IRC | 00:01 | |
| mordred | WOOT! | 00:02 |
|---|---|---|
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Allow Magnum to conform to global requirements process https://review.openstack.org/170830 | 00:02 |
| openstackgerrit | Merged openstack/requirements: Remove lockfile from requirements https://review.openstack.org/151224 | 00:02 |
| *** sdake has quit IRC | 00:03 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Simple script to setup and test log server https://review.openstack.org/169117 | 00:05 |
| *** mtanino has quit IRC | 00:05 | |
| *** emagana_ has quit IRC | 00:06 | |
| *** stevemar has joined #openstack-infra | 00:06 | |
| *** emagana has joined #openstack-infra | 00:06 | |
| fungi | pbr tested with testtools installed by pbr... <singing>it's the circle of liiiife</singing> | 00:07 |
| *** zz_jgrimm has quit IRC | 00:10 | |
| *** emagana has quit IRC | 00:10 | |
| *** zz_jgrimm has joined #openstack-infra | 00:11 | |
| *** otter768 has joined #openstack-infra | 00:13 | |
| *** spzala has quit IRC | 00:13 | |
| *** sputnik13 has quit IRC | 00:15 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 00:16 | |
| *** zz_jgrimm has quit IRC | 00:17 | |
| *** otter768 has quit IRC | 00:17 | |
| *** zz_ja has quit IRC | 00:17 | |
| *** zz_ja has joined #openstack-infra | 00:18 | |
| *** zz_jgrimm has joined #openstack-infra | 00:18 | |
| jeblair | fungi: bravo! (i didn't want you to think the extended awkward silence was at all due to your singing!) | 00:22 |
| fungi | jeblair: well, to be fair... it usually is | 00:25 |
| fungi | i need to take lessons from SpamapS | 00:26 |
| openstackgerrit | Merged openstack-infra/git-review: Push language override down into the cmd wrapper https://review.openstack.org/179841 | 00:27 |
| *** zhiwei has joined #openstack-infra | 00:31 | |
| *** stevemar has quit IRC | 00:32 | |
| *** armax has joined #openstack-infra | 00:33 | |
| *** tiswanso has quit IRC | 00:40 | |
| *** hashar has quit IRC | 00:42 | |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Add files table https://review.openstack.org/179979 | 00:44 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Attach comments to files https://review.openstack.org/179980 | 00:44 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Add database pruning https://review.openstack.org/179981 | 00:44 |
| *** mpaolino has joined #openstack-infra | 00:44 | |
| jeblair | those changes are the result of two weeks of plane flights :) they could probably benefit from some careful review and testing -- but if you do feel like testing, it would probably be a good idea to back up your gertty database first | 00:45 |
| *** mpaolino has quit IRC | 00:45 | |
| jeblair | the first two changes involve a schema change, and the third actually deletes things | 00:45 |
| jeblair | (also, all of them are likely to take some time to execute the first time they are run) | 00:46 |
| mordred | jeblair: what if I want to run them, but not test things, and also complain when it breaks something? | 00:46 |
| jeblair | mordred: you've got this 'user' thing down | 00:48 |
| mordred | jeblair: I'm trying to empathize | 00:48 |
| jeblair | mordred: so regarding 175891 | 00:49 |
| mordred | jeblair: talk to me | 00:49 |
| jeblair | mordred: the deprecation note says "until it's reasonable to expect people to have upgraded to 1.10" | 00:49 |
| jeblair | mordred: we can remove that after crossing a shade release boundary, right? | 00:50 |
| mordred | jeblair: ish. the danger people are people who are running ansible and shade from pip install and who are just upgrading both with an unbounded instal | 00:50 |
| mordred | honestly, that's likely a very small set | 00:50 |
| jeblair | mordred: does ansible have a reasonable semver dep on shade? | 00:51 |
| mordred | if you installed ansible to run things, and installed shade to support it, and you upgrade ansible, it's not unreasonable that you would also upgrade shade | 00:51 |
| mordred | jeblair: ansible has no way to actually depend on shade - it has a way to give an error if you don't have shade installed | 00:51 |
| jeblair | mordred: do i want to ask followup questions? | 00:52 |
| *** shashankhegde has quit IRC | 00:52 | |
| mordred | which is to say - you may be running this code on a machien other than the one you ahve ansible installed on | 00:52 |
| mordred | and it's on _that_ machine that you want shade installed | 00:52 |
| jeblair | oh, hrm. i thought the ansible shade thing was only useful in ansible proper | 00:53 |
| mordred | jeblair: but yes - I do not htink we need to maintain it indefinitely - I think I more would want to wait until at least another ansible release | 00:53 |
| mordred | I think releasing a break to taht api while the only released bit is in a previous release would be rude | 00:53 |
| jeblair | mordred: okay. glad i asked. :) | 00:54 |
| *** dprince has joined #openstack-infra | 00:54 | |
| jeblair | mordred: i do feel like the dependency relationship here is problematic. but i'm also okay with long deprecation cycles in libraries. :) | 00:54 |
| mordred | jeblair: yeah - I agree- but also feel like supporting thigns for a decent stretch | 00:55 |
| mordred | beacuse otherwise we're evil | 00:55 |
| jeblair | i would like it if we could specify the lib dependency that we need, and the lib never changed. :) | 00:56 |
| mordred | jeblair: have I mentioned this language I like called rust ... | 00:56 |
| *** nikita09 has quit IRC | 00:56 | |
| mordred | jeblair: it solves none of these problems - but it's fun to talk about | 00:57 |
| mordred | jeblair: oh - I have a patch I need to send up for you and clarkb to look at | 00:57 |
| mordred | I wrote an ansible module in shell not python | 00:57 |
| mordred | which turns out to be both easy and non-insane | 00:57 |
| *** freerunner has quit IRC | 00:58 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/devstack-gate: Remove arbitrary devstack config options https://review.openstack.org/179982 | 01:01 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/project-config: Pass through arbitrary devstack config options https://review.openstack.org/179983 | 01:02 |
| samueldmq | mordred, ^ | 01:02 |
| samueldmq | mordred, I could find 3 options that could be passed through to devstack directly, instead of having d-g options for them | 01:02 |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Add 'shared_lock' package https://review.openstack.org/163666 | 01:04 |
| *** alexsyip has quit IRC | 01:08 | |
| *** jtriley has quit IRC | 01:09 | |
| *** tiswanso has joined #openstack-infra | 01:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering https://review.openstack.org/179988 | 01:14 |
| clarkb | jeblair: zaro ^ that passes mvn tests but I have not tested it with a real jenkins yet | 01:14 |
| clarkb | jeblair: zaro its also getting late so I probably made all sorts of mistakes, but I hope the general goals are apparent | 01:14 |
| *** tnovacik has quit IRC | 01:15 | |
| *** asettle has joined #openstack-infra | 01:16 | |
| mordred | clarkb: btw - I did a test on teh flamel question from earlier | 01:18 |
| mordred | >>> yaml.dump(dict(foo=u'100')) | 01:18 |
| mordred | "{foo: !!python/unicode '100'}\n" | 01:18 |
| mordred | clarkb: ^^ that's the thing I was talking about | 01:18 |
| clarkb | mordred: oh you are supposed to convert it to utf8 | 01:19 |
| clarkb | mordred: then its never an issue | 01:19 |
| clarkb | because yaml will assume utf8 reading it back | 01:19 |
| mordred | right. but I'm getting a giant dict from elsehwere | 01:19 |
| mordred | and it's showing up as u'' values | 01:19 |
| mordred | clarkb: I could rewrite teh unicode_representer to concert the u'' to utf-8 when it passes it to the yaml.ScalarNode constructor | 01:21 |
| jogo | debian packages point to github as the upstream: http://metadata.ftp-master.debian.org/changelogs//main/k/keystone/keystone_2015.1.0-1_copyright | 01:22 |
| clarkb | mordred: ya its odd to me that pyyaml doesn't do this by default | 01:22 |
| clarkb | mordred: since utf8 should cover the code point space right? | 01:22 |
| mordred | clarkb: it is EXCEPTIONALLY odd to me that it does not | 01:22 |
| clarkb | mordred: eg unicode perfectly serialized to utf8 | 01:22 |
| mordred | clarkb: one would think | 01:22 |
| mordred | clarkb: with as messy as the py2/py3 unicode transition is - I'd have hoped that pyyaml would have gotten it right | 01:23 |
| clarkb | jogo: file a bug with debian? | 01:23 |
| jogo | clarkb: don't even know where to start for that | 01:23 |
| clarkb | jogo: I say that somewhat sarcastically since debian bugs are basically a mail tracker ticket system from the dawn of computers | 01:23 |
| clarkb | jogo: but it apparntly works for them so ya | 01:24 |
| *** sarob has quit IRC | 01:27 | |
| *** purp_2 is now known as purp | 01:31 | |
| openstackgerrit | Huang Rui proposed openstack-infra/project-config: enable python-jobs for networking-zvm https://review.openstack.org/179730 | 01:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: DO NOT MERGE: Examples of doing bash library commands https://review.openstack.org/179990 | 01:33 |
| mordred | clarkb: that's just an example for reference ^^ | 01:34 |
| mordred | clarkb: I think it we wanted to make use of that ability, we'd want to think abot structure more | 01:34 |
| *** purp is now known as stealth_purp | 01:36 | |
| clarkb | ianw: is there a reason you pushed a new change for https://review.openstack.org/#/c/178905/1 ? | 01:37 |
| clarkb | ianw: you could just push a new patchset to the existing change | 01:37 |
| *** Longgeek has joined #openstack-infra | 01:46 | |
| *** jtriley has joined #openstack-infra | 01:47 | |
| *** unicell has quit IRC | 01:54 | |
| *** dprince has quit IRC | 01:56 | |
| ianw | clarkb: i was just in a stack of other stuff to get -e working as it should | 01:58 |
| *** Longgeek has quit IRC | 01:58 | |
| ianw | i can modify the change-id if you feel strongly | 01:59 |
| clarkb | no, its just really confusing for me as the reviewer to go "wait I wrote this change" | 02:00 |
| ianw | clarkb: let's call it "great minds" syndrome, i think we both saw the issue for different reasons | 02:02 |
| ianw | at pretty much the same time | 02:03 |
| *** julim has joined #openstack-infra | 02:03 | |
| *** patrickeast has quit IRC | 02:04 | |
| *** baoli has joined #openstack-infra | 02:05 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 02:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 02:06 |
| mordred | clarkb, jeblair: ^^ ok. I respun to fix the pep8 error, and added a verbose commend about why-flamel. I'm also happy to replace flamel with a copy of the relevant code if either of you would prefer | 02:06 |
| *** dboik has joined #openstack-infra | 02:07 | |
| *** dboik_ has joined #openstack-infra | 02:09 | |
| *** hloeung has quit IRC | 02:09 | |
| *** pcrews has quit IRC | 02:09 | |
| *** cody-somerville has joined #openstack-infra | 02:11 | |
| *** achanda has quit IRC | 02:11 | |
| *** dboik has quit IRC | 02:11 | |
| *** otter768 has joined #openstack-infra | 02:13 | |
| *** zz_dimtruck is now known as dimtruck | 02:16 | |
| *** otter768 has quit IRC | 02:18 | |
| *** baoli has quit IRC | 02:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add script to split the secret hieradata file https://review.openstack.org/179653 | 02:25 |
| mordred | jhesketh: I think I addressed your concerns ^^ - thanks, good feedback | 02:25 |
| greghaynes | mordred: looks like the https://review.openstack.org/#/c/179616/ stack needs some rebasing, mind if I go for it? | 02:26 |
| jhesketh | mordred: cool, thanks. I'll take a look later though sorry | 02:26 |
| mordred | greghaynes: go for it | 02:26 |
| mordred | jhesketh: totally no rush at al- very unurgent | 02:27 |
| *** samueldmq has quit IRC | 02:27 | |
| *** sarob has joined #openstack-infra | 02:28 | |
| *** shashankhegde has joined #openstack-infra | 02:31 | |
| *** asettle has quit IRC | 02:31 | |
| greghaynes | merge conflicts in autogenerated files = awesome | 02:33 |
| *** sarob has quit IRC | 02:33 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 02:33 | |
| *** ivar-laz_ has joined #openstack-infra | 02:35 | |
| *** baoli has joined #openstack-infra | 02:36 | |
| mordred | greghaynes: yah | 02:37 |
| *** yamahata has quit IRC | 02:37 | |
| *** stealth_purp is now known as purp | 02:37 | |
| *** ivar-lazzaro has quit IRC | 02:38 | |
| mordred | oh! nice | 02:38 |
| * mordred missed ianw's change | 02:38 | |
| greghaynes | heh, yea | 02:38 |
| greghaynes | im about to rebase https://review.openstack.org/#/c/179612/2 onto master so you can abandon the two below it | 02:38 |
| mordred | cool | 02:39 |
| *** sarob has joined #openstack-infra | 02:39 | |
| *** ivar-laz_ has quit IRC | 02:39 | |
| mordred | abandon is done | 02:39 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Start building ubuntu-trusty nodes https://review.openstack.org/179612 | 02:41 |
| *** dimtruck is now known as zz_dimtruck | 02:41 | |
| *** niska has quit IRC | 02:48 | |
| nibalizer | clarkb: https://github.com/nibalizer/hiera-garbage | 02:48 |
| nibalizer | clarkb: so 1) install taht in apply-test, 2) no more need for defaults in site.pp | 02:49 |
| nibalizer | for hiera() calls | 02:49 |
| *** Kennan2 has joined #openstack-infra | 02:49 | |
| *** Kennan has quit IRC | 02:50 | |
| mordred | I like nt having defaults in site.pp ... I find them very dangerous | 02:51 |
| mordred | s/nt/not/ | 02:51 |
| *** jyuso1 has joined #openstack-infra | 02:52 | |
| *** sputnik13 has joined #openstack-infra | 02:53 | |
| *** Longgeek has joined #openstack-infra | 02:56 | |
| *** wenlock has quit IRC | 02:58 | |
| *** jtriley has quit IRC | 03:01 | |
| *** niska has joined #openstack-infra | 03:03 | |
| *** shashankhegde has quit IRC | 03:03 | |
| crinkle | ++ | 03:06 |
| *** Longgeek has quit IRC | 03:08 | |
| *** tiswanso has quit IRC | 03:14 | |
| nibalizer | mordred: ya i didn't even realize the danger until clark pointed it out | 03:17 |
| nibalizer | then im like wat noooooooooo | 03:17 |
| *** niska has quit IRC | 03:18 | |
| *** pcrews has joined #openstack-infra | 03:21 | |
| *** coolsvap|afk is now known as coolsvap | 03:23 | |
| *** dims_ has quit IRC | 03:23 | |
| *** niska has joined #openstack-infra | 03:23 | |
| *** yamahata has joined #openstack-infra | 03:24 | |
| *** jtriley has joined #openstack-infra | 03:25 | |
| *** otter768 has joined #openstack-infra | 03:25 | |
| *** signed8bit has quit IRC | 03:25 | |
| *** jtriley has quit IRC | 03:29 | |
| *** asettle has joined #openstack-infra | 03:30 | |
| *** beisner- has joined #openstack-infra | 03:31 | |
| *** asettle has quit IRC | 03:31 | |
| *** beisner has quit IRC | 03:31 | |
| *** beisner- is now known as beisner | 03:31 | |
| *** asettle has joined #openstack-infra | 03:31 | |
| *** links has joined #openstack-infra | 03:31 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Update Networking L2 gateway meeting https://review.openstack.org/180016 | 03:33 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Update Nova API Meeting time https://review.openstack.org/180017 | 03:33 |
| *** otter768 has quit IRC | 03:33 | |
| *** otter768 has joined #openstack-infra | 03:35 | |
| *** sputnik13 has quit IRC | 03:38 | |
| *** Kennan has joined #openstack-infra | 03:39 | |
| *** shashankhegde has joined #openstack-infra | 03:40 | |
| *** Kennan2 has quit IRC | 03:40 | |
| *** zz_dimtruck is now known as dimtruck | 03:41 | |
| jamielennox | jhesketh: around https://review.openstack.org/#/c/172803/ - when we were discussing whether to use the _ or - for doa-kerberos nobody really cared | 03:42 |
| jamielennox | jhesketh: there is no inherent reason that doa uses _ and we figured we may as well go with the more standard scheme | 03:42 |
| jamielennox | but the first person with any strong opinion is probably going to carry the argument as no one else really cares | 03:43 |
| *** stevemar has joined #openstack-infra | 03:43 | |
| *** Kennan has quit IRC | 03:45 | |
| *** Kennan has joined #openstack-infra | 03:47 | |
| *** MarkAtwood has quit IRC | 03:47 | |
| *** armax has quit IRC | 03:47 | |
| *** sks has joined #openstack-infra | 03:48 | |
| *** sabeen has quit IRC | 03:49 | |
| *** sabeen has joined #openstack-infra | 03:49 | |
| *** achanda has joined #openstack-infra | 03:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 03:52 |
| *** baoli has quit IRC | 03:52 | |
| *** shashankhegde has quit IRC | 03:56 | |
| *** Somay has joined #openstack-infra | 03:59 | |
| *** yfried|afk is now known as yfried | 04:01 | |
| *** Longgeek has joined #openstack-infra | 04:05 | |
| *** ayoung-dadmode has quit IRC | 04:05 | |
| *** otter768 has quit IRC | 04:09 | |
| *** achanda has quit IRC | 04:12 | |
| *** gokrokve has joined #openstack-infra | 04:16 | |
| *** dimtruck is now known as zz_dimtruck | 04:19 | |
| *** ildikov has quit IRC | 04:20 | |
| *** gokrokve has quit IRC | 04:20 | |
| *** Kennan2 has joined #openstack-infra | 04:21 | |
| *** Kennan has quit IRC | 04:22 | |
| *** dims has joined #openstack-infra | 04:23 | |
| *** jtriley has joined #openstack-infra | 04:25 | |
| *** dims has quit IRC | 04:28 | |
| *** Longgeek has quit IRC | 04:29 | |
| *** sarob has quit IRC | 04:30 | |
| *** jtriley has quit IRC | 04:31 | |
| *** ildikov has joined #openstack-infra | 04:33 | |
| *** unicell has joined #openstack-infra | 04:35 | |
| *** sdake_ has quit IRC | 04:38 | |
| *** sks has quit IRC | 04:41 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:43 | |
| *** maishsk has quit IRC | 04:43 | |
| *** mrmartin has joined #openstack-infra | 04:45 | |
| *** gokrokve has joined #openstack-infra | 04:45 | |
| *** ildikov has quit IRC | 04:45 | |
| ianw | hmm, why does centos7 have eth0 & eth1, while ubuntu has only eth0 ... | 04:45 |
| mordred | ianw because | 04:48 |
| *** swat30 has quit IRC | 04:48 | |
| mordred | ianw: that actually should be fixed in the new minimal images | 04:50 |
| *** sabeen has quit IRC | 04:51 | |
| ianw | so it's hpcloud v rax difference | 04:51 |
| clarkb | mordred I dont think images fix that | 04:51 |
| clarkb | mordred we need both interfaces | 04:51 |
| mordred | it depends | 04:52 |
| clarkb | ianw yes rax gives you private and publoc nets | 04:52 |
| clarkb | hpcloud only does private with nat | 04:52 |
| *** swat30 has joined #openstack-infra | 04:54 | |
| *** yfried is now known as yfried|afk | 04:54 | |
| clarkb | mordred we use both on rax | 04:55 |
| *** sks has joined #openstack-infra | 04:56 | |
| *** sdake has joined #openstack-infra | 04:57 | |
| *** yfried|afk is now known as yfried | 04:58 | |
| mordred | clarkb: yah. the thing I was poorly saying before beacuse I was on my phone ... | 05:00 |
| *** sks has quit IRC | 05:01 | |
| mordred | clarkb: is that some of the images have baked in eth0 and eth1 files regardless of whether the instance actually has 2 interfaces or not | 05:01 |
| mordred | clarkb: and I believe that default is different across ubuntu and centos | 05:01 |
| mordred | but that has nothiing to do with what ianw was describing | 05:02 |
| clarkb | really? on rax they write in network startup scripts for you, and on hpcloud I have only ever seen tge one interface | 05:02 |
| mordred | clarkb: yeah - possible I'm thinking about base distro images | 05:03 |
| *** abregman has joined #openstack-infra | 05:03 | |
| mordred | clarkb: in any case - chalk it up to travel brain :) | 05:03 |
| clarkb | writing those scripts is what broke our local dns resolver setup because they just overrode everything | 05:03 |
| mordred | yah. so nice of them :) | 05:03 |
| clarkb | which is great because their dna doesnt work | 05:03 |
| mordred | yup | 05:03 |
| clarkb | *dns | 05:03 |
| mordred | dna too | 05:03 |
| mordred | I tried to boot their dna into a freshly cloned host organism and it did not get me anywhere | 05:04 |
| clarkb | that would be a neat trick, next step jurassic park | 05:04 |
| ianw | well, yeah, just clutching at straws trying to get some handle on these centos7 failures. debugging neutron is very, very far down the fun->painful spectrum | 05:07 |
| *** asettle has quit IRC | 05:07 | |
| SpamapS | mordred: sorry if I missed the middle, but I noticed the yaml.dump thing from earlier while reading backscroll.. | 05:09 |
| SpamapS | >>> yaml.safe_dump(dict(foo=u'100')) | 05:09 |
| SpamapS | "{foo: '100'}\n" | 05:09 |
| *** achanda has joined #openstack-infra | 05:10 | |
| SpamapS | mordred: you basically never want yaml.dump | 05:10 |
| SpamapS | unless you want the insanity of object serialization | 05:10 |
| *** Somay has quit IRC | 05:12 | |
| mordred | ah - gotcha. that probably makes sense as to why clarkb and I were missing each other there for a second | 05:12 |
| mordred | SpamapS: dare I ask why "dump" isn't the corrent API call to call/ | 05:12 |
| mordred | ? | 05:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 05:14 |
| mordred | SpamapS, clarkb: ^^ there. That should take care of the concerns | 05:14 |
| *** emagana has joined #openstack-infra | 05:16 | |
| lifeless | mordred: because we CANT HAVE NICE THINGS | 05:16 |
| *** BharatK has joined #openstack-infra | 05:17 | |
| *** shashankhegde has joined #openstack-infra | 05:18 | |
| mordred | lifeless: I hear that's a problem in the python world | 05:20 |
| lifeless | Do NOT get me started. | 05:22 |
| SpamapS | It so should be the default | 05:22 |
| SpamapS | and the other one should be 'unsafe_dump' and 'unsafe_load' | 05:23 |
| clarkb | mordred pretty sure its a problem with computers | 05:23 |
| clarkb | speaking of maybe I should tackle the jenkins thread leak next, I have already been writing java anyways | 05:24 |
| SpamapS | mordred: it really goes down to the fact that yaml can represent arbitrary things, and load will happily try to define them. | 05:24 |
| greghaynes | WCPGW | 05:24 |
| * greghaynes looks at ruby | 05:24 | |
| SpamapS | " Warning: It is not safe to call yaml.load with any data received from an untrusted source! yaml.load is as powerful as pickle.load and so may call any Python function. Check the yaml.safe_load function though." | 05:25 |
| SpamapS | http://pyyaml.org/wiki/PyYAMLDocumentation | 05:25 |
| * mordred hands SpamapS a half-used jar of pickles | 05:26 | |
| SpamapS | also the most performant version is 'yaml.load(x, Loader=yaml.CSafeLoader)' .. but it doesn't allow all the nice things from flamel IIRC | 05:26 |
| SpamapS | mordred: delicious | 05:26 |
| SpamapS | wait | 05:26 |
| SpamapS | how did you "use" them? | 05:26 |
| greghaynes | are those safe or unsafe pickles | 05:27 |
| clarkb | there are no safe pickles | 05:27 |
| mordred | SpamapS: it's best if you don't ask too many questions | 05:27 |
| greghaynes | Reviews on https://review.openstack.org/#/c/179209/ requested, its a pretty no-brainer | 05:28 |
| greghaynes | the depends-on just merged | 05:28 |
| purp | Check taint. That's all I'm sayin'. | 05:28 |
| mordred | ++ | 05:29 |
| mordred | greghaynes: +2 | 05:29 |
| greghaynes | ty | 05:29 |
| SpamapS | actually it looks like CSafeLoader does work fine | 05:31 |
| clarkb | SpamapS doesnt that require you to have libyaml available and linked against though? | 05:32 |
| SpamapS | clarkb: yeah, you have to do a conditional import | 05:32 |
| SpamapS | clarkb: it won't be present if libyaml isn't | 05:32 |
| clarkb | gotcha | 05:32 |
| mordred | SpamapS: we should make a library called yamlyaml which wraps pyyaml but renames dump and load to unsafe_ and makes safe_ the default - and also set default_flow_whatever to the opposite | 05:33 |
| SpamapS | Have oft thought of making a pypi project called 'cyaml' which just always fails if libyaml isn't available but otherwise is a passthrough to the C backed yaml calls | 05:33 |
| SpamapS | For large yaml docs, the pure python one can use 400x more memory and CPU, no joke. | 05:33 |
| *** tnovacik has joined #openstack-infra | 05:33 | |
| SpamapS | mordred: we could call that library 'flamel' ;) | 05:34 |
| mordred | SpamapS: bah. clearly flamel should use yamlyaml | 05:34 |
| SpamapS | reduce all the things | 05:34 |
| purp | Spamaps: Ugh. That's pretty rough. How's performance? | 05:34 |
| *** ildikov has joined #openstack-infra | 05:35 | |
| SpamapS | purp: not sure I know how to answer your question. | 05:35 |
| mordred | SpamapS: parsecs. always answer in parsecs. | 05:37 |
| lifeless | forty-2 | 05:37 |
| purp | SpamapS: with large docs, how long does load/and parse require for pure python vs. c lib? | 05:37 |
| lifeless | purp: 400x | 05:38 |
| lifeless | purp: for pathological cases | 05:38 |
| purp | lifeless: sorry, read the earlier as 400x mem, not time. I'm slow today. | 05:39 |
| SpamapS | It's logarithmic IIRC | 05:39 |
| SpamapS | or exponential, can't remember | 05:39 |
| SpamapS | We found the problem while developing juju which used yaml to serialize things in zookeeper. | 05:40 |
| lifeless | purp: mem & cpu | 05:40 |
| lifeless | purp: the cpu is where a direct time increase comes in | 05:40 |
| SpamapS | as soon as you got to like 30 services to deserialize suddenly juju would crumble. | 05:40 |
| lifeless | purp: enough memory pressure can cause perf issues too, of course | 05:40 |
| lifeless | gc overhead | 05:40 |
| lifeless | thrashing | 05:40 |
| lifeless | anyow | 05:40 |
| lifeless | there's a C thing, and we can all use it. \o/ and lets move on wif life | 05:41 |
| SpamapS | with the cdumper/loader we pushed it to a simulated 50,000 services before we just ran out of ram to store the result. | 05:41 |
| SpamapS | lifeless: it has deficiencies to the python one. | 05:41 |
| lifeless | SpamapS: box had < 50M of ram ? | 05:41 |
| SpamapS | lifeless: each service had a _VERY_ large document. :) | 05:42 |
| SpamapS | and relations were big too | 05:42 |
| SpamapS | And I think we were probably storing 3 or 4 copies of the result | 05:42 |
| mordred | SpamapS: it would have performed better if you'd kept the original name | 05:43 |
| *** hdd has joined #openstack-infra | 05:43 | |
| mordred | well, now that the new ansible modules have landed - I suppose putting off writing the docs for it is bad form | 05:44 |
| *** cody-somerville has quit IRC | 05:46 | |
| *** asettle has joined #openstack-infra | 05:46 | |
| *** HeOS has quit IRC | 05:48 | |
| *** SumitNaiksatam has quit IRC | 05:48 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:49 | |
| SpamapS | mordred: yeah ensemble was way faster than juju ;) | 05:49 |
| *** beecee has quit IRC | 05:52 | |
| *** sushilkm has joined #openstack-infra | 05:56 | |
| *** sushilkm has left #openstack-infra | 05:56 | |
| *** gokrokve_ has joined #openstack-infra | 05:58 | |
| *** Longgeek has joined #openstack-infra | 05:59 | |
| openstackgerrit | Merged openstack-infra/project-config: Add django-openstack-auth-kerberos project https://review.openstack.org/172803 | 05:59 |
| *** mrunge has joined #openstack-infra | 05:59 | |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Remove invalid ensure parameter from apt::ppa https://review.openstack.org/179618 | 06:00 |
| *** gokrokve has quit IRC | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/180039 | 06:02 |
| *** gokrokve_ has quit IRC | 06:02 | |
| *** maishsk has joined #openstack-infra | 06:06 | |
| *** asettle has quit IRC | 06:06 | |
| *** otter768 has joined #openstack-infra | 06:10 | |
| SpamapS | aha | 06:11 |
| SpamapS | found my yaml test code | 06:11 |
| SpamapS | http://paste.ubuntu.com/10988468/ | 06:11 |
| SpamapS | 5420 clint 20 0 1627556 1.521g 3392 R 99.7 9.8 0:47.70 python | 06:12 |
| *** SamYaple_ is now known as SamYaple | 06:13 | |
| SpamapS | http://paste.ubuntu.com/10988471/ results | 06:13 |
| SpamapS | purp: ^ in case you were still curious. | 06:14 |
| *** otter768 has quit IRC | 06:14 | |
| *** Longgeek_ has joined #openstack-infra | 06:14 | |
| *** jtriley has joined #openstack-infra | 06:15 | |
| SpamapS | oh interesting, I think I just found a python3 bug in pyyaml | 06:15 |
| *** Longgeek has quit IRC | 06:17 | |
| SpamapS | python3 performs worse for both C and pure python cases. Interesting. | 06:19 |
| *** masayukig has quit IRC | 06:19 | |
| *** jtriley has quit IRC | 06:19 | |
| *** masayukig has joined #openstack-infra | 06:19 | |
| *** openstackgerrit has quit IRC | 06:23 | |
| *** openstackgerrit has joined #openstack-infra | 06:23 | |
| *** bradm_ has joined #openstack-infra | 06:24 | |
| SpamapS | wow | 06:24 |
| SpamapS | pypy matches C | 06:24 |
| SpamapS | lifeless: ^ | 06:24 |
| lifeless | <3 pypy | 06:24 |
| SpamapS | http://paste.ubuntu.com/10988501/ | 06:24 |
| SpamapS | had to make it flexible to run via pypy | 06:24 |
| SpamapS | but wow | 06:25 |
| *** vponomaryov has quit IRC | 06:25 | |
| lifeless | SpamapS: so summary data for the different interpreters? | 06:25 |
| *** sslypushenko has quit IRC | 06:26 | |
| *** sslypushenko has joined #openstack-infra | 06:26 | |
| *** scheuran has joined #openstack-infra | 06:27 | |
| SpamapS | so, for a 100,000 key dict: python2, 47s, 1.6G RAM. python3, 53s, 250MB RAM. pypy, 7s, 154MB RAM, and libyaml, 8s, 170MB RAM | 06:28 |
| * SpamapS just ran each one 3 times and threw out highs/lows | 06:28 | |
| SpamapS | so python3 fixed the RAM insanity but made CPU worse.. better gc? | 06:28 |
| SpamapS | pypy is actually _beating_ C | 06:29 |
| SpamapS | consistently | 06:29 |
| clarkb | SpamapS is libyaml there via pyyaml binding? | 06:29 |
| SpamapS | clarkb: yes | 06:29 |
| lifeless | yes | 06:29 |
| *** woodster_ has quit IRC | 06:30 | |
| SpamapS | slight inconsistency in those results, I did 3.10 for python/python3, but 3.11 for pypy. Retrying all w/ 3.11. | 06:31 |
| SpamapS | (3.11 claims to just be a rebuild release but lets just make sure) | 06:31 |
| *** imcsk8 has quit IRC | 06:31 | |
| SpamapS | yeah same outcome | 06:31 |
| *** emagana has quit IRC | 06:32 | |
| * SpamapS wakes up from optimization death match to realize time has been wasted | 06:32 | |
| SpamapS | so.. if you need to parse giant yaml documents, DEFINITELY... use... pypy? ;) | 06:32 |
| *** mrunge has quit IRC | 06:33 | |
| *** emagana has joined #openstack-infra | 06:33 | |
| *** bradm_ has quit IRC | 06:36 | |
| *** bradm has quit IRC | 06:36 | |
| *** bradm_ has joined #openstack-infra | 06:37 | |
| *** mrunge has joined #openstack-infra | 06:37 | |
| *** shashankhegde has quit IRC | 06:38 | |
| greghaynes | now just to make all the openstack bits work with pypy :) | 06:38 |
| *** emagana has quit IRC | 06:38 | |
| *** shashankhegde has joined #openstack-infra | 06:39 | |
| *** vponomaryov has joined #openstack-infra | 06:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Minor translation cleanups https://review.openstack.org/179643 | 06:44 |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 06:44 |
| *** jcoufal has joined #openstack-infra | 06:50 | |
| *** e0ne has joined #openstack-infra | 06:51 | |
| *** yfried has quit IRC | 06:56 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/180039 | 06:56 |
| *** HeOS has joined #openstack-infra | 06:59 | |
| *** shashankhegde has quit IRC | 06:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 07:04 |
| *** e0ne has quit IRC | 07:06 | |
| *** sandywalsh has quit IRC | 07:11 | |
| *** cody-somerville has joined #openstack-infra | 07:12 | |
| *** cody-somerville has quit IRC | 07:12 | |
| *** cody-somerville has joined #openstack-infra | 07:12 | |
| *** sandywalsh has joined #openstack-infra | 07:12 | |
| *** markus_z has joined #openstack-infra | 07:12 | |
| *** yamahata has quit IRC | 07:14 | |
| *** achanda has quit IRC | 07:16 | |
| mgedmin | is bindep ready for alpha testing yet? | 07:16 |
| *** dizquierdo has joined #openstack-infra | 07:16 | |
| * mgedmin checks https://pypi.python.org/pypi/bindep | 07:16 | |
| mgedmin | looks like not :( | 07:16 |
| *** sushilkm has joined #openstack-infra | 07:17 | |
| *** sushilkm has left #openstack-infra | 07:18 | |
| lifeless | mgedmin: hmm? | 07:20 |
| lifeless | mgedmin: we haven't cut a release is all | 07:20 |
| mgedmin | well I can't pip install bindep yet ;) | 07:20 |
| mgedmin | a more on-topic observation: https://github.com/openstack-infra/bindep#writing-requirements-files doesn't mention comments | 07:20 |
| mgedmin | are comments allowed? because I really really want comments | 07:20 |
| mgedmin | # these are build-deps for Pillow: | 07:21 |
| mgedmin | etc. | 07:21 |
| lifeless | mgedmin: the grammar is in bindep/depends.py | 07:21 |
| *** achanda has joined #openstack-infra | 07:22 | |
| *** stevemar has quit IRC | 07:22 | |
| mgedmin | looks like the answer is "no" | 07:22 |
| mgedmin | afaiu there's no separate lexer/tokenizer? parsley works directly with a stream of characters? | 07:23 |
| lifeless | stream of objects | 07:23 |
| lifeless | this particular grammar is a character based one yes | 07:24 |
| *** achanda has quit IRC | 07:24 | |
| lifeless | should be easy enough to add | 07:25 |
| lifeless | comment = '#' any '\n' | 07:26 |
| lifeless | add a rule alternate for comment | 07:26 |
| *** luqas has joined #openstack-infra | 07:29 | |
| *** luqas has quit IRC | 07:30 | |
| *** maishsk has quit IRC | 07:30 | |
| lifeless | mgedmin: anyhow, as far as getting a release, I don't have the release acl for bindep for whatever reason (odd since I started it, but others are driving it so meh) | 07:30 |
| lifeless | mgedmin: chat to fungi, whose been driving it and should definitely have the release acl, and will be awake in oh, 4-5 hours | 07:31 |
| *** maishsk has joined #openstack-infra | 07:32 | |
| *** luqas has joined #openstack-infra | 07:32 | |
| *** luqas has quit IRC | 07:37 | |
| *** Hal has joined #openstack-infra | 07:38 | |
| *** Hal is now known as Guest10224 | 07:38 | |
| mgedmin | why are py34 and pypy in the default envlist in tox.ini if tests are known not to be passing on py34 and pypy? | 07:39 |
| mgedmin | (I assume they're known not to be passing) | 07:39 |
| mgedmin | (mox is a dependency and mox doesn't support python 3) | 07:39 |
| mgedmin | (the pypy error about lack of _bsddb is more mysterious to me) | 07:39 |
| mgedmin | where's the bug tracker for bindep? | 07:41 |
| mgedmin | I've a feeling I asked about this before :( | 07:41 |
| mgedmin | if the README had a link, I wouldn't need to ask again... | 07:41 |
| mgedmin | huh, perhaps I should read CONTRIBUTING.rst instead | 07:42 |
| mgedmin | waaah https://bugs.launchpad.net/bindep is a 404 | 07:42 |
| lifeless | I don't think we've made a bug tracker | 07:42 |
| *** jlanoux has joined #openstack-infra | 07:43 | |
| *** luqas has joined #openstack-infra | 07:43 | |
| *** chlong has quit IRC | 07:44 | |
| *** maishsk has quit IRC | 07:44 | |
| *** maishsk has joined #openstack-infra | 07:46 | |
| *** hdd_ has joined #openstack-infra | 07:48 | |
| *** ajo has joined #openstack-infra | 07:49 | |
| *** btran has quit IRC | 07:50 | |
| *** sushilkm has joined #openstack-infra | 07:50 | |
| *** sushilkm has left #openstack-infra | 07:50 | |
| *** bradm_ is now known as bradm | 07:50 | |
| *** hdd has quit IRC | 07:51 | |
| mgedmin | are there tests for the grammar? | 07:53 |
| *** aarefiev has joined #openstack-infra | 07:54 | |
| mgedmin | hm, okay | 07:54 |
| Daviey | lifeless: Sorry, i sent my mail - then went to bed. Hai o/ | 07:57 |
| mgedmin | AttributeError: 'Grammar' object has no attribute 'rule_any' | 07:57 |
| mgedmin | hmm | 07:57 |
| * mgedmin accidentally discovers himself reading http://en.wikipedia.org/wiki/Parsley | 07:57 | |
| mgedmin | ah, it's anything, not any | 07:59 |
| *** Somay has joined #openstack-infra | 08:02 | |
| *** samuelBartel has joined #openstack-infra | 08:02 | |
| mgedmin | any constraints on the grammar accepted by parsley? context-free? LL(1)? LR(1)? LALR(1)? (not that I remember what those are) | 08:03 |
| *** arxcruz has joined #openstack-infra | 08:03 | |
| mgedmin | I'm just wondering why it's not matching my blank line rule ... | 08:03 |
| mgedmin | grammar from http://paste.ubuntu.com/10988797/ fails the test with http://paste.ubuntu.com/10988798/ | 08:04 |
| *** liam_ has joined #openstack-infra | 08:04 | |
| *** liam_ is now known as Guest74585 | 08:05 | |
| * mgedmin reads about http://en.wikipedia.org/wiki/Parsing_expression_grammar | 08:06 | |
| lifeless | mgedmin: its ometa | 08:07 |
| lifeless | mgedmin: which is a PEG with extensions | 08:07 |
| lifeless | http://tinlizzie.org/ometa/ | 08:07 |
| lifeless | Daviey: no wakka | 08:07 |
| lifeless | mgedmin: yes there are | 08:07 |
| *** Guest74585 has quit IRC | 08:09 | |
| *** hashar has joined #openstack-infra | 08:11 | |
| *** hashar has quit IRC | 08:11 | |
| *** hashar has joined #openstack-infra | 08:11 | |
| *** otter768 has joined #openstack-infra | 08:11 | |
| *** fhubik has joined #openstack-infra | 08:14 | |
| *** _nadya_ has joined #openstack-infra | 08:15 | |
| *** otter768 has quit IRC | 08:16 | |
| frickler | hmm, zuul getting stuck again? all changes in the check pipeline have queued tasks and do not seem to progress | 08:21 |
| *** derekh has joined #openstack-infra | 08:23 | |
| *** fhubik is now known as fhubik_afk | 08:24 | |
| mgedmin | !logs | 08:25 |
| openstack | mgedmin: Error: "logs" is not a valid command. | 08:25 |
| mgedmin | (wifi loss == I don't know if I missed any replies) | 08:26 |
| *** e0ne has joined #openstack-infra | 08:28 | |
| *** e0ne is now known as e0ne_ | 08:28 | |
| *** vponomaryov has quit IRC | 08:28 | |
| *** fhubik_afk is now known as fhubik | 08:28 | |
| *** vponomaryov has joined #openstack-infra | 08:33 | |
| *** MaxV has joined #openstack-infra | 08:34 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/jenkins-job-builder: Fix Builder.update_job() on Python 3 https://review.openstack.org/172238 | 08:36 |
| *** Somay has quit IRC | 08:38 | |
| mgedmin | a-ha! anything eats the '\n', which makes the pattern not match | 08:38 |
| *** tkelsey has joined #openstack-infra | 08:38 | |
| mgedmin | okay, pull request time! | 08:40 |
| mgedmin | "Pull requests submitted through GitHub will be ignored." how friendly | 08:40 |
| openstackgerrit | Marius Gedminas proposed openstack-infra/bindep: Allow comments and blank lines in other-requirements.txt https://review.openstack.org/180060 | 08:41 |
| mgedmin | cool, it worked | 08:41 |
| *** e0ne_ is now known as e0ne | 08:44 | |
| *** SumitNaiksatam has quit IRC | 08:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable a check job for stackforge/dragonflow https://review.openstack.org/175152 | 08:45 |
| *** e0ne is now known as e0ne_ | 08:45 | |
| *** SumitNaiksatam has joined #openstack-infra | 08:46 | |
| *** e0ne_ is now known as e0ne | 08:46 | |
| lifeless | mgedmin: and you has a review :) | 08:46 |
| mgedmin | I see "(2 comments)" and "One small change requested"; how do I view the 2 comments? | 08:47 |
| lifeless | click on README.rst | 08:48 |
| anteaya | frickler: gerrit is still broadcasting events | 08:49 |
| mgedmin | got it | 08:49 |
| mgedmin | BTW my patch is wrong but the test suite doesn't catch the bug | 08:49 |
| *** dguitarbite has joined #openstack-infra | 08:49 | |
| *** ssam2 has joined #openstack-infra | 08:50 | |
| anteaya | frickler: I see queued and aborted, and the only job I see running is the top job in the post queue | 08:50 |
| frickler | anteaya: if you look at the graph of test nodes, it seems like more and more are made available, but not used | 08:51 |
| anteaya | frickler: I don't see any jobs running in the gate, though some are queued | 08:51 |
| anteaya | frickler: agreed | 08:51 |
| anteaya | I wonder what is happening | 08:51 |
| anteaya | SergeyLukjanov: are you about? | 08:51 |
| lifeless | mgedmin: so fix it up, then commit --amend; then git review | 08:51 |
| lifeless | mgedmin: :) | 08:51 |
| mgedmin | ugh, gerrit | 08:52 |
| mgedmin | does it support any kind of markup in comments? | 08:52 |
| lifeless | mgedmin: yes | 08:52 |
| lifeless | mgedmin: prefix lines with space to make them <pre> | 08:52 |
| anteaya | jhesketh: you aren't still online are you? | 08:53 |
| *** MaxV has quit IRC | 08:53 | |
| anteaya | frickler: what time zone are you? I'm soon back to bed | 08:54 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add keystone services/endpoints methods https://review.openstack.org/177621 | 08:54 |
| *** sergsh_ has joined #openstack-infra | 08:55 | |
| frickler | anteaya: GMT+2, so its morning here still ;) not sure how I can help with this, though | 08:55 |
| *** sergsh has joined #openstack-infra | 08:56 | |
| anteaya | frickler: if you see SergeyLukjanov around share your observations | 08:56 |
| anteaya | if you would | 08:56 |
| anteaya | he has root access to the servers | 08:56 |
| frickler | k, will do | 08:56 |
| anteaya | I don't and everyone else who does is fast asleep | 08:56 |
| anteaya | frickler: thank you | 08:56 |
| *** jistr has joined #openstack-infra | 08:56 | |
| anteaya | not sure what can be done, but at least he can get more information that I can | 08:57 |
| anteaya | frickler: if you don't see him, not to worry | 08:57 |
| anteaya | and thanks | 08:57 |
| frickler | np, good night to you, then | 08:57 |
| anteaya | thanks, good night | 08:58 |
| openstackgerrit | Marius Gedminas proposed openstack-infra/bindep: Allow comments and blank lines in other-requirements.txt https://review.openstack.org/180060 | 08:58 |
| *** deepakcs has joined #openstack-infra | 08:59 | |
| *** pelix has joined #openstack-infra | 08:59 | |
| lifeless | mgedmin: thanks, needs a second reviewer - again, nab fungi | 09:02 |
| *** scheuran has quit IRC | 09:02 | |
| *** fhubik is now known as fhubik_afk | 09:02 | |
| lifeless | night all | 09:03 |
| mgedmin | o/ | 09:10 |
| *** niska has quit IRC | 09:12 | |
| mgedmin | hmm, bindep takes 4 seconds to compute that none of my 30 packages are missing | 09:13 |
| *** shardy_ has joined #openstack-infra | 09:13 | |
| AJaeger | clarkb, jeblair, fungi : Post queue is again at 6 hours ;( | 09:14 |
| mgedmin | apt-get install --no-install-recommends $(sed e 's/#.*//' < other-requirements.txt) takes 0.8 seconds | 09:14 |
| *** shardy has quit IRC | 09:14 | |
| *** HeOS_ has quit IRC | 09:14 | |
| AJaeger | anteaya, that might be again the gearman bug that fungi, clarkb and jeblair are chasing for a week now. | 09:15 |
| frickler | AJaeger: anteaya is gone already | 09:15 |
| frickler | AJaeger: seems liek new nodes are getting prepared but never used | 09:16 |
| AJaeger | frickler, once fungi is up, he can rerun a command to get this sped up | 09:16 |
| *** scheuran has joined #openstack-infra | 09:16 | |
| *** HeOS_ has joined #openstack-infra | 09:16 | |
| AJaeger | frickler, http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-05-04.log - search for gearman | 09:17 |
| *** cdent has joined #openstack-infra | 09:17 | |
| *** fhubik_afk is now known as fhubik | 09:18 | |
| *** hdd__ has joined #openstack-infra | 09:18 | |
| *** shardy_ has quit IRC | 09:19 | |
| *** shardy has joined #openstack-infra | 09:19 | |
| AJaeger | frickler, but this looks today worse than "normal" ;( | 09:20 |
| openstackgerrit | Merged openstack-infra/release-tools: Include 'source_url' in notes and use readme parser function https://review.openstack.org/176528 | 09:20 |
| *** hdd_ has quit IRC | 09:21 | |
| *** jyuso1 has quit IRC | 09:21 | |
| *** jyuso has joined #openstack-infra | 09:21 | |
| *** samueldmq has joined #openstack-infra | 09:25 | |
| *** alexpilotti has joined #openstack-infra | 09:25 | |
| *** maishsk has quit IRC | 09:26 | |
| *** maishsk has joined #openstack-infra | 09:26 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/infra-specs: Allow reviewstats to be used with any issue tracker https://review.openstack.org/172886 | 09:27 |
| *** rlandy has joined #openstack-infra | 09:29 | |
| *** isviridov_away is now known as isviridov | 09:32 | |
| *** maishsk has quit IRC | 09:32 | |
| *** maishsk has joined #openstack-infra | 09:32 | |
| *** gilllliard is now known as gilliard_afk | 09:33 | |
| openstackgerrit | Merged openstack-infra/release-tools: Move highest_semver.py from oslo-incubator https://review.openstack.org/179243 | 09:34 |
| openstackgerrit | Merged openstack-infra/release-tools: update make_library_stable_branch.sh to set defaultbranch https://review.openstack.org/173919 | 09:34 |
| *** isviridov is now known as isviridov_away | 09:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/release-tools: Add script to cleanup of translations https://review.openstack.org/172827 | 09:36 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Networking L2 gateway meeting https://review.openstack.org/180016 | 09:37 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Nova API Meeting time https://review.openstack.org/180017 | 09:37 |
| *** markvoelker has quit IRC | 09:48 | |
| *** ociuhandu has joined #openstack-infra | 09:51 | |
| sambetts | Hey Infra, I have created a stable branch for the stackforge/networking-cisco repo, which all works great, however I have just cherrypicked a patch that I wanted from master and it doesn't appear that Jenkins is working with that repo, have I missed something in the configuration to make that work? | 09:51 |
| *** fhubik is now known as fhubik_afk | 09:52 | |
| AJaeger | sambetts, our CI is a bit slow right now. Check http://status.openstack.org/zuul/ and search for your change. I see three queued for the repo | 09:53 |
| *** niska has joined #openstack-infra | 09:53 | |
| SergeyLukjanov | anteaya, frickler, hi, (was driving to office), what's up? | 09:53 |
| AJaeger | sambetts, please have some patience | 09:53 |
| *** sergsh has quit IRC | 09:54 | |
| sambetts | AJaeger: Ah, I thought I might have missed something again, thanks for a quick response | 09:54 |
| AJaeger | SergeyLukjanov, seems we run again into the gearman issue where we basically loose all workers | 09:54 |
| *** zz_dimtruck is now known as dimtruck | 09:54 | |
| *** sergsh has joined #openstack-infra | 09:54 | |
| AJaeger | fungi has done some magic in the past for that - not sure what he did and I'm also not sure it's the same issue. | 09:54 |
| AJaeger | the bug would lead to stale ready notes | 09:55 |
| SergeyLukjanov | AJaeger, heh, I don't know what fungi was doing to fix it... :( | 09:56 |
| AJaeger | SergeyLukjanov, I didn't ping you ;) | 09:56 |
| *** scheuran has quit IRC | 09:57 | |
| AJaeger | SergeyLukjanov, might be best to wait for fungi | 09:57 |
| *** Guest10224 has quit IRC | 09:57 | |
| AJaeger | SergeyLukjanov, Fungi mentioned "i've gone ahead and deleted any nodes in a ready state for more than an hour, which should hopefully get things moving again rsn" | 09:58 |
| *** samueldmq has quit IRC | 09:58 | |
| AJaeger | SergeyLukjanov, that was during the weekend | 09:59 |
| *** zhiwei has quit IRC | 09:59 | |
| SergeyLukjanov | AJaeger, looking for bash history on zuul server :) | 10:00 |
| *** ajo has quit IRC | 10:01 | |
| BobBall | Anyone know how many posts to /servers are made for each node created by nodepool? I imagine at least 2, but if it's more I'm kinda interested. I've got a limit for 1000 requests to /servers per 24 hour period which I exceeded. Trying to understnd how many failed server builds that equates to. | 10:01 |
| *** niska has quit IRC | 10:03 | |
| *** maishsk has quit IRC | 10:07 | |
| *** maishsk has joined #openstack-infra | 10:07 | |
| *** ajo has joined #openstack-infra | 10:08 | |
| openstackgerrit | prabhu murthy proposed openstack/requirements: Added pyVim & pyVmomi to global requirements https://review.openstack.org/180088 | 10:09 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: use optimized url for zuul status https://review.openstack.org/170935 | 10:10 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: add dbg function to hideci https://review.openstack.org/179573 | 10:10 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: Serve up jquery-visibility.js for gerrit https://review.openstack.org/179765 | 10:10 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: add jquery visibility https://review.openstack.org/179362 | 10:10 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: create proxy for testing hideci https://review.openstack.org/179360 | 10:10 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: Revert "Revert zuul status in gerrit" https://review.openstack.org/179361 | 10:10 |
| *** dimtruck is now known as zz_dimtruck | 10:10 | |
| *** yfried has joined #openstack-infra | 10:11 | |
| *** _nadya_ has quit IRC | 10:11 | |
| *** otter768 has joined #openstack-infra | 10:12 | |
| *** fhubik_afk is now known as fhubik | 10:12 | |
| *** sushilkm has joined #openstack-infra | 10:14 | |
| *** niska has joined #openstack-infra | 10:14 | |
| *** sushilkm has left #openstack-infra | 10:14 | |
| *** vryzhenkin is now known as freerunner | 10:15 | |
| *** freerunner is now known as vryzhenkin | 10:15 | |
| *** vryzhenkin is now known as freerunner | 10:15 | |
| *** jyuso1 has joined #openstack-infra | 10:15 | |
| *** otter768 has quit IRC | 10:16 | |
| *** jyuso194 has joined #openstack-infra | 10:18 | |
| *** jyuso has quit IRC | 10:19 | |
| *** dims has joined #openstack-infra | 10:20 | |
| *** wznoinsk has joined #openstack-infra | 10:20 | |
| *** jyuso1 has quit IRC | 10:21 | |
| *** katyafervent is now known as katyafervent_awa | 10:22 | |
| *** dims_ has joined #openstack-infra | 10:22 | |
| *** samueldmq has joined #openstack-infra | 10:23 | |
| *** dims has quit IRC | 10:26 | |
| *** jlanoux_ has joined #openstack-infra | 10:27 | |
| *** maishsk is now known as maishsk_afk | 10:29 | |
| *** wznoinsk has quit IRC | 10:29 | |
| *** claudiub has joined #openstack-infra | 10:29 | |
| samueldmq | morning | 10:29 |
| *** wznoinsk has joined #openstack-infra | 10:30 | |
| *** jlanoux has quit IRC | 10:31 | |
| *** armax has joined #openstack-infra | 10:34 | |
| *** e0ne is now known as e0ne_ | 10:35 | |
| *** jyuso1 has joined #openstack-infra | 10:36 | |
| *** e0ne_ is now known as e0ne | 10:36 | |
| *** Hal has joined #openstack-infra | 10:36 | |
| *** Hal is now known as Guest21373 | 10:37 | |
| *** katyafervent_awa is now known as katyafervent | 10:38 | |
| *** jyuso194 has quit IRC | 10:39 | |
| *** armax has quit IRC | 10:42 | |
| *** pc_m has joined #openstack-infra | 10:44 | |
| *** jyuso has joined #openstack-infra | 10:46 | |
| *** jyuso1 has quit IRC | 10:49 | |
| *** scheuran has joined #openstack-infra | 10:54 | |
| *** e0ne is now known as e0ne_ | 10:56 | |
| *** yfried is now known as yfried|afk | 10:58 | |
| *** armax has joined #openstack-infra | 10:59 | |
| *** jlanoux has joined #openstack-infra | 11:00 | |
| *** fhubik has quit IRC | 11:01 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: use optimized url for zuul status https://review.openstack.org/170935 | 11:01 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: Serve up jquery-visibility.js for gerrit https://review.openstack.org/179765 | 11:01 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: add jquery visibility https://review.openstack.org/179362 | 11:01 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: Revert "Revert zuul status in gerrit" https://review.openstack.org/179361 | 11:01 |
| *** fhubik has joined #openstack-infra | 11:02 | |
| *** jlanoux_ has quit IRC | 11:02 | |
| *** yfried|afk is now known as yfried | 11:04 | |
| *** niska has quit IRC | 11:04 | |
| *** sandywalsh has quit IRC | 11:12 | |
| *** sandywalsh has joined #openstack-infra | 11:14 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 11:20 |
| *** niska has joined #openstack-infra | 11:21 | |
| *** freerunn_ has joined #openstack-infra | 11:25 | |
| *** jyuso1 has joined #openstack-infra | 11:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 11:32 |
| *** jyuso has quit IRC | 11:33 | |
| *** niska has quit IRC | 11:36 | |
| *** woodster_ has joined #openstack-infra | 11:38 | |
| *** freerunn_ has quit IRC | 11:40 | |
| *** niska has joined #openstack-infra | 11:41 | |
| *** marcusvrn1 has joined #openstack-infra | 11:46 | |
| *** marcusvrn has quit IRC | 11:48 | |
| deepakcs | mtreinish, when u get back here, pls take a look @ https://review.openstack.org/#/c/172841/ where i quoted ianw's opinion which seems to conflict with yours. | 11:50 |
| *** sushilkm has joined #openstack-infra | 11:51 | |
| *** markvoelker has joined #openstack-infra | 11:51 | |
| *** e0ne_ is now known as e0ne | 11:51 | |
| *** sushilkm has left #openstack-infra | 11:51 | |
| jklare | hi, could somebody give this one https://review.openstack.org/#/c/179721/ a quick push (we currently have no gates for our openstack-chef-repo) | 11:52 |
| *** Somay has joined #openstack-infra | 11:54 | |
| *** dprince has joined #openstack-infra | 11:54 | |
| *** ldnunes has joined #openstack-infra | 11:56 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Adjust precedence levels of params https://review.openstack.org/179459 | 11:59 |
| *** armax has quit IRC | 12:00 | |
| *** freerunner has quit IRC | 12:01 | |
| *** freerunner has joined #openstack-infra | 12:02 | |
| sdague | fungi: once you wake up, zuul seems stuck | 12:02 |
| *** maishsk_afk has quit IRC | 12:04 | |
| *** fhubik has quit IRC | 12:05 | |
| *** fhubik has joined #openstack-infra | 12:06 | |
| *** freerunner has quit IRC | 12:06 | |
| *** pc_m has quit IRC | 12:07 | |
| *** _nadya_ has joined #openstack-infra | 12:12 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: make zuul_inline = true https://review.openstack.org/180116 | 12:12 |
| *** otter768 has joined #openstack-infra | 12:13 | |
| *** kgiusti has joined #openstack-infra | 12:13 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support default string values for variables https://review.openstack.org/180117 | 12:13 |
| *** deepakcs has quit IRC | 12:15 | |
| *** mwagner_lap has quit IRC | 12:16 | |
| *** dkranz has joined #openstack-infra | 12:16 | |
| *** otter768 has quit IRC | 12:18 | |
| *** freerunner has joined #openstack-infra | 12:19 | |
| openstackgerrit | Michal Vanco proposed openstack-infra/jenkins-job-builder: Support for image gallery publisher https://review.openstack.org/175747 | 12:19 |
| *** niska has quit IRC | 12:21 | |
| openstackgerrit | Julien Danjou proposed openstack-dev/pbr: builddoc: allow to use fnmatch-style exclusion for autodoc https://review.openstack.org/180119 | 12:22 |
| jhesketh | anteaya: sorry I missed you earlier, I'm around at the moment if you are | 12:23 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org https://review.openstack.org/178026 | 12:24 |
| *** fhubik is now known as fhubik_afk | 12:25 | |
| *** dkranz has quit IRC | 12:26 | |
| *** niska has joined #openstack-infra | 12:27 | |
| *** fhubik_afk is now known as fhubik | 12:27 | |
| jhesketh | sdague: thanks for addressing my comments | 12:31 |
| *** mwagner_lap has joined #openstack-infra | 12:33 | |
| *** gordc has joined #openstack-infra | 12:34 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: use optimized url for zuul status https://review.openstack.org/170935 | 12:34 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Serve up jquery-visibility.js for gerrit https://review.openstack.org/179765 | 12:34 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: add jquery visibility https://review.openstack.org/179362 | 12:34 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: make zuul_inline = true https://review.openstack.org/180116 | 12:34 |
| *** jyuso has joined #openstack-infra | 12:37 | |
| *** freerunner has quit IRC | 12:37 | |
| *** Somay has quit IRC | 12:38 | |
| *** rfolco has joined #openstack-infra | 12:38 | |
| *** dkranz has joined #openstack-infra | 12:38 | |
| *** gyee has joined #openstack-infra | 12:39 | |
| *** jyuso1 has quit IRC | 12:39 | |
| *** niska has quit IRC | 12:40 | |
| *** baoli has joined #openstack-infra | 12:40 | |
| *** aysyd has joined #openstack-infra | 12:40 | |
| sergmelikyan | folks, is Zuul feeling well? | 12:41 |
| *** dims_ has quit IRC | 12:41 | |
| sergmelikyan | http://status.openstack.org/zuul/ to big available nodes number and some of the patches are hours in waiting queue | 12:41 |
| *** dims has joined #openstack-infra | 12:41 | |
| sergmelikyan | SergeyLukjanov ^ | 12:42 |
| SergeyLukjanov | sergmelikyan, yup it's known, waiting for fungi | 12:42 |
| fungi | hrm, looking | 12:43 |
| *** isviridov_away is now known as isviridov | 12:43 | |
| *** hashar is now known as hasharAway | 12:43 | |
| jhesketh | yeah that looks bad | 12:44 |
| sergmelikyan | thx :) | 12:45 |
| *** maishsk has joined #openstack-infra | 12:45 | |
| fungi | i suspect it's the jenkins-gearman bug clarkb's working on fixing, but manifsted in a much worse way than we'd seen yet | 12:45 |
| *** niska has joined #openstack-infra | 12:47 | |
| fungi | it'll take a bit to get these workers replaced and readded, but things should start picking back up again shortly | 12:47 |
| *** aufi has joined #openstack-infra | 12:47 | |
| *** spredzy is now known as spredzy|afk | 12:47 | |
| jhesketh | fungi: if it's jenkins-gearman wouldn't that only be the jenkins masters? | 12:48 |
| fungi | yes, it does | 12:48 |
| *** notnownikki has joined #openstack-infra | 12:48 | |
| *** mattt has joined #openstack-infra | 12:48 | |
| fungi | the workers end up in an odd state where they don't start running new jobs even though they're ready to do so | 12:48 |
| jhesketh | fungi: so possibly easiest to flush nodepool? | 12:49 |
| fungi | and as we cycle through workers, we slowly build up more and more in a ready state which never run jobs | 12:49 |
| fungi | jhesketh: great idea--that's what i'm doing! | 12:49 |
| jhesketh | that would explain the high number of test nodes | 12:49 |
| jhesketh | :-) | 12:49 |
| fungi | it just takes a few minutes to delete them all, and a little more to boot replacements for them | 12:50 |
| * jhesketh is confident in fungi | 12:50 | |
| jhesketh | yep | 12:50 |
| fungi | when there are >600 to churn through | 12:50 |
| jhesketh | let me know if I can do anything to help | 12:50 |
| fungi | thanks--also why are you awake? | 12:50 |
| fungi | it | 12:50 |
| fungi | 's gotta be late there | 12:50 |
| *** ssam2 has quit IRC | 12:51 | |
| jhesketh | nearly 11pm | 12:51 |
| jhesketh | so not too bad | 12:51 |
| jhesketh | I can do nights, it's the 5am mornings that I suck at | 12:51 |
| jhesketh | (the time of the infra-meeting) | 12:51 |
| fungi | indeed. and that's tomorrow for you... get some rest! | 12:51 |
| *** ssam2 has joined #openstack-infra | 12:51 | |
| *** links has quit IRC | 12:52 | |
| jhesketh | nah this is when I'm productive ;-) | 12:52 |
| jhesketh | fungi: is this the change you think is relevant https://review.openstack.org/#/c/179988/1 | 12:53 |
| fungi | jhesketh: looks like it! he apparently uploaded it after i crashed for the night | 12:54 |
| fungi | that's a pretty significant refactor | 12:54 |
| jhesketh | yeah and my java foo is weak | 12:54 |
| jhesketh | but I'll take a look | 12:54 |
| fungi | though a lot of it's just indents | 12:54 |
| fungi | yeah, mine too :/ | 12:54 |
| jhesketh | has anything changed that might have caused this to manifest? | 12:56 |
| AJaeger | the commit message looks fine ;) | 12:56 |
| fungi | what i find odd is that it's just really gotten obvious in the past couple weeks... i don't think we've upgraded or reconfigured anything which should have triggered it | 12:56 |
| jhesketh | yeah that's what I'm wondering | 12:56 |
| AJaeger | More load - or faster machines? | 12:56 |
| fungi | unless there were recent patches to gear or zuul which changed timing around some calls and i didn't spot them | 12:56 |
| jhesketh | yeah I was also thinking possibly related to the network issues we had a week ago | 12:57 |
| jhesketh | I try to keep an eye on those projects and don't recall anything | 12:57 |
| fungi | gear's seen no new commits since late march | 12:57 |
| fungi | even zuul's been nearly a month with nothing merged to it | 12:57 |
| *** vhoward has quit IRC | 13:00 | |
| *** NithyaG has joined #openstack-infra | 13:00 | |
| *** vhoward has joined #openstack-infra | 13:00 | |
| *** radez_g0n3 is now known as radez | 13:00 | |
| AJaeger | fungi, for some time later, a few words of wisdom are needed for https://review.openstack.org/173408 - could you push this on your review queue, please? | 13:01 |
| * AJaeger doesn't know what's the best way forward is | 13:01 | |
| fungi | hrm, jenkins03 also seems mildly unhappy | 13:01 |
| fungi | or could be this browser | 13:02 |
| *** Krinkle|detached is now known as Krinkle | 13:02 | |
| *** tiswanso has joined #openstack-infra | 13:04 | |
| *** pabelanger has joined #openstack-infra | 13:05 | |
| *** pabelanger has joined #openstack-infra | 13:05 | |
| *** armax has joined #openstack-infra | 13:05 | |
| *** eharney has joined #openstack-infra | 13:05 | |
| *** changbl has quit IRC | 13:08 | |
| *** miqui has joined #openstack-infra | 13:10 | |
| *** dustins has joined #openstack-infra | 13:11 | |
| *** sarob has joined #openstack-infra | 13:12 | |
| sdague | fungi: so... is there a flush on gearman now? or are things still stuck for a while | 13:16 |
| sdague | jhesketh: thanks for the last round of reviews on the hideci stuff | 13:16 |
| *** Krinkle is now known as Krinkle|detached | 13:17 | |
| jhesketh | you're welcome | 13:17 |
| *** BharatK has quit IRC | 13:17 | |
| mestery | Good morning infra folks! | 13:19 |
| mestery | I have a question on this review (https://review.openstack.org/#/c/179305/), and similar ones for the other neutron service repos | 13:19 |
| mestery | Looks like it's trying to merge stable/kilo into master | 13:19 |
| mestery | I have no idea why, and this is really odd. Any ideas what would trigger the bot to do this? | 13:19 |
| mestery | dhellmann: ^^^^ | 13:19 |
| mestery | I'm just going to abandon those patches I think. | 13:20 |
| claudiub | hello. I have another question: how do I create a new branch on a stackforge project? (aka stable/kilo branch) | 13:20 |
| mordred | mestery: so - we started tagging on stable rather than proposed branches this time | 13:21 |
| mestery | mordred: Ack, yes, I recall that | 13:21 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add port resource methods https://review.openstack.org/179968 | 13:21 |
| mordred | mestery: looking at your repos real quick | 13:21 |
| mestery | mordred: Thanks! | 13:21 |
| *** gyee has quit IRC | 13:22 | |
| *** bswartz has quit IRC | 13:23 | |
| *** freerunner has joined #openstack-infra | 13:23 | |
| mordred | mestery: ok. I honestly have no clue - still waking up ... fungi ^^ ? | 13:24 |
| *** sabeen1 has joined #openstack-infra | 13:24 | |
| mestery | mordred: Thanks for looking for me! I'll see what fungi may know. | 13:24 |
| mestery | mordred: For now, I put -1s on them, I'll just put -2s for now to block them until we sort this out. | 13:24 |
| mordred | or ttx may remember the status | 13:25 |
| dguerri | something funny is happening to shade functional tests... check-shade-dsvm-functional-neutron and check-shade-dsvm-functional-nova just won't start | 13:25 |
| fungi | sdague: i've deleted all the nodes which had been in a ready state for at least an hour. a lot of them are in a delete state now and we've got a lot of new ready nodes but only about 70 seem to be running jobs at the moment | 13:25 |
| *** marcusvrn has joined #openstack-infra | 13:25 | |
| *** dkranz has quit IRC | 13:26 | |
| dguerri | they have been queued for more than 6 hrs now | 13:26 |
| *** marcusvrn1 has quit IRC | 13:26 | |
| sdague | fungi: yeh, things seem to remain pretty gummed up | 13:26 |
| fungi | mestery: that's a release tag merge. you see them on all our integrated release repos every 6 months | 13:26 |
| *** bknudson has joined #openstack-infra | 13:26 | |
| *** gyee has joined #openstack-infra | 13:27 | |
| mestery | fungi: It was weird, because it looks like it's merging phantom changes, but if you say it's safe to go in, I'm with you :) | 13:27 |
| fungi | mestery: gerrit displays them oddly. checkout that merge commit and diff it against the branch tip | 13:27 |
| mestery | fungi: Cool, I'll try that! | 13:27 |
| *** dkranz has joined #openstack-infra | 13:27 | |
| *** mikeit has joined #openstack-infra | 13:27 | |
| sdague | fungi: once things get cleaned, should jobs get scheduled again, or are we going to need to do a bunch of rechecks? | 13:29 |
| *** mase_x200 has joined #openstack-infra | 13:29 | |
| sdague | trying to figure out if approving patches at this point is useful at all | 13:29 |
| fungi | sdague: they'll run. it's just right now we have very few workers picking up jobs. i'm going to start going through and quiescing jenkins masters and restarting them to see if that helps | 13:30 |
| sdague | ok | 13:30 |
| *** jtriley has joined #openstack-infra | 13:30 | |
| mestery | fungi: So just to be clear, it's ok to merge those then? | 13:31 |
| *** BharatK has joined #openstack-infra | 13:31 | |
| *** prad has joined #openstack-infra | 13:32 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Move writexml fix for older minidom to same class as caller https://review.openstack.org/180150 | 13:32 |
| fungi | mestery: yes, it's so that things like git describe will look right on master now | 13:32 |
| mestery | fungin: Ack, thanks! | 13:32 |
| fungi | mestery: it should be bringing in no code changes at all, just merging in the commit history of the 2015.1.0 tag | 13:32 |
| *** prad_ has joined #openstack-infra | 13:32 | |
| mestery | fungi: Yes, that's what it's doing, it just looked odd. I pinged dhellmann earlier, even he was confused a bit :) Thanks for clarifying! | 13:33 |
| fungi | so that git tag --contains 2015.1.0 will return both stable/kilo and master rather than just stable/kilo (because we tagged it after we branched from master) | 13:33 |
| *** signed8bit has joined #openstack-infra | 13:33 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Move writexml fix for older minidom to same module as caller https://review.openstack.org/180150 | 13:33 |
| dhellmann | oh, so we want that? it just looked weird | 13:33 |
| jeblair | fungi: pls delay restarting jenkins03 | 13:34 |
| fungi | jeblair: will do | 13:34 |
| jeblair | fungi: (i'm investigating on-host there) | 13:34 |
| fungi | jeblair: awesome. i'll start up at 07 anyway | 13:34 |
| *** prad_ has quit IRC | 13:36 | |
| *** pc_m has joined #openstack-infra | 13:36 | |
| *** BharatK has quit IRC | 13:36 | |
| *** peristeri has joined #openstack-infra | 13:36 | |
| fungi | i would just try toggling the jenkins-gearman plugin enable state but not sure if that's safe while jobs are running | 13:37 |
| *** mugsie_ is now known as mugsie | 13:38 | |
| *** jyuso1 has joined #openstack-infra | 13:39 | |
| *** signed8b_ has joined #openstack-infra | 13:39 | |
| *** signed8bit has quit IRC | 13:40 | |
| *** tonytan4ever has joined #openstack-infra | 13:40 | |
| claudiub | hello. I have another question: how do I create a new branch on a stackforge project? (aka stable/kilo branch) | 13:41 |
| *** jyuso has quit IRC | 13:41 | |
| *** mase_x200 has quit IRC | 13:42 | |
| *** Ala has quit IRC | 13:42 | |
| *** mase_x200 has joined #openstack-infra | 13:44 | |
| fungi | claudiub: make sure your gerrit account is a member of a group granted create permissions in the refs/heads/stable/* section of your repo's acl | 13:45 |
| jeblair | fungi, jhesketh, mordred, zaro, clarkb: i have a theory for the triggering event: the updates around april 28 which changed gerrit-git-prep necessitated updates to all 5500 jenkins jobs. that exceeds the puppet timeout for that exec, so it never completes. therefore each time it's triggered (by a config change) it tries to update all jobs again | 13:46 |
| *** dboik_ has quit IRC | 13:46 | |
| fungi | ooh, i didn't think to look at whether jjb is spinning constantly | 13:46 |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Make glanceclient-dsvm-functional gating https://review.openstack.org/179415 | 13:47 |
| fungi | that could be putting the api under some strain | 13:47 |
| jeblair | fungi, jhesketh, mordred, zaro, clarkb: this suggests the problem has always been present, but we're seeing it now because we are triggering those updates so often which causes registerjobs to be called more often | 13:47 |
| fungi | i can do manual jjb updates on each master to get that back under control | 13:47 |
| jeblair | fungi: yeah, i think that will help; i'm running one on 03 but nowhere else | 13:48 |
| jeblair | we also should see about increasing the timeout there... | 13:48 |
| jeblair | fungi: be sure to run yours with "time" so we know the current runtime for a full update | 13:48 |
| *** Ala has joined #openstack-infra | 13:48 | |
| fungi | will do | 13:48 |
| *** BharatK has joined #openstack-infra | 13:49 | |
| jeblair | fungi: (also, it's not constant, it only happens when project-config changes. however, that is _many_ times per day, so is "constant" for some values of "constant") | 13:49 |
| fungi | timing one starting on jenkins06 right now | 13:49 |
| *** claudiub_ has joined #openstack-infra | 13:50 | |
| jeblair | fungi: and finally, this is going to make the immediate problem worse while this runs, so we'll need to keep deleting stuck ready nodes | 13:50 |
| fungi | i am | 13:50 |
| * jeblair goes back to reviewing clarkb's patch | 13:51 | |
| fungi | i've been periodically re-firing it to clear our more ready nodes | 13:51 |
| *** mriedem has joined #openstack-infra | 13:51 | |
| fungi | thanks--i started looking at it but tend to go cross-eyed looking at java so haven't finished wrapping my head around it yet | 13:51 |
| jeblair | fungi: do you understand 179988 to supercede 178929 | 13:51 |
| jeblair | ? | 13:51 |
| *** claudiub has quit IRC | 13:51 | |
| *** yfried is now known as yfried|afk | 13:52 | |
| fungi | jeblair: yes, i believe it does. 178929 was based on a previous hunch which got ruled out but i think he hasn't remembered to abandon yet | 13:52 |
| *** bswartz has joined #openstack-infra | 13:56 | |
| *** Sukhdev has joined #openstack-infra | 13:56 | |
| *** yamahata has joined #openstack-infra | 13:56 | |
| *** e0ne is now known as e0ne_ | 13:56 | |
| *** niska has quit IRC | 13:57 | |
| claudiub_ | fungi: I see. So after than, I just git push gerrit stable/kilo? | 13:57 |
| claudiub_ | after that * | 13:57 |
| claudiub_ | (after git checkout -b stable/kilo) | 13:58 |
| *** fawadkhaliq has joined #openstack-infra | 13:59 | |
| *** e0ne_ is now known as e0ne | 13:59 | |
| jhesketh | jeblair: good find | 13:59 |
| jhesketh | is there anything I can do to help you guys? | 13:59 |
| *** e0ne is now known as e0ne_ | 13:59 | |
| *** e0ne_ is now known as e0ne | 14:00 | |
| fungi | claudiub_: then you should be able to create branches in the gerrit webui by browsing to projects->list->(filter for your repo name)->branches | 14:00 |
| *** e0ne is now known as e0ne_ | 14:00 | |
| *** e0ne_ is now known as e0ne | 14:00 | |
| anteaya | jhesketh: my motivation for my earlier ping was to get eyes on this fire when it started | 14:01 |
| anteaya | as I couldn't do anything myself | 14:01 |
| *** wenlock has joined #openstack-infra | 14:01 | |
| fungi | claudiub_: due to quirks of gerrit acls i think pushing to create a branch requires push --force permission, which starts to get into foot-cannon territory | 14:01 |
| *** ildikov has quit IRC | 14:01 | |
| jhesketh | anteaya: ah okay, sorry I wasn't around (although not sure I could have helped much anwyay0 | 14:02 |
| anteaya | jhesketh: you can't _always_ be around, I took a chance | 14:02 |
| anteaya | :) | 14:02 |
| *** pcm_ has joined #openstack-infra | 14:02 | |
| *** ayoung has joined #openstack-infra | 14:03 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:03 | |
| anteaya | moral support counts though | 14:03 |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to networking-hyperv https://review.openstack.org/180160 | 14:03 |
| *** niska has joined #openstack-infra | 14:05 | |
| claudiub_ | fungi: I see, thanks for the info! :D | 14:05 |
| *** pc_m has quit IRC | 14:06 | |
| *** zz_jgrimm is now known as jgrimm | 14:06 | |
| *** esker has joined #openstack-infra | 14:06 | |
| *** craigbr has joined #openstack-infra | 14:06 | |
| *** ajmiller has joined #openstack-infra | 14:07 | |
| *** dboik has joined #openstack-infra | 14:09 | |
| *** dboik has quit IRC | 14:09 | |
| mordred | jeblair: ah! that explains why we're seeing it all of a sudden | 14:10 |
| *** dboik has joined #openstack-infra | 14:10 | |
| *** gokrokve has joined #openstack-infra | 14:10 | |
| *** yfried|afk is now known as yfried | 14:12 | |
| *** mpavone has joined #openstack-infra | 14:13 | |
| *** otter768 has joined #openstack-infra | 14:14 | |
| *** asselin has joined #openstack-infra | 14:15 | |
| *** sabeen1 has quit IRC | 14:15 | |
| *** fhubik is now known as fhubik_afk | 14:16 | |
| *** ildikov has joined #openstack-infra | 14:17 | |
| *** freerunner has quit IRC | 14:18 | |
| fungi | clarkb: when you're around, are you done poking at jenkins01? can i restart it and bring it back into operation now? | 14:18 |
| *** armax has quit IRC | 14:18 | |
| *** otter768 has quit IRC | 14:18 | |
| *** lifeless has quit IRC | 14:20 | |
| *** wenlock_ has joined #openstack-infra | 14:20 | |
| * jhesketh heads off for some zzz's before the meeting | 14:21 | |
| *** tosky has joined #openstack-infra | 14:22 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Use plumbing rev-parse to get the branch name https://review.openstack.org/179937 | 14:22 |
| openstackgerrit | Alexandre Viau proposed openstack-infra/project-config: New stackforge project: Bansho https://review.openstack.org/180166 | 14:23 |
| *** panda has joined #openstack-infra | 14:23 | |
| jeblair | clarkb: 179988 reviewed; i think it has a problem, but do agree the refactor is clearer. :) | 14:24 |
| panda | what is the correct contact to notify a problem in gerrit ? | 14:25 |
| jeblair | also, reviewing java takes 2x energy points | 14:25 |
| jeblair | panda: right here | 14:25 |
| *** lifeless has joined #openstack-infra | 14:27 | |
| panda | jeblair: ok | 14:27 |
| *** dangers_away is now known as dangers | 14:28 | |
| panda | I'm trying to update offline contact information, but I get a "server error: cannot store contact information" | 14:28 |
| * mordred hands jeblair a box of extra energy points | 14:29 | |
| *** mrmartin has quit IRC | 14:29 | |
| *** annegentle has joined #openstack-infra | 14:29 | |
| jeblair | panda: see the instructions here: http://docs.openstack.org/infra/manual/developers.html#account-setup | 14:30 |
| jeblair | panda: in particular, make sure your email address matches your foundation individual membership | 14:30 |
| *** armax has joined #openstack-infra | 14:31 | |
| Shrews | jeblair: I still haven't recovered my lost energy points from writing the original WIP feature for gerrit | 14:31 |
| * Shrews shudders at the memory | 14:31 | |
| *** mtanino has joined #openstack-infra | 14:32 | |
| *** aufi has quit IRC | 14:32 | |
| *** maishsk is now known as maishsk_afk | 14:33 | |
| *** amitgandhinz has joined #openstack-infra | 14:33 | |
| jeblair | Shrews: you might get them back in gerrit 2.11 which supports the WIP plugin :) | 14:34 |
| SergeyLukjanov | fungi, good morning, so, for the today's odd zuul behavior nodepool delete all ready slaves is the temp solution? | 14:34 |
| *** nelsnelson has joined #openstack-infra | 14:34 | |
| fungi | SergeyLukjanov: workaround, yes | 14:35 |
| *** fhubik_afk is now known as fhubik | 14:35 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jenkins: Add option to send workers on jjb https://review.openstack.org/180173 | 14:35 |
| fungi | SergeyLukjanov: though as jeblair pointed out we're rerunning and then killing jenkins-jobs update over and over on the masters too, so that's probably what's causing it to expose this race so much recently. cleaning that up ought to limp us along while clarkb's jenkins0gearman fix gets polished | 14:36 |
| anteaya | I'm seeing more jobs running, gate queue doesn't have all the nodes it wants yet so check won't have any nodes available for a bit | 14:36 |
| *** mattfarina has joined #openstack-infra | 14:36 | |
| fungi | this is going to be a slow recovery but i suspect in the next few hours we'll have crawled back to normal throughput again | 14:36 |
| anteaya | wonderful | 14:37 |
| anteaya | do you have to constantly restart jenkinses today? | 14:38 |
| anteaya | jenkinsi? | 14:38 |
| anteaya | I still don't know the plural form | 14:38 |
| *** btully has joined #openstack-infra | 14:38 | |
| openstackgerrit | Alexandre Viau proposed openstack-infra/project-config: New stackforge project: Bansho https://review.openstack.org/180166 | 14:39 |
| anteaya | all the gate patches have some jobs running now, hopefully the check queue will start to get nodes soon | 14:39 |
| fungi | first declension nominative, jenkinsae | 14:40 |
| anteaya | ah | 14:40 |
| anteaya | never would have arrived at that | 14:40 |
| anteaya | thank goodness for your latin | 14:40 |
| * anteaya looks up declension | 14:40 | |
| *** emagana has joined #openstack-infra | 14:40 | |
| * fungi promises that will be his only latin grammar joke for today | 14:40 | |
| * anteaya bets $5 it won't be | 14:41 | |
| fungi | itym $v | 14:41 |
| fungi | (didn't promise to obstain from roman number jokes on the other hand) | 14:42 |
| fungi | abstain either | 14:42 |
| *** hichihara has joined #openstack-infra | 14:42 | |
| *** dboik_ has joined #openstack-infra | 14:42 | |
| *** gokrokve_ has joined #openstack-infra | 14:43 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Use 'http' to download war to allow urlopen use proxies https://review.openstack.org/180178 | 14:43 |
| anteaya | ha ha ha | 14:43 |
| *** sdake_ has joined #openstack-infra | 14:43 | |
| sigmavirus24 | Uhh | 14:44 |
| anteaya | anyone have a sanskrit comment? | 14:45 |
| *** dboik has quit IRC | 14:45 | |
| Daviey | caput inter nubilia condo | 14:46 |
| *** gokrokve has quit IRC | 14:46 | |
| anteaya | that's sanskrit? | 14:46 |
| anteaya | the stars in the apartment have gone out | 14:47 |
| *** sdake has quit IRC | 14:47 | |
| *** hasharAway is now known as hashar | 14:47 | |
| *** sputnik13 has joined #openstack-infra | 14:48 | |
| *** coolsvap is now known as coolsvap|afk | 14:48 | |
| panda | jeblair: solved, thanks. (at the time of my first contribution foundation membership was not required) | 14:50 |
| *** sputnik13 has quit IRC | 14:50 | |
| jeblair | mordred: 175891 left me confused; see comment. | 14:52 |
| clarkb | jeblair: so I removed the comment about sleeping because in this case it goes directly on the queue that work() processes so should be handled 100% of the time. I believe that is what we want for the post initialize and post work fail due to unknown function cases but not the post register functions case | 14:53 |
| clarkb | jeblair: I can switch post register back to using the session event queueing which will check the state, and keep the other two as a regular enqueue | 14:53 |
| *** hdd__ has quit IRC | 14:54 | |
| *** hdd__ has joined #openstack-infra | 14:54 | |
| *** mattfarina has quit IRC | 14:55 | |
| zxiiro | for a matrix job http://ci.openstack.org/jenkins-job-builder/project_matrix.html | 14:55 |
| zxiiro | is it possible to pass a variable to the value? | 14:55 |
| jeblair | clarkb: oh, then the other half of the comment is incorrect because it says that it's processed by the gearman session. in that case, we'll end up sending lots of grab_job packets which means we'll get extra assigned jobs. | 14:55 |
| *** notnownikki has quit IRC | 14:55 | |
| *** mattfarina has joined #openstack-infra | 14:55 | |
| zxiiro | i want my "project" definition to pass a variable containing the values | 14:55 |
| jd__ | if someone has 5 minutes, it'd be cool to git branch -d 1.0 in openstack/gnocchi which I created by mistake | 14:55 |
| jd__ | no urgency though :) | 14:55 |
| clarkb | jeblair: ya so I think we need both cases, one for when we first init and post fail, and the other for the general hey we updated, do a grab job if necessary | 14:56 |
| jeblair | clarkb: ++ | 14:56 |
| *** hdd__ has quit IRC | 14:56 | |
| jeblair | clarkb: (agreed; i can't think of a way to combine them, so will need minor duplication) | 14:56 |
| *** fhubik has quit IRC | 14:56 | |
| anteaya | and a check job just got a node | 14:57 |
| anteaya | progress | 14:57 |
| *** imcsk8 has joined #openstack-infra | 14:57 | |
| *** abregman has quit IRC | 14:58 | |
| *** Sukhdev has quit IRC | 14:58 | |
| *** reed_ has joined #openstack-infra | 14:59 | |
| openstackgerrit | Jerome Hourquebie proposed stackforge/python-jenkins: Adds Cloudbees folder plugin support This patch adds cloudbees folder plugin support. Folder should be specified in job name : <folder>/<job> This update also extend job_builder (http://ci.openstack.org/jenkins-job-builder/) tool without any modification https://review.openstack.org/180185 | 15:00 |
| *** sdake_ is now known as sdake | 15:01 | |
| *** achanda has joined #openstack-infra | 15:01 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering https://review.openstack.org/179988 | 15:03 |
| clarkb | jeblair: something like ^ | 15:03 |
| *** hichihara has quit IRC | 15:03 | |
| *** vhoward has left #openstack-infra | 15:03 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to compute-hyperv https://review.openstack.org/180187 | 15:04 |
| *** erikmwilson is now known as Guest32970 | 15:04 | |
| *** erikmwil_ has joined #openstack-infra | 15:04 | |
| *** Guest32970 has quit IRC | 15:04 | |
| *** erikmwil_ is now known as erikmwilson | 15:04 | |
| *** erikmwilson_ has joined #openstack-infra | 15:04 | |
| *** mpavone has quit IRC | 15:04 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support copyartifact build selector param https://review.openstack.org/133774 | 15:05 |
| *** achanda has quit IRC | 15:06 | |
| claudiub_ | AJaeger: Hello, If I may disturb you once more, could you take a look at another 2 commits for project-config? It adds rights for creating stable branches on networking-hyperv and compute-hyperv: https://review.openstack.org/#/c/180160/ https://review.openstack.org/#/c/180187/ Thanks! | 15:06 |
| AJaeger | claudiub_, will do later... | 15:08 |
| jaypipes | clarkb: is something up with etherpad.openstack.org or is it just me? Keeps reconnecting... | 15:08 |
| claudiub_ | AJaeger: ok. :) | 15:08 |
| clarkb | jaypipes: it can do that if you have the same pad open in multiple tabs, I would check that first, if that isn't the case let me know what our etherpad is and I can check logs | 15:09 |
| anteaya | jaypipes: what browser are you using? | 15:09 |
| jaypipes | ah, clarkb that may be it... | 15:09 |
| jaypipes | anteaya: chromium | 15:09 |
| *** mrunge has quit IRC | 15:09 | |
| anteaya | really, I never had a problem with chromium | 15:10 |
| *** shashankhegde has joined #openstack-infra | 15:10 | |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Make glanceclient-dsvm-functional gating https://review.openstack.org/179415 | 15:12 |
| *** jaypipes has quit IRC | 15:13 | |
| *** dboik_ has quit IRC | 15:13 | |
| *** gyee has quit IRC | 15:13 | |
| *** tonytan4ever has quit IRC | 15:13 | |
| *** dboik has joined #openstack-infra | 15:13 | |
| *** mase_x200 has quit IRC | 15:13 | |
| *** abregman has joined #openstack-infra | 15:13 | |
| *** gyee has joined #openstack-infra | 15:14 | |
| zxiiro | Figured it out. the answer is to use {obj:value} | 15:15 |
| *** asselin has quit IRC | 15:15 | |
| claudiub_ | sdague: Hello. If it's not much to ask, could you take a look at a few project-config commits? They're a few, only 1-5 lines long, similar in purpose, etc. Thanks! https://review.openstack.org/#/q/status:open+owner:%22Claudiu+Belu%22+project:openstack-infra/project-config,n,z | 15:16 |
| *** yfried is now known as yfried|afk | 15:18 | |
| clarkb | fungi: are you ok with holding 01 a bit longer? I want to get a thread dump from it, may try to fix the bug in that plugin now that I am writing java anyways | 15:18 |
| clarkb | I am going to toggle these ready nodes on jenkins nodes by disconnect/connect ing them | 15:19 |
| *** arxcruz has quit IRC | 15:21 | |
| *** claudiub_ has quit IRC | 15:21 | |
| clarkb | maybe, that doesn't seem to have done much for a bare-(centos6|precise|trusty) node | 15:22 |
| *** Ala has quit IRC | 15:24 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Switch to requests to support proxying of 'https' https://review.openstack.org/180178 | 15:25 |
| fungi | clarkb: yeah, i haven't done anything with 01 other than disabling puppet agent and starting an update of its job configs | 15:26 |
| *** mpaolino has joined #openstack-infra | 15:26 | |
| *** nelsnelson has quit IRC | 15:26 | |
| fungi | which already completed. i've reenabled puppet agent on it now | 15:27 |
| fungi | jeblair: point of interest, on jenkins01 where the service was already quiesced, wall clock time for a full jjb update was a mere 9m14.676s | 15:27 |
| fungi | the rest are taking way, waaaay longer | 15:28 |
| clarkb | cool, going to take a thread dump now will let you know when that is done | 15:28 |
| clarkb | fungi: likely because it doesn't register any jobs when in shutdown mode | 15:28 |
| *** abregman has quit IRC | 15:28 | |
| *** shashankhegde has quit IRC | 15:28 | |
| fungi | oh, great point | 15:28 |
| clarkb | when we do it with gearman enabled its a bunch of network data | 15:28 |
| jeblair | fungi: gtk | 15:28 |
| *** freerunner has joined #openstack-infra | 15:28 | |
| fungi | i'll fire off a second one later when we restart the service | 15:28 |
| jeblair | clarkb: yeah, the regstration churn is huge | 15:28 |
| *** yfried|afk is now known as yfried | 15:29 | |
| *** shashankhegde has joined #openstack-infra | 15:29 | |
| *** Swami has joined #openstack-infra | 15:29 | |
| *** yamahata has quit IRC | 15:30 | |
| clarkb | jeblair: ya when I was tcpdumping I noticed that our window size would fall to zero regularly | 15:30 |
| clarkb | looked like registraration was to blame as we essentially fill the pipe for a bit | 15:30 |
| anteaya | so for anyone who cares, I just downloaded firefox 37.0.2 and don't have a reconnection problem so far with one tab in firefox and one tab in chromium at the same time | 15:31 |
| anteaya | reconnected if I open a second tab in firefox though, which I expected | 15:32 |
| *** dims_ has joined #openstack-infra | 15:32 | |
| *** dboik has quit IRC | 15:32 | |
| *** dizquierdo has quit IRC | 15:33 | |
| anteaya | this is etherpad, sorry I didn't say that | 15:33 |
| *** mpaolino has quit IRC | 15:34 | |
| *** dims has quit IRC | 15:34 | |
| clarkb | anteaya: thats good news, so possibly related to 37.0.1? | 15:34 |
| clarkb | fungi: I have the thread dump now, feel free to restart 01 | 15:34 |
| fungi | clarkb: thanks. doing! | 15:35 |
| anteaya | clarkb: that is the idea I'm tying myself to for the moment | 15:35 |
| anteaya | clarkb: only one datapoint so weak right now | 15:36 |
| anteaya | no idea how to explain the safari and chromium reconnections | 15:36 |
| *** dannywilson has joined #openstack-infra | 15:36 | |
| anteaya | unless jaypipes did have more than one chromium tab open for the same etherpad | 15:36 |
| openstackgerrit | Sergey Kolekonov proposed openstack-infra/project-config: Add apps-catalog/ci group for apps-catalog project https://review.openstack.org/180201 | 15:36 |
| *** maishsk_afk is now known as maishsk | 15:37 | |
| *** craigbr has quit IRC | 15:37 | |
| *** freerunner has quit IRC | 15:37 | |
| *** nelsnelson has joined #openstack-infra | 15:37 | |
| *** sdake has quit IRC | 15:37 | |
| *** freerunner has joined #openstack-infra | 15:38 | |
| *** yfried is now known as yfried|afk | 15:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 15:40 |
| *** samuelBartel has quit IRC | 15:40 | |
| *** sdake has joined #openstack-infra | 15:40 | |
| *** mwagner_lap has quit IRC | 15:40 | |
| *** dannywilson has quit IRC | 15:41 | |
| *** abregman has joined #openstack-infra | 15:41 | |
| *** dannywilson has joined #openstack-infra | 15:42 | |
| *** freerunner has quit IRC | 15:42 | |
| mordred | jeblair: thank you- very helpful feedback - I added docstrings, renamed the param so that the boolean isn't confusing, and added unittests for it | 15:43 |
| anteaya | the 37.0.2 release notes leave me with the feeling I'm missing something: https://www.mozilla.org/en-US/firefox/37.0.2/releasenotes/ | 15:44 |
| mordred | SpamapS: if you want to take a look at that ^^ and tell me how I should write the tests with mock instead of monkeypatch, I'd love to learn | 15:44 |
| *** reed_ has quit IRC | 15:45 | |
| *** changbl has joined #openstack-infra | 15:45 | |
| *** dboik has joined #openstack-infra | 15:47 | |
| *** hemnafk is now known as hemna | 15:48 | |
| hemna | hey guys, any chance I can get someone to +A this guy for me today? https://review.openstack.org/#/c/177372/ | 15:48 |
| *** jcoufal has quit IRC | 15:48 | |
| *** smccully has joined #openstack-infra | 15:49 | |
| *** yamahata has joined #openstack-infra | 15:51 | |
| *** freerunner has joined #openstack-infra | 15:53 | |
| *** maishsk has quit IRC | 15:53 | |
| *** jlanoux has quit IRC | 15:54 | |
| *** whoops has joined #openstack-infra | 15:54 | |
| *** armax has quit IRC | 15:54 | |
| *** tosky has quit IRC | 15:55 | |
| anteaya | can someone else look at the gate please? http://status.openstack.org/zuul/ | 15:56 |
| anteaya | why didn't the gate reset on that devstack failure? | 15:56 |
| anteaya | https://review.openstack.org/#/c/173593/ | 15:56 |
| anteaya | the whole gate is finished and I had been thinking it is calculating a reset | 15:57 |
| cinerama | AJaeger, around? | 15:57 |
| anteaya | but it has been for some minutes already | 15:57 |
| *** bhunter71 has joined #openstack-infra | 15:58 | |
| *** scheuran has quit IRC | 15:58 | |
| *** MarkAtwood has joined #openstack-infra | 15:58 | |
| *** yfried|afk is now known as yfried | 15:59 | |
| anteaya | gerrit is still broadcasting events | 15:59 |
| *** rloo has joined #openstack-infra | 15:59 | |
| *** tosky has joined #openstack-infra | 16:00 | |
| fungi | rerunning a full jjb update on jenkins01 a few minutes after it was restarted took 2m36.326s wall clock time, though that was after it had already been previously caught up | 16:00 |
| anteaya | fungi: can you take a look at teh gate queue? | 16:01 |
| *** chlong has joined #openstack-infra | 16:01 | |
| fungi | anteaya: yes, i have been since you mentioned it | 16:01 |
| anteaya | ah thanks | 16:01 |
| fungi | looks like events and results are slowly climbing | 16:01 |
| anteaya | ah | 16:01 |
| anteaya | okay | 16:01 |
| *** dboik_ has joined #openstack-infra | 16:01 | |
| fungi | which may still indicate a problem | 16:01 |
| anteaya | oh | 16:01 |
| anteaya | I don't understand why the whole gate finished, it should have reset at the first failure | 16:02 |
| *** mikeit has quit IRC | 16:02 | |
| gordc | ... is bob hansen a bot? | 16:02 |
| anteaya | gordc: what channel? | 16:02 |
| jeblair | clarkb: have you run your latest patch? | 16:02 |
| *** ildikov has quit IRC | 16:03 | |
| gordc | anteaya: https://review.openstack.org/#/q/reviewer:hansenb%2540us.ibm.com+status:open,n,z | 16:03 |
| *** tonytan4ever has joined #openstack-infra | 16:03 | |
| gordc | anteaya: seems like it's mimicing jenkins. | 16:03 |
| *** freerunner has quit IRC | 16:03 | |
| anteaya | gordc: yup, thanks for letting me know, will get that account disabled | 16:03 |
| anteaya | that kind of activity is verboten | 16:04 |
| *** dboik has quit IRC | 16:04 | |
| *** hashar is now known as hasharMeeting | 16:04 | |
| *** freerunner has joined #openstack-infra | 16:04 | |
| gordc | anteaya: ok, thanks! i wasn't sure what was happening. seems to be tracking a bunch of recent patches | 16:04 |
| clarkb | jeblair: not beyond `mvn package` which builds it then runs the built in tests. Seems to not be happy with the test start stop thread test, but I thought it was likely a race (traceback isn't too useful) | 16:04 |
| *** craigbr has joined #openstack-infra | 16:05 | |
| clarkb | still waking up, need to go make breakfast soon, but will get back to it after that | 16:05 |
| anteaya | gordc: thanks for asking, humans only to comment on human accounts, we have lots of pathways for bots and that isn't one of them | 16:05 |
| *** craigbr has quit IRC | 16:05 | |
| *** freerunner has quit IRC | 16:05 | |
| rloo | thanks for handling Bob Hansen, I was wondering about it! :) | 16:05 |
| jeblair | clarkb: okay, i'm not sure there are any _actual_ tests on this; last time i used it, i ran it along with a geard and a simple script to trigger jobs | 16:06 |
| jeblair | clarkb: i think that might be worthwhile; i'm starting to feel like a java runtime :) | 16:06 |
| anteaya | can someone please disable gerrit account 15674: http://paste.openstack.org/show/215011/ | 16:06 |
| anteaya | rloo: thanks :) | 16:07 |
| *** ddieterly has joined #openstack-infra | 16:07 | |
| *** freerunner has joined #openstack-infra | 16:08 | |
| mordred | jeblair: yay for no tests | 16:08 |
| ddieterly | is zuul sick? | 16:09 |
| anteaya | ddieterly: yes | 16:09 |
| ddieterly | ok, thanks | 16:10 |
| anteaya | welcome | 16:10 |
| fungi | watching its logs, it seems to be starting jobs and getting results | 16:10 |
| fungi | the server itself is not loaded heavily | 16:10 |
| anteaya | we hope in a few hours to have a better flow, if we can catch up | 16:10 |
| fungi | the zuul-merger workers are basically idle | 16:10 |
| anteaya | :( | 16:10 |
| anteaya | mordred: have you disabled a gerrit account before? | 16:11 |
| anteaya | mordred: would you like to try with gerrit account id 15674? | 16:11 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to convert append only tables to MyISAM https://review.openstack.org/179964 | 16:11 |
| fungi | the results count seems to be catching up some now | 16:11 |
| mtreinish | mordred: ^^^ I'm not sure how bad that is | 16:11 |
| mtreinish | I just played copy and paste with the one we put together to add the column | 16:12 |
| mordred | mtreinish: oh, crappit. I forgot to look at your thing last night | 16:12 |
| mordred | anteaya: I'm in a meeting thing, so doing things that require attention is a bad idea | 16:12 |
| mtreinish | mordred: no worries, it's not super important | 16:12 |
| anteaya | mordred: ah yes, a bad idea okay thanks | 16:12 |
| mtreinish | we still ahve plenty of space on the db (I think) | 16:13 |
| *** unicell has quit IRC | 16:13 | |
| *** dguitarbite has quit IRC | 16:14 | |
| *** ashleighfarnham has joined #openstack-infra | 16:14 | |
| *** bknudson has quit IRC | 16:14 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-zuul: Attempt to add an acceptance test with beaker for puppet-zuul https://review.openstack.org/180214 | 16:14 |
| *** otter768 has joined #openstack-infra | 16:15 | |
| fungi | jeblair: clarkb: seeing quite a few of these tracebacks in the zuul log http://paste.openstack.org/show/215013/ collecting stats now to see if it's recent and to gauge their frequency | 16:15 |
| jeblair | clarkb: okay, i think i found 2 problems with simple solutions | 16:15 |
| anteaya | pleia2: let me know if you are around to disable gerrit account id 15674 | 16:16 |
| fungi | looks like we normally get a few a day, but today we have ~500 and counting | 16:16 |
| EmilienM | nibalizer: you may want to see https://review.openstack.org/180214 | 16:16 |
| *** dboik_ has quit IRC | 16:17 | |
| anteaya | fungi: that is a few more than usual | 16:17 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Reorder imports to match hacking guidelines https://review.openstack.org/133465 | 16:17 |
| *** dboik has joined #openstack-infra | 16:17 | |
| cinerama | pleia2: hi | 16:17 |
| fungi | looks like it climbed steadily from when the logs rotated at 07:00 to present time | 16:17 |
| fungi | so possibly related to the bug under discussion | 16:18 |
| *** chlong has quit IRC | 16:18 | |
| *** pblaho has quit IRC | 16:18 | |
| anteaya | jeblair: I'm curious about your problems and solutions | 16:18 |
| *** rloo has quit IRC | 16:18 | |
| anteaya | unless you want to wait for clarkb | 16:18 |
| anteaya | the gate seems stuck | 16:19 |
| jeblair | anteaya: feel free to look at the patch if you are curious | 16:19 |
| anteaya | jeblair: ah okay thanks | 16:19 |
| anteaya | fungi: do you think the gate will move at some point? | 16:19 |
| *** otter768 has quit IRC | 16:20 | |
| zaro | morning | 16:20 |
| anteaya | morning zaro | 16:21 |
| anteaya | zaro: you know java | 16:21 |
| fungi | anteaya: at some point, yes. not sure yet whether the current situation is slowing things down to the point where zuul is having trouble handling its event queue as quickly as it's being filled or what. it does seem to be logging that new jobs are starting and running jobs are completing | 16:21 |
| anteaya | zaro: this patch could use your review: https://review.openstack.org/#/c/179988/2 | 16:21 |
| zaro | anteaya: thanks. i'm aware of the problem. going thru backlog now. | 16:22 |
| anteaya | zaro: k | 16:22 |
| *** freerunner has quit IRC | 16:22 | |
| anteaya | fungi: the gate hasn't move in over 20 minutes | 16:22 |
| mordred | anteaya: yah - it's a bit of a bad one / all hands on deck to solve it | 16:22 |
| fungi | anteaya: its waiting job queue was up over 2500 earlier and that has been steadily dropping (almost down to 1000 now) | 16:22 |
| anteaya | fungi: okay | 16:23 |
| *** isviridov is now known as isviridov_away | 16:23 | |
| *** annegentle has quit IRC | 16:23 | |
| anteaya | fungi: I guess I can't see that number | 16:23 |
| fungi | anteaya: graph at the bottom of the status page | 16:23 |
| fungi | also in graphite | 16:23 |
| *** maishsk has joined #openstack-infra | 16:23 | |
| *** sputnik13 has joined #openstack-infra | 16:23 | |
| fungi | i have a feeling the gate reset processing is slow because of sheer size of the backlog we accumulated | 16:24 |
| anteaya | I'll try my hand at graphite | 16:24 |
| anteaya | fungi: okay | 16:24 |
| *** annegentle has joined #openstack-infra | 16:25 | |
| *** jistr has quit IRC | 16:26 | |
| *** maishsk has quit IRC | 16:26 | |
| fungi | looks like we haven't turned zuul status queries back on in hideci.js, so i don't think it's that | 16:26 |
| *** salv-orlando has quit IRC | 16:27 | |
| *** salv-orlando has joined #openstack-infra | 16:27 | |
| *** annegent_ has joined #openstack-infra | 16:27 | |
| *** annegentle has quit IRC | 16:28 | |
| jeblair | fungi: those jjb runs still continuing? | 16:28 |
| jeblair | fungi: because i think geard is cpu bound at this point | 16:28 |
| anteaya | fungi: thank you now I can see the numbers you see | 16:28 |
| *** baoli has quit IRC | 16:29 | |
| jeblair | fungi: mine from earlier is still chugging along very slowly | 16:29 |
| fungi | jeblair: yep, all but 01 which ran while jenkins was stopped and then very quickly reran once it was restarted | 16:29 |
| *** tiswanso has quit IRC | 16:30 | |
| fungi | ~9m to run while jenkins01 was quiesced, and ~2 minutes to run again once it was started back | 16:30 |
| *** alexsyip has joined #openstack-infra | 16:31 | |
| *** dboik_ has joined #openstack-infra | 16:31 | |
| *** tiswanso has joined #openstack-infra | 16:31 | |
| *** baoli_ has joined #openstack-infra | 16:31 | |
| fungi | jeblair: should we stop all but one, leave puppet disabled on them, and then update one at a time to completion before starting the next? | 16:31 |
| *** tiswanso_ has joined #openstack-infra | 16:31 | |
| fungi | that would keep us from beating up the geard quite so badly | 16:31 |
| *** annegent_ has quit IRC | 16:33 | |
| jeblair | fungi: not a bad idea; and you can do that without interrupting the process. i'd probably do 2 at a time. | 16:33 |
| jeblair | or 4. | 16:33 |
| fungi | jeblair: well, we have 6 underway, i can kill 3 of mine for noew | 16:34 |
| *** dboik has quit IRC | 16:34 | |
| jeblair | ok | 16:34 |
| yolanda | here, can i get some help? i need to know the meanings of the zuul times in status.openstack.org/zuul | 16:35 |
| yolanda | first number is the estimated one, second one is the total elapsed? | 16:35 |
| mordred | yolanda: hey - in the middle of a bit of an outage - might be a few minutes | 16:35 |
| *** tonytan4ever has quit IRC | 16:35 | |
| yolanda | ok | 16:35 |
| *** tiswanso has quit IRC | 16:35 | |
| fungi | 93m elapsed on jenkins07, 112m elapsed on jenkins02, 116m elapsed on jenkins04, now all stopped. still running on jenkins03 (yours), jenkins05 and jenkins06 | 16:36 |
| jeblair | yolanda: yes, the second is time in queue | 16:36 |
| yolanda | jeblair, it includes the waiting time as well? | 16:36 |
| jeblair | yolanda: yep | 16:36 |
| yolanda | thx | 16:36 |
| *** jaypipes has joined #openstack-infra | 16:38 | |
| fungi | i'll resume 02, 04 and 07 once the others complete | 16:38 |
| *** craigbr has joined #openstack-infra | 16:38 | |
| jeblair | fungi: i accidentally restarted geard on zuul. | 16:38 |
| fungi | oh, oops | 16:39 |
| fungi | is that going to need a scheduler restart as well? | 16:39 |
| jeblair | fungi: nah | 16:39 |
| *** gokrokve_ has quit IRC | 16:39 | |
| fungi | okay, just making sure | 16:39 |
| *** ivar-lazzaro has joined #openstack-infra | 16:41 | |
| *** ivar-lazzaro has quit IRC | 16:41 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:42 | |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/project-config: Make neutron-dvr job voting at the gate https://review.openstack.org/180230 | 16:42 |
| jogo | is launchpad timing out for anyone else? | 16:43 |
| nibalizer | fungi: jeblair could I have a branch on openstack-infra/publications called 'puppet-overview' or similar ? | 16:43 |
| anteaya | https://launchpad.net/ renders for me | 16:43 |
| anteaya | jogo: do you mean login? | 16:43 |
| jogo | anteaya: filing a bug errors out | 16:43 |
| nibalizer | I have a deck from a talk I gave last night on the subject | 16:43 |
| *** tim_o has joined #openstack-infra | 16:43 | |
| fungi | nibalizer: branched from the tip of template? | 16:44 |
| anteaya | jogo: just logged into the wiki | 16:44 |
| anteaya | jogo: ah okay | 16:44 |
| nibalizer | fungi: yes | 16:44 |
| *** freerunner has joined #openstack-infra | 16:44 | |
| fungi | anteaya: gerrit account 15674 is now disabled | 16:45 |
| jogo | anteaya: I think it was just launchpad failing as usual | 16:45 |
| *** SumitNaiksatam has quit IRC | 16:45 | |
| fungi | nibalizer: done | 16:45 |
| anteaya | fungi: thank you | 16:45 |
| *** sdake_ has joined #openstack-infra | 16:45 | |
| *** unicell has joined #openstack-infra | 16:45 | |
| fungi | yw | 16:46 |
| *** Guest21373 has quit IRC | 16:46 | |
| anteaya | jogo: ah well then, don't let me interrupt your usual fun | 16:46 |
| *** craigbr has quit IRC | 16:46 | |
| nibalizer | fungi: thanks | 16:46 |
| anteaya | I expect that account to still be commenting for a while since we don't have close connection yet, but at least it is disabled now | 16:46 |
| anteaya | and we will soon have close connection | 16:46 |
| *** freerunner has quit IRC | 16:46 | |
| jeblair | fungi: i'm not sure my geard restart is happy; i'm thinking of doing a zuul restart | 16:46 |
| *** freerunner has joined #openstack-infra | 16:47 | |
| anteaya | gordc rloo if you are able to check https://review.openstack.org/#/q/reviewer:hansenb%2540us.ibm.com+status:open,n,z tomorrow and let me know if you see anymore bot activity from bob | 16:47 |
| *** freerunner has quit IRC | 16:47 | |
| fungi | jeblair: okay. it does look like zuul's job backlog has leveled off rather than continued dropping. i guess we're going to lose the last ~1.75 hours of gerrit events | 16:47 |
| anteaya | gordc rloo the connection may still be open until it expires but it should be expired by tomorrow | 16:48 |
| jeblair | fungi: oh that's weird, it just bursted | 16:48 |
| *** AJaeger_ has joined #openstack-infra | 16:48 | |
| *** sdake has quit IRC | 16:49 | |
| clarkb | ok back, I have cookies and hot tea | 16:51 |
| anteaya | clarkb: perfect for firefighting | 16:52 |
| fungi | jeblair: looks like you went ahead and restarted the scheduler daemon? | 16:52 |
| jeblair | ya | 16:52 |
| *** maishsk_ has joined #openstack-infra | 16:52 | |
| *** cody-somerville has quit IRC | 16:52 | |
| *** spzala has joined #openstack-infra | 16:52 | |
| *** doude has quit IRC | 16:52 | |
| anteaya | check is empty (well 2 now) but gate is still the same | 16:52 |
| *** maishsk_ is now known as maishsk | 16:52 | |
| *** doude has joined #openstack-infra | 16:52 | |
| fungi | not any longer | 16:53 |
| jeblair | fungi: i put jenkins 03,05,06 in shutdown mode | 16:54 |
| fungi | jeblair: okay, thanks. that will likely help speed this up | 16:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Puppet: don't run beaker jobs on previous stable branches https://review.openstack.org/180233 | 16:54 |
| fungi | i'll delete all their ready nodes | 16:54 |
| anteaya | the gate just restarted but it didn't move anything | 16:54 |
| jeblair | fungi: thanks | 16:54 |
| anteaya | the top four patches are still the same as they were | 16:55 |
| anteaya | so I'm guessing the rest are too | 16:55 |
| clarkb | jeblair: I agree with your comments, what do you think about pulling the driveSessionIO out of submitFunction instead of using a continue after running the function? | 16:55 |
| *** markus_z has quit IRC | 16:56 | |
| clarkb | also feel free it ignore me while fires are fought | 16:56 |
| *** dustins has quit IRC | 16:56 | |
| *** mattfarina has quit IRC | 16:57 | |
| jeblair | clarkb: that changes the ordering related to availability unlocking; unsure if that's important | 16:57 |
| *** mattfarina has joined #openstack-infra | 16:57 | |
| *** mattfarina has quit IRC | 16:57 | |
| *** BharatK has quit IRC | 16:57 | |
| AJaeger_ | cinerama: I'm around now | 16:57 |
| clarkb | jeblair: good point, probably easiest for now to do what you suggest | 16:58 |
| *** HeOS has quit IRC | 16:58 | |
| cinerama | AJaeger_: oh awesome. did you see my question in my mail about the percentage completion? if it's per-project i can streamline some of the stuff i've done | 16:59 |
| fungi | ?status notify zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing | 16:59 |
| fungi | jeblair: ^ ? | 16:59 |
| *** deepakcs has joined #openstack-infra | 16:59 | |
| *** ildikov has joined #openstack-infra | 16:59 | |
| fungi | are you planning to requeue the prior check pipeline as well? | 17:00 |
| fungi | if not, i can adjust wording accordingly | 17:00 |
| *** patrickeast has joined #openstack-infra | 17:00 | |
| deepakcs | mtreinish, ping | 17:00 |
| jeblair | fungi: enqueing in progress | 17:00 |
| fungi | cool | 17:00 |
| AJaeger_ | cinerama: haven't checked mail again, will do now. | 17:00 |
| fungi | the jenkins prepare to shutdown has seriously sped up jjb updates | 17:00 |
| jeblair | fungi: lgtm (unsure about those times, but i don't think we can be) | 17:00 |
| *** ivar-laz_ has joined #openstack-infra | 17:01 | |
| fungi | jeblair: i'm going on the age-in-queue of the oldest change in the check pipeline prior to the restart | 17:01 |
| *** ivar-laz_ has quit IRC | 17:01 | |
| fungi | which was about 2 hours ago now | 17:01 |
| *** ivar-laz_ has joined #openstack-infra | 17:01 | |
| jeblair | fungi: shouldn't it be newest in check? | 17:01 |
| fungi | er, newest is what i meant | 17:02 |
| jeblair | fungi: sounds good then | 17:02 |
| *** ivar-laz_ has quit IRC | 17:02 | |
| fungi | yes. sorry for the brainbreak | 17:02 |
| *** dustins_ has joined #openstack-infra | 17:02 | |
| fungi | bottom one in the pipeline on the status page | 17:02 |
| *** ivar-laz_ has joined #openstack-infra | 17:02 | |
| fungi | #status notify zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing | 17:03 |
| openstackstatus | fungi: unknown command | 17:03 |
| *** maishsk has quit IRC | 17:03 | |
| fungi | #status notice zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing | 17:03 |
| openstackstatus | fungi: sending notice | 17:03 |
| fungi | notisomething | 17:03 |
| *** dustins_ is now known as dustins | 17:03 | |
| reed | are bugs for CI logged in Launchpad kept in sync with Storyboard? | 17:03 |
| AJaeger_ | cinerama: I missed your question, thanks for asking back! I answered. Should I explain more? | 17:03 |
| jeblair | reed: the projects that moved to storyboard have moved; there is no syncing | 17:04 |
| -openstackstatus- NOTICE: zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing | 17:04 | |
| *** dustins has quit IRC | 17:04 | |
| jeblair | reed: but "openstack-gate" is still in launchad, and holds bugs related to CI | 17:04 |
| reed | jeblair, to be clear this for example: https://bugs.launchpad.net/openstack-ci/+bug/1265591 may already have been fixed, right? I should check on storyboard | 17:04 |
| openstack | Launchpad bug 1265591 in OpenStack Core Infrastructure "jeepyb git url hard-coded to openstack" [Medium,Triaged] - Assigned to Sergey Lukjanov (slukjanov) | 17:04 |
| *** ivar-lazzaro has quit IRC | 17:04 | |
| reed | i'm looking for low-hanging-fruit bugs | 17:05 |
| *** btully has quit IRC | 17:05 | |
| jeblair | reed: yes, that should be in the jeepyb project on storyboard | 17:05 |
| openstackstatus | fungi: finished sending notice | 17:05 |
| pleia2 | anteaya: sorry, had some other things come up this morning, available to do gerrit things now :) | 17:05 |
| fungi | reed: it's possible. the bug numbers in storyboard should be a 1:1 correspondence to lp too | 17:05 |
| *** e0ne has quit IRC | 17:05 | |
| reed | gotcha | 17:06 |
| reed | thanks | 17:06 |
| anteaya | pleia2: fungi did it, thanks though | 17:06 |
| jeblair | reed: also, how did you find it? https://bugs.launchpad.net/openstack-ci/ is fairly dead-endish | 17:06 |
| *** hodos has joined #openstack-infra | 17:06 | |
| patrickeast | i might be way late to the party, but is anyone else seeing or know about the devstack error with swift failing? is that related to the zuul restart? | 17:06 |
| anteaya | pleia2: and I understand was just covering my options | 17:06 |
| anteaya | patrickeast: what error? | 17:06 |
| pleia2 | anteaya: ok, next time! :) | 17:06 |
| patrickeast | http://ec2-54-67-51-189.us-west-1.compute.amazonaws.com/83/178783/4/check/PureISCSIDriver-tempest-dsvm-volume-multipath/5a9ef90/logs/devstacklog.txt.gz#_2015-05-05_16_46_26_411 | 17:06 |
| pleia2 | cinerama: hey | 17:06 |
| patrickeast | anteaya: ^ | 17:06 |
| anteaya | patrickeast: if it occured in the last 8 hours, then yes probably it is | 17:06 |
| hodos | is ok for thirdparty ci not run all the 304 tests ? | 17:06 |
| *** gokrokve has joined #openstack-infra | 17:07 | |
| cinerama | AJaeger_: i did see that. the issue i'm having is that the transifex client itself also handles percentages. if you have a look at propose_translation_update.sh | 17:07 |
| hodos | can you manually disable some of them ? | 17:07 |
| anteaya | hodos: ask the project you are testing | 17:07 |
| *** gokrokve has quit IRC | 17:07 | |
| hodos | thanks | 17:07 |
| cinerama | AJaeger_: there's a minimum_perc flag | 17:07 |
| patrickeast | anteaya: ok, cool, i’ll wait and see if it goes away then | 17:07 |
| anteaya | patrickeast: so this is your ci system not ours | 17:07 |
| fungi | anteaya: patrickeast: i can't imagine a third-party ci test error would be related to issues on our jenkins masters | 17:07 |
| patrickeast | anteaya: it is in mine, but http://bit.ly/1IdXRc7 shows it in jenkins too | 17:08 |
| AJaeger_ | cinerama: brb | 17:08 |
| anteaya | fungi: yeah, I didn't think so either | 17:08 |
| *** harlowja has quit IRC | 17:08 | |
| fungi | patrickeast: those in ours are potentially related. in yours they should not me | 17:08 |
| fungi | er, should not be | 17:08 |
| patrickeast | fungi: makes sense | 17:09 |
| *** harlowja has joined #openstack-infra | 17:09 | |
| anteaya | patrickeast: you may have uncovered a new bug that is unrelated to our current fire | 17:09 |
| fungi | patrickeast: er, actually no even in ours they should not be. it looks like a bug encountered while testing, not a bug in uploading our logs to our swift container | 17:09 |
| DuncanT | Could anybody core for global requirements looks at https://review.openstack.org/#/c/177372/ please? It has one +2 and is holding up cinder work.... been there for a while with no objections, lots of +1s | 17:09 |
| AJaeger_ | cinerama: we only sat the minimum_perc flag in file http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh | 17:10 |
| AJaeger_ | cinerama: see line 108 (PERC variable) for openstack-manuals | 17:10 |
| anteaya | DuncanT: here is your list of people: https://review.openstack.org/#/admin/groups/131,members | 17:10 |
| jeblair | fungi: my jjb run on 03 is deleting lots of jobs, including ones in use | 17:10 |
| fungi | hrm | 17:10 |
| fungi | that sounds not right | 17:10 |
| DuncanT | anteaya: I pinged a bunch of them directly, only Monty got back to me (and +2d) :-( | 17:10 |
| anteaya | DuncanT: k, kind of fighting a fire in here | 17:11 |
| cinerama | AJaeger_: we also do in propose_* scripts | 17:11 |
| anteaya | DuncanT: ttx might still be around or markmcclain | 17:11 |
| *** btully has joined #openstack-infra | 17:11 | |
| jeblair | fungi: i'm removing the cache and restarting | 17:11 |
| cinerama | AJaeger_: are they not used? | 17:11 |
| *** gokrokve has joined #openstack-infra | 17:12 | |
| AJaeger_ | cinerama: ah, now remembered details ;) | 17:12 |
| jeblair | jjb cache writing is unsafe for program interrupts :( | 17:12 |
| AJaeger_ | So, for python projects we use tx pull --minimum-perc like in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_translation_update.sh#n37 | 17:12 |
| *** gokrokve has quit IRC | 17:12 | |
| DuncanT | anteaya: Thanks | 17:12 |
| cinerama | AJaeger_: so basically, if the percentages are per project rather than per file, i can rip out a bunch of potentially brittle code | 17:12 |
| AJaeger_ | for openstack-manuals, we add to the config file | 17:12 |
| fungi | DuncanT: yeah, there's really not an irc channel for the openstack/requirements reviewers in aggregate. a lot of them may be hanging out in this channel, but the infra team generally refrains from weighing in on requirements changes unless they're infra-related | 17:12 |
| AJaeger_ | cinerama: openstack-manuals is the only repo that uses per file, everything else is per project | 17:13 |
| fungi | jeblair: yeah, i'm rerunning mine that just finished too | 17:13 |
| cinerama | AJaeger_: we have to do it a little differently for zanata because we don't have an equivalent of "tx set" and when we pull there is no built-in percentage option | 17:13 |
| cinerama | AJaeger_: so that has resulted in a bunch of work building a workaround for that | 17:14 |
| cinerama | AJaeger_: ah ok so for the manuals i will still need to calculate the percentage for each file? | 17:14 |
| *** sdake_ has quit IRC | 17:14 | |
| *** freerunner has joined #openstack-infra | 17:14 | |
| *** yfried is now known as yfried|afk | 17:14 | |
| *** gyee has quit IRC | 17:14 | |
| AJaeger_ | cinerama: so, openstack-manuals has two files - the glossary and common - that use 8 %, the rest 75 | 17:14 |
| cinerama | AJaeger_: the other hangup is that i don't seem to be able to pull an individual file from the project so i had to build that | 17:14 |
| cinerama | AJaeger_: but if it | 17:15 |
| pleia2 | ooh, I didn't realize there was a 8% one | 17:15 |
| *** SumitNaiksatam has joined #openstack-infra | 17:15 | |
| cinerama | s just for the two manuals then maybe i i can do it an easier way | 17:15 |
| AJaeger_ | cinerama: let us talk in 30 mins or so, ok? | 17:15 |
| cinerama | AJaeger_: np | 17:15 |
| anteaya | DuncanT: welcome | 17:15 |
| cinerama | pleia2: shenanigans! | 17:15 |
| AJaeger_ | pleia2: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n117 | 17:15 |
| pleia2 | cinerama, AJaeger_ - also just sent out the email for meeting up at 0600 on May 6th | 17:15 |
| *** achanda has joined #openstack-infra | 17:16 | |
| cinerama | pleia2: just trying to unblock so i can wrap up what i have | 17:16 |
| *** btully has quit IRC | 17:16 | |
| fungi | DuncanT: probably the most appropriate irc channel to use for requesting requirements reviews is #openstack-dev | 17:17 |
| *** dboik_ has quit IRC | 17:17 | |
| pleia2 | cinerama: oh yes, please do! | 17:17 |
| DuncanT | fungi: I'll try there in future, thanks. anteaya and PMs has worked for now | 17:17 |
| *** dboik has joined #openstack-infra | 17:17 | |
| *** shashankhegde has quit IRC | 17:18 | |
| jeblair | fungi: running jjb on 03 again to make sure it's stable | 17:18 |
| cinerama | pleia2: so i really want to get us out of the business of having to pull individual files because it's a bit hairy | 17:18 |
| mriedem | lifeless: according to http://docs.openstack.org/developer/pbr/semver.html#semantic-versioning-specification-semver item 9, shouldn't a pre-release version like 2015.1.1.a1 work on a project? | 17:18 |
| AJaeger_ | cinerama: ok, that was a quick 30mins - I'm back... | 17:18 |
| mriedem | lifeless: like nova on stable/kilo | 17:18 |
| gordc | anteaya: will do. | 17:18 |
| AJaeger_ | cinerama: percentages calculation is done here for example: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n350 | 17:19 |
| anteaya | gordc: thanks | 17:19 |
| cinerama | pleia2: actually if you could cast a second set of eyes on "zanata-cli pull" just to double check i'm not missing somethign obvious | 17:19 |
| AJaeger_ | So, really your call what is easier to do - filtering like that function after pulling or as part of zanata-cli | 17:20 |
| *** kambiz has joined #openstack-infra | 17:20 | |
| pleia2 | cinerama: will do | 17:20 |
| cinerama | AJaeger_: yes, i've seen that code...i'm talking about this: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_translation_update.sh#n37 | 17:21 |
| *** gyee has joined #openstack-infra | 17:21 | |
| *** liam_ has joined #openstack-infra | 17:21 | |
| jeblair | fungi: it seems to be flapping and now deleting jobs again | 17:21 |
| *** liam_ is now known as Guest95660 | 17:21 | |
| AJaeger_ | cinerama: so, what's the issue with that parameter? | 17:21 |
| cinerama | pleia2: so i don't think we can pull just the one file...it looks like we can start from a file and go in alphabetical order, or do includes | 17:21 |
| cinerama | AJaeger_: the problem is that we don't have it in zanata-cli | 17:22 |
| cinerama | AJaeger_: so i need to know exactly what it's doing so we can try to work around that | 17:22 |
| *** baoli_ has quit IRC | 17:22 | |
| AJaeger_ | cinerama: then let's not do it. I do something like in the other function to replicate that. | 17:22 |
| *** sergsh has quit IRC | 17:22 | |
| *** sergsh_ has quit IRC | 17:22 | |
| AJaeger_ | cinerama: If we just have "pull down all files", I can filter in bash like in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n350 | 17:23 |
| fungi | jeblair: yeah, i'm seeing similar behavior too | 17:23 |
| AJaeger_ | cinerama: what that help you? | 17:24 |
| cinerama | AJaeger_: oh ok, i see - so you're saying we should switch to grabbing everything and using msgfmt? | 17:24 |
| cinerama | AJaeger_: rather than trying to do stuff with zanata | 17:24 |
| AJaeger_ | cinerama: Yes, that's my suggestion. | 17:24 |
| AJaeger_ | cinerama: Just offering suggestions since I hear that it's difficult with zanata... | 17:25 |
| *** Krinkle|detached is now known as Krinkle | 17:25 | |
| cinerama | AJaeger_: i think that's good actually - i was kind of getting focused on the zanata client's capabilities because i wasn't seeing the whole picture | 17:25 |
| *** Somay has joined #openstack-infra | 17:25 | |
| pleia2 | it may even end up being faster than individual analysis through percentage statuses via REST pre-download | 17:26 |
| cinerama | AJaeger_, pleia2: yeah grabbing the stats from zanata is slow | 17:26 |
| * pleia2 nods | 17:26 | |
| AJaeger_ | cinerama: we have the client and we have our scripts, together they form our framework and we can decide on where to do what we need | 17:26 |
| cinerama | AJaeger_, pleia2: simplifying the pull is a big win because it avoids us having to do crazy stuff to reassemble the po files | 17:27 |
| AJaeger_ | cinerama: if grabbing the stats is slow, let's do a local msgfmt. It means more files to download... | 17:27 |
| *** tnovacik has quit IRC | 17:27 | |
| cinerama | AJaeger_, pleia2: so from my stuff we have the zanata.xml generation which is great | 17:27 |
| AJaeger_ | cinerama: then let's go for that - and if we later figure out downloading is issue, we can change the architecture again | 17:28 |
| cinerama | AJaeger_, pleia2: so in between now and the meeting i will simplify what i have | 17:28 |
| *** freerunner has quit IRC | 17:28 | |
| AJaeger_ | cinerama: go for it ;) | 17:29 |
| AJaeger_ | cinerama: thanks a lot! | 17:29 |
| pleia2 | cinerama: wonderful, thank you | 17:29 |
| jeblair | fungi: i'm resetting 03 and prepping to debug | 17:29 |
| fungi | ok | 17:29 |
| cinerama | AJaeger_: no, thank you. your broader perspective on this has been hugely helpful. and i like deleting code that's going to be hard for others to maintain :) | 17:29 |
| *** yfried|afk is now known as yfried | 17:30 | |
| *** Somay has quit IRC | 17:30 | |
| *** baoli has joined #openstack-infra | 17:30 | |
| AJaeger_ | cinerama: ;) | 17:31 |
| *** tiswanso_ has quit IRC | 17:31 | |
| *** tiswanso has joined #openstack-infra | 17:32 | |
| *** pradk has joined #openstack-infra | 17:32 | |
| clarkb | zaro: how do I get logs when running `mvn test`? | 17:32 |
| *** dims_ has quit IRC | 17:33 | |
| *** mwagner_lap has joined #openstack-infra | 17:33 | |
| zaro | clarkb: -X flag? | 17:34 |
| *** dims has joined #openstack-infra | 17:34 | |
| zaro | clarkb: reviewing your change now. i ran build and didn't see any test failures. | 17:34 |
| clarkb | zaro: that appears to give me mvn's logs, I want logs generated by the tests | 17:35 |
| *** Sukhdev has joined #openstack-infra | 17:35 | |
| *** Guest95660 has quit IRC | 17:35 | |
| *** freerunner has joined #openstack-infra | 17:35 | |
| clarkb | zaro: the start stop thread fails for me, I think it may have to do with how my change handles interrupted exceptions, I could try jdb instead but logs seem so much simpler here | 17:37 |
| *** maishsk has joined #openstack-infra | 17:39 | |
| *** dboik_ has joined #openstack-infra | 17:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/gearman-plugin: Stop sending status updates https://review.openstack.org/180249 | 17:39 |
| zaro | clarkb: here are all the options for mvn test https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html | 17:40 |
| jeblair | clarkb, zaro: ^ while we're digging around in there, i think that would be a good change to land to help reduce zuul busy-work | 17:40 |
| *** sdake has joined #openstack-infra | 17:41 | |
| *** dboik has quit IRC | 17:42 | |
| zaro | clarkb: that failure existed before your change: http://logs.openstack.org/28/145828/1/check/gate-gearman-plugin-build/3b586a1/console.html.gz | 17:42 |
| *** melwitt_ has joined #openstack-infra | 17:42 | |
| *** melwitt_ has quit IRC | 17:42 | |
| clarkb | jeblair: I think you need to still loop until done right? | 17:43 |
| zaro | clarkb: i also get that failure on my build. | 17:43 |
| clarkb | zaro: ah ok, that makes me feel better | 17:43 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Make command line parsers available as modules. https://review.openstack.org/180252 | 17:45 |
| jeblair | clarkb: future.get should take care of that; the one with no args should mean "wait until complete" | 17:45 |
| jeblair | clarkb: (the call in the loop had a timeout) | 17:45 |
| *** dustins has joined #openstack-infra | 17:45 | |
| clarkb | jeblair: gotcha | 17:45 |
| clarkb | jeblair: lgtm | 17:45 |
| jeblair | clarkb: i think, we should double check that :) | 17:45 |
| *** abregman has quit IRC | 17:46 | |
| zaneb | sdague: any chance of a re-review on https://review.openstack.org/#/c/174233/ ? Your comment was addressed and mordred gave +2 | 17:47 |
| zaneb | would be much appreciated :) | 17:47 |
| fungi | jeblair: i cleared the cache and rebuilt the jenkins configs on 05 and 06 | 17:47 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add floating IP pool resource. https://review.openstack.org/180255 | 17:48 |
| *** AndroUser has joined #openstack-infra | 17:49 | |
| *** AndroUser has quit IRC | 17:49 | |
| jeblair | clarkb: yeah, reread docs and i think what's there is sufficient | 17:49 |
| jeblair | fungi: does it flap if you re-run? | 17:49 |
| *** shashankhegde has joined #openstack-infra | 17:49 | |
| jeblair | fungi: or are you saying you are in a state ready for a bugfix? :) | 17:49 |
| *** e0ne has joined #openstack-infra | 17:50 | |
| *** mrmartin has joined #openstack-infra | 17:50 | |
| *** e0ne is now known as e0ne_ | 17:50 | |
| zaro | fungi: are we increasing timeout for jjb or want to make it upload jobs faster? | 17:50 |
| fungi | jeblair: yeah, it flaps if i keep rerunning with --delete-old | 17:50 |
| jeblair | fungi: i believe i have confirmed a bug masked by an over-broad exception | 17:50 |
| clarkb | possible new change to jjb breaking us? | 17:50 |
| jeblair | clarkb: yes, and i think i know the one... more soon | 17:50 |
| *** e0ne_ is now known as e0ne | 17:51 | |
| fungi | also 05 and 06 are available for further testing or to get thrown back to the gate wolves now | 17:51 |
| zaro | hmm, big change that just landed was parallel uploading of jobs. | 17:51 |
| *** chriscowley has joined #openstack-infra | 17:51 | |
| *** packet has joined #openstack-infra | 17:51 | |
| waynr | ruh roh | 17:51 |
| *** maishsk is now known as maishsk_afk | 17:52 | |
| *** crc32 has joined #openstack-infra | 17:52 | |
| waynr | i was just preparing to test that out, hopefuly it's not too searious breakage | 17:52 |
| *** ZZelle has quit IRC | 17:53 | |
| kambiz | jeblair: bswartz radez and I are working on a hardware refresh and will need assistance with DNS for trystack.org at some point in the not too distant future. would you recommend emails to hostmaster@openstack.org for those sorts of things ? | 17:53 |
| *** ZZelle has joined #openstack-infra | 17:53 | |
| chriscowley | Hi all, first time in the infra meeting. Will be lurking to see what's up | 17:53 |
| fungi | waynr: basic symptom is that if you run `jenkins-jobs update --delete-old /etc/jenkins_jobs/config` twice, the second run seems to delete all the jobs *at least while the jenkins service is quiesced) | 17:54 |
| fungi | chriscowley: welcome--hopefully we won't disappoint | 17:54 |
| waynr | okay that doesn't seem like it will affect me yet...i haven't been using --delete-old | 17:55 |
| lifeless | mriedem: yes | 17:55 |
| *** yfried is now known as yfried|afk | 17:55 | |
| lifeless | mriedem: have you had an issue w/ it? | 17:56 |
| mriedem | lifeless: hmm, didn't seem to be working for me locally | 17:56 |
| mriedem | lifeless: yeah, couldn't use 2015.1.0.1 either | 17:56 |
| lifeless | 2015.1.0.1 is definitely not supported | 17:56 |
| mriedem | yeah i expect not after reading the docs | 17:56 |
| mriedem | would have to be like .a1 or something | 17:56 |
| jeblair | kambiz: probably better to ask folks in here, or send something to openstack-infra@lists.o.o | 17:56 |
| lifeless | mriedem: what error do you get? | 17:57 |
| lifeless | mriedem: or what symptoms are you seeing? | 17:57 |
| *** tiswanso has quit IRC | 17:57 | |
| mriedem | lifeless: http://paste.openstack.org/show/215036/ | 17:58 |
| *** deepakcs has quit IRC | 17:59 | |
| lifeless | mriedem: oh, don't do that. | 17:59 |
| lifeless | mriedem: set the version in setup.cfg to 2015.1.1, if thats the release you are working *towards* | 17:59 |
| *** tkelsey has quit IRC | 18:00 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering https://review.openstack.org/179988 | 18:00 |
| mriedem | lifeless: so do dev versions not work anymore? | 18:00 |
| jeblair | zaro, waynr: is david caro here? | 18:00 |
| lifeless | mriedem: they totally do | 18:00 |
| clarkb | jeblair: zaro ^ ok I am going to work on spinning up a jenkins and trigger jobs with that now | 18:00 |
| *** ssam2 has quit IRC | 18:00 | |
| *** bknudson has joined #openstack-infra | 18:00 | |
| *** baoli has quit IRC | 18:00 | |
| clarkb | giving ./run-fast a go | 18:00 |
| jeblair | clarkb: yeah that should work | 18:01 |
| clarkb | hrm that exploded | 18:01 |
| lifeless | mriedem: there may be some disconnect here; I presume you want to make an alpha of nova? | 18:01 |
| waynr | jeblair: i'm not sure | 18:01 |
| lifeless | mriedem: if so, tag it. | 18:01 |
| clarkb | NPE and doesn't seem to have touched the gearman plugin so unlikely related to my changes | 18:01 |
| fungi | clarkb: any reason not to test on jenkins-dev? | 18:01 |
| lifeless | mriedem: 2015.1.1.dev1 is < 2015.1.1.a1 - pbr won't bump up to a1 until you tag it | 18:01 |
| clarkb | fungi: probably not, I can give that a go | 18:01 |
| mriedem | lifeless: ok | 18:02 |
| clarkb | zaro: any idea why run-fast fails? appears to be an error in the codehaus launcher | 18:02 |
| *** baoli_ has joined #openstack-infra | 18:02 | |
| *** yfried|afk is now known as yfried | 18:02 | |
| mriedem | i don't really want an alpha, it's kind of a weird situation | 18:02 |
| lifeless | mriedem: tell me more | 18:02 |
| mriedem | lifeless: we basically release our 2015.1 version of openstack a few weeks after community | 18:02 |
| lifeless | O\___ <- my couch | 18:02 |
| fungi | heh | 18:02 |
| mriedem | so we branch off kilo at 2015.1 tag | 18:02 |
| mriedem | then we backport fixes as needed until we release our 2015.1 | 18:03 |
| openstackgerrit | Alexandre Viau proposed openstack-infra/project-config: Use surveil gerrit acl for python-surveilclient https://review.openstack.org/180257 | 18:03 |
| *** dannywilson has quit IRC | 18:03 | |
| *** tiswanso_ has joined #openstack-infra | 18:03 | |
| mriedem | but it's really like 2015.1.0.1 since this is before community releases 2015.1.1 | 18:03 |
| mriedem | but 2015.1.0.1 isn't a thing in pbr land | 18:03 |
| jeblair | pelix: ^ see conversation about jjb and parallelization | 18:03 |
| zaro | jeblair: i have not seen him in here, i think yolanda might be able to get his attention somehow | 18:03 |
| mriedem | lifeless: anyway, our packages (rpms) are version 2015.1 but the source is like 2015.1 + some patches | 18:03 |
| mriedem | lifeless: and i'm trying to figure out how to versoin that | 18:04 |
| lifeless | ok, so we haven't exposed it in the CLI yet | 18:04 |
| lifeless | but there is a function for this | 18:04 |
| *** freerunner has quit IRC | 18:04 | |
| zaro | clarkb: no idea. is this run-fast from gearman-plugin? | 18:04 |
| yolanda | i can ping Darragh on hipchat, what happens? | 18:04 |
| clarkb | zaro: yup, I have also replicated using the debugging a plugin steps at https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial | 18:04 |
| jeblair | yolanda: no i pinged darragh here, i'm asking about david caro | 18:05 |
| *** ddieterly has quit IRC | 18:05 | |
| yolanda | ah, i know David Caro, i can talk with him, and he is dcaro on freenode, he will answer if pinged in private | 18:05 |
| yolanda | what's up with him? | 18:06 |
| zaro | clarkb: don't know, but 'mvn clean hpi:run' works for me | 18:06 |
| jeblair | i'm about to revert his patch | 18:06 |
| *** ildikov has quit IRC | 18:06 | |
| yolanda | jeblair, why? is there some issue? | 18:06 |
| jeblair | yolanda: yes, it deletes all of the jobs | 18:06 |
| lifeless | mriedem: have a look at rpm_version on semantic version (and or the semver spec in oslo-specs) | 18:06 |
| clarkb | zaro: huh let me try that | 18:07 |
| lifeless | mriedem: rpm doesn't have a sort-before operator | 18:07 |
| lifeless | mriedem: and so its fugly :( | 18:07 |
| clarkb | zaro: that failed for me | 18:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/jenkins-job-builder: Revert "Added parallelization options" https://review.openstack.org/180258 | 18:07 |
| clarkb | its fine, I have a container running jenkins somewhere I can just resurrect that and put my plugin in it, then migrated to jenkins-dev if that looks good | 18:07 |
| jeblair | i'm going to update the commit msg on that ^ gimme a min | 18:08 |
| clarkb | kk | 18:08 |
| *** chriscowley_ has joined #openstack-infra | 18:08 | |
| yolanda | jeblair, deletes jobs from jenkins, as if they were old? | 18:08 |
| zaro | clarkb: what's not working about it? it seems to be ok for me. maybe try 'mvn install' first? | 18:08 |
| *** gokrokve has joined #openstack-infra | 18:09 | |
| *** patrickeast_ has joined #openstack-infra | 18:09 | |
| *** btully has joined #openstack-infra | 18:09 | |
| clarkb | zaro: http://paste.openstack.org/show/215037/ | 18:09 |
| *** hdd has joined #openstack-infra | 18:10 | |
| zaro | jeblair: ohh that sux! i tested --delete-old and that seemed to worked for me. maybe i did it with a clean cache. | 18:10 |
| yolanda | zaro, i tested that as well, under your suite of tests, and it passed tests for me as well | 18:11 |
| yolanda | we need to be testing something incompletely | 18:11 |
| *** patrickeast has quit IRC | 18:11 | |
| *** patrickeast_ is now known as patrickeast | 18:11 | |
| *** chriscowley has quit IRC | 18:11 | |
| *** chriscowley_ is now known as chriscowley | 18:11 | |
| *** gokrokve has quit IRC | 18:12 | |
| lifeless | vishy: tsk! 3d1de2a breaks pbr.. <A0>* Remo | 18:12 |
| lifeless | vishy: *how* did you get that into git :) | 18:13 |
| clarkb | btw https://github.com/cboylan/jjbhowto/tree/master/dockerstuff is the docker stuff for jenkins, not perfect but handy | 18:13 |
| clarkb | zaro: jeblair ^ I can copy/move that into the plugins if it helps testing too | 18:13 |
| zaro | clarkb: looks like maven went wacko there. i would try to kill your local maven repo at ~/.m2 and rerun the command | 18:14 |
| *** btully has quit IRC | 18:14 | |
| *** packet has quit IRC | 18:14 | |
| *** ddieterly has joined #openstack-infra | 18:15 | |
| *** otter768 has joined #openstack-infra | 18:15 | |
| *** dizquierdo has joined #openstack-infra | 18:16 | |
| jogo | so mellanox CI is spamming nova | 18:16 |
| jogo | and they are a non-voting CI | 18:17 |
| jogo | last 30 comments http://paste.openstack.org/show/215039 | 18:17 |
| *** sparkycollier has joined #openstack-infra | 18:18 | |
| zaro | clarkb: you mean gearman plugin ? seems too specific for inclusion. | 18:18 |
| *** ildikov has joined #openstack-infra | 18:18 | |
| anteaya | lennyb are you here | 18:18 |
| clarkb | zaro: any plugin, its pretty handy to say docker run and get a jenkins running with your plugin to test again | 18:19 |
| clarkb | zaro: considering that hpi:run seems to not work | 18:19 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Reorganize the configuration loading/munging code. https://review.openstack.org/180262 | 18:19 |
| jogo | anteaya: looks like he isn't around | 18:19 |
| jogo | can we email him and turn it off? | 18:19 |
| zaneb | markmcclain: thanks for the review on https://review.openstack.org/#/c/174233/ ! Any reason we're holding back on the +A? | 18:19 |
| jogo | because all the merge conflict messages are frustrating and misleading | 18:19 |
| *** ivar-laz_ has quit IRC | 18:20 | |
| anteaya | jogo: well strictly speaking they are just commenting | 18:20 |
| jogo | anteaya: and commenting spam | 18:20 |
| anteaya | jogo: right but we never told them they can't do that | 18:20 |
| *** _nadya_ has quit IRC | 18:20 | |
| mriedem | anteaya: people are thinking they are being told their patch is in merge conflict and has to be rebased | 18:20 |
| *** ivar-lazzaro has joined #openstack-infra | 18:20 | |
| jogo | commenting spam should never be allowed | 18:20 |
| anteaya | jogo: I can disable but know they they didn't know they can't do this | 18:20 |
| anteaya | mriedem: no argument here | 18:20 |
| zaro | clarkb: with a specific version of java and jenkins on ubuntu? | 18:20 |
| *** otter768 has quit IRC | 18:20 | |
| mriedem | i don't think this is intentional | 18:20 |
| anteaya | jogo: sure but define what spam is for nova | 18:20 |
| anteaya | jogo: so I can tell them | 18:20 |
| jogo | anteaya: thanks, should I contact the maintainer or what is the standard procedure | 18:21 |
| zaro | clarkb: i'm guessing not everyone's gonna want that | 18:21 |
| jogo | anteaya: what mriedem said, reporting merge failed on everything | 18:21 |
| mriedem | just, it's busted so let's turn it off if they aren't going to have 24x7 converage | 18:21 |
| anteaya | I will have them disabled and post to third-party-announce and have them contact you | 18:21 |
| mriedem | cool, thanks anteaya | 18:21 |
| anteaya | I'm not arguing with you | 18:21 |
| anteaya | I'm agreeing | 18:21 |
| jogo | anteaya: great, thank you!! | 18:21 |
| mriedem | now jogo can stop hurting all of us with his blue language | 18:21 |
| jogo | mriedem: lol | 18:21 |
| anteaya | but I'm telling you from their perspective they might not know what they are doing isn't helpful | 18:21 |
| anteaya | as lennyb isn't a dev | 18:22 |
| jogo | anteaya: understood, I will be happy to walk them through the issue if needed | 18:22 |
| clarkb | zaro: ya its a specific version of java and jenkins, you don't have to use it and can edit it to use different things | 18:22 |
| *** ivar-lazzaro has quit IRC | 18:22 | |
| anteaya | jogo: great thank you I will direct lenny to you | 18:23 |
| *** ivar-lazzaro has joined #openstack-infra | 18:23 | |
| anteaya | jogo: he will show up, he is good for persisting, he just has no idea what he is doing but he means well | 18:23 |
| jogo | anteaya: thanks! | 18:23 |
| *** gokrokve has joined #openstack-infra | 18:24 | |
| anteaya | thank you | 18:25 |
| anteaya | can we get gerrit account id 9732 disabled please: http://paste.openstack.org/show/215041/ | 18:25 |
| anteaya | pleia2: if you are about? | 18:25 |
| pleia2 | anteaya: o/ | 18:25 |
| anteaya | can you disable 9732 please? | 18:25 |
| zaro | clarkb: i'm fine with it. maybe just with a little more documentation. | 18:26 |
| pleia2 | anteaya: sure, let's see... | 18:26 |
| *** tosky has quit IRC | 18:27 | |
| *** ildikov has quit IRC | 18:27 | |
| *** hdd has quit IRC | 18:28 | |
| yolanda | jeblair, you were still running that change on single mode, right? without --workers=0 | 18:28 |
| clarkb | fwiw I am not seeing GET_JOB requests on the geard side | 18:29 |
| pleia2 | clarkb: so when we disable an account we use --active, I assume disabling is --inactive? (that's what I get from reading the docs, just want to make sure I have the right one) | 18:29 |
| *** gokrokve has quit IRC | 18:29 | |
| clarkb | I see the RESET_ABILITIES and CAN_DOs though | 18:29 |
| *** Ala has joined #openstack-infra | 18:29 | |
| pleia2 | clarkb: re: gerrit 3rd party accounts | 18:29 |
| clarkb | pleia2: disabling is --inactive, enabling is --active | 18:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/jenkins-job-builder: Revert "Added parallelization options" https://review.openstack.org/180258 | 18:30 |
| pleia2 | got it right \o/ | 18:30 |
| pleia2 | ok thanks | 18:30 |
| jeblair | yolanda, pelix, zaro: ^ see updated commit message | 18:30 |
| *** fawadkhaliq has quit IRC | 18:30 | |
| jeblair | clarkb, fungi: ^ | 18:30 |
| pleia2 | anteaya: done | 18:30 |
| jeblair | yolanda: yes, i have no interest in running multi-threaded. i think it's a great way to kill our already over-burdened jenkinns masters. | 18:31 |
| anteaya | pleia2: thank you | 18:31 |
| fungi | jeblair: also worth noting, zuul still seems fairly bogged down throughput-wise even though we don't have jjb currently running anywhere now | 18:31 |
| *** Somay has joined #openstack-infra | 18:31 | |
| *** AL34N1X has joined #openstack-infra | 18:31 | |
| *** shardy_ has joined #openstack-infra | 18:33 | |
| *** gokrokve has joined #openstack-infra | 18:34 | |
| *** shardy has quit IRC | 18:34 | |
| *** ivar-laz_ has joined #openstack-infra | 18:34 | |
| lifeless | vishy: https://bugs.launchpad.net/pbr/+bug/1451976 <- I'm interested in how that got into nova | 18:34 |
| openstack | Launchpad bug 1451976 in PBR "non-utf8 commits break getting version info" [Undecided,New] | 18:34 |
| *** sparkycollier has quit IRC | 18:34 | |
| fungi | looks like devstack/tempest may just be hosed | 18:34 |
| *** ivar-lazzaro has quit IRC | 18:35 | |
| *** ivar-laz_ has quit IRC | 18:35 | |
| *** prad has quit IRC | 18:35 | |
| fungi | yeah, image upload failures in devstack | 18:35 |
| zaro | clarkb: yeah, i don't see a GRAB_JOB after CAN_DO from the worker. | 18:35 |
| *** hdd has joined #openstack-infra | 18:35 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:35 | |
| jeblair | fungi: asymmetric gate? | 18:36 |
| *** prad has joined #openstack-infra | 18:36 | |
| fungi | jeblair: maybe, though there's an xattr-related traceback http://logs.openstack.org/56/177356/9/gate/gate-tempest-dsvm-full/d186960//logs/devstacklog.txt.gz#_2015-05-05_17_53_13_450 | 18:36 |
| fungi | might be a new release of some dep | 18:37 |
| jeblair | ah | 18:37 |
| *** cody-somerville has joined #openstack-infra | 18:37 | |
| fungi | digging now | 18:37 |
| jeblair | fungi: my best guess for zuul slowness is geard slowness since job submission is synchronous | 18:37 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Issue #1451976: handle commits with non-utf8 text https://review.openstack.org/180265 | 18:38 |
| lifeless | mriedem: ok, now I've shaved that yak... | 18:38 |
| *** shardy_ has quit IRC | 18:38 | |
| *** packet has joined #openstack-infra | 18:39 | |
| mriedem | lifeless: oh yeah? | 18:39 |
| *** shardy has joined #openstack-infra | 18:39 | |
| lifeless | mriedem: yeah, I went to poke at nova stable/kilo and found that pbr broke on it | 18:39 |
| lifeless | mriedem: for totally different reasons to your need | 18:39 |
| jeblair | fungi: are there any jjb runs still in progress? | 18:40 |
| lifeless | mriedem: anyhow, did you find the spec? we defined a mapping to rpm versions - its nonintuitive because we have to map *up* a version to deal with pre-releases | 18:40 |
| lifeless | >>> pbr.version.VersionInfo('nova').semantic_version() | 18:40 |
| lifeless | pbr.version.SemanticVersion(2015.1.1.0a2.dev4) | 18:40 |
| lifeless | >>> pbr.version.VersionInfo('nova').semantic_version().rpm_string() | 18:41 |
| lifeless | '2015.1.0.a2.dev4' | 18:41 |
| fungi | jeblair: as i said, none for the past hour or so... so not sure that theory panned out after all | 18:41 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return Bunch objects instead of plain dicts https://review.openstack.org/180269 | 18:41 |
| mriedem | lifeless: yeah i found the spec | 18:41 |
| mriedem | lifeless: so rpm-version is busted with pbr 0.11 and nova stable/kilo? | 18:42 |
| mriedem | i see the 2015.1.1 is missing | 18:42 |
| lifeless | mriedem: I think publishing dev versions should be fine for you - 2015.1.1.0a2.dev4 (what stable/kilo's semver version is right now) is > 2015.1.0 and < 2015.1.1 | 18:42 |
| *** ivar-laz_ has joined #openstack-infra | 18:43 | |
| *** ivar-laz_ has quit IRC | 18:43 | |
| lifeless | mriedem: the 1.1 in the rpm mapping is missing because there isn't a version that both starts with 2015.1.1 and is < 2015.1.1 in rpm land | 18:43 |
| mriedem | lifeless: ok | 18:43 |
| *** sdake_ has joined #openstack-infra | 18:43 | |
| *** ildikov has joined #openstack-infra | 18:43 | |
| lifeless | mriedem: (AIUI, harlowja did most of the work there :)) | 18:43 |
| mriedem | lifeless: i have other issues anyway to deal with - like we are building the packages with pbr 0.6 on the build nodes, | 18:43 |
| lifeless | I just translated to code | 18:43 |
| harlowja | lifeless i thought u did most of the work | 18:43 |
| harlowja | oh | 18:43 |
| lifeless | mriedem: oh!. Heh :) | 18:43 |
| *** fawadkhaliq has joined #openstack-infra | 18:43 | |
| mriedem | but on our test nodes, we're pip installing tempest's requirements (we don't package tempest), which is putting pbr 0.11 on the test nodes | 18:43 |
| mriedem | so shit breaks | 18:44 |
| *** ivar-laz_ has joined #openstack-infra | 18:44 | |
| mriedem | b/c pbr 0.11 doesn't like pbr 0.6 | 18:44 |
| lifeless | yeah, 0.6 is, uhm, dated. | 18:44 |
| * anteaya shakes a finger at mriedem | 18:44 | |
| mriedem | lifeless: pbr 0.6 is still the min version in g-r | 18:44 |
| zaro | clarkb: seems to be blocked on ioAvailable.select() line 344 | 18:44 |
| mriedem | so we haven't moved up | 18:44 |
| mriedem | anteaya: i learned it from watching jogo | 18:44 |
| lifeless | mriedem: whats the issue? we might be able to fix it in the next pbr release | 18:44 |
| harlowja | lifeless ah ya, i remember, i've done to much with rpm specs and sorting and crap prior, lol, so i guess i braindumped some of that to u :-P | 18:44 |
| mriedem | lifeless: we have a 2015.1.ibm.rc1 tag on our source | 18:44 |
| * anteaya shakes a finger at jogo for good measure | 18:44 | |
| mriedem | lifeless: so that's what's in our pkg-info | 18:45 |
| mriedem | b/c pbr 0.6 is cool with that | 18:45 |
| lifeless | fungi: mordred: https://review.openstack.org/180265 moderately important. Possibly critically :) | 18:45 |
| clarkb | zaro: I am trying to get logging set to allow debug lines but that is what I expected | 18:45 |
| anteaya | I'm having difficulty reading the graph did all our in use nodes just drasticly shrink? | 18:45 |
| mriedem | lifeless: but at runtime, pbr 0.11 is hating the 2015.1.ibm.rc1 | 18:45 |
| lifeless | mriedem: oh wow. Yes, we have no defined mapping for that at all. | 18:45 |
| anteaya | not that we had many to start off with | 18:46 |
| mriedem | lifeless: as in http://paste.openstack.org/show/215043/ | 18:46 |
| *** ivar-lazzaro has quit IRC | 18:46 | |
| lifeless | I don't believe PEP-440 has a vendor labelling mechanism; we could possibly squash it into local versions | 18:46 |
| *** sdake has quit IRC | 18:47 | |
| mriedem | lifeless: well, fwiw, python setup.py build was giving us the userwarning: http://paste.openstack.org/show/215045/ | 18:47 |
| mriedem | we weren't heeding it | 18:47 |
| mriedem | b/c python setup.py build doesn't fail on that | 18:47 |
| lifeless | mriedem: we could in principle partse 2015.1.ibm.rc1 -> 2015.1.0.rc1+ibm | 18:47 |
| lifeless | mriedem: I was commenting in the nova patch the other day that the distro package override logic should move to pbr | 18:47 |
| lifeless | so we can offer it to all projects consistently. I don't know if vendor strings are part of that. | 18:48 |
| *** maishsk_afk has quit IRC | 18:48 | |
| lifeless | but if they are, I suspect local identifiers will be needed | 18:48 |
| lifeless | mriedem: heh yes. | 18:48 |
| mriedem | well, i think my short term answer is to not allow tempest to install pbr 0.11 from pip on our test nodes | 18:48 |
| lifeless | uhm, pbr 0.6 in g-r. | 18:48 |
| mriedem | for kilo | 18:48 |
| lifeless | I think thats bogus. | 18:48 |
| lifeless | For everything | 18:49 |
| *** maishsk has joined #openstack-infra | 18:49 | |
| mriedem | yeah, because you can't build with pbr 0.6 and run with pbr 0.11 if the versions aren't kosher with 0.11 | 18:49 |
| lifeless | because: we had to fixup pep-440 issues in the versions generated about 4 months ago | 18:49 |
| lifeless | it was a mad scramble | 18:49 |
| lifeless | and the only fixed versions are 0.10 and 0.11 | 18:49 |
| mriedem | lifeless: i'd be +1 on updating the min version on pbr to .10 in g-r on master :) | 18:49 |
| lifeless | the reason g-r didn't change is oversight, because nothing has (and nothing can have) a versioned setup_requires | 18:49 |
| clarkb | zaro: I am just using a new build with log.debug replaced with log.info... | 18:50 |
| lifeless | mriedem: I'll propose a 0.11 bump, if we're bumping, we may as well get everything consistent | 18:50 |
| mriedem | lifeless: yeah https://github.com/openstack/nova/blob/master/setup.py#L29 | 18:50 |
| clarkb | zaro: but I want to confirm that we process the NOOP event | 18:50 |
| *** dboik has joined #openstack-infra | 18:50 | |
| mriedem | lifeless: that used to have a min version in setup_requires | 18:50 |
| *** dboik_ has quit IRC | 18:50 | |
| mriedem | long ago | 18:50 |
| lifeless | yup | 18:50 |
| mriedem | that's why our pbr package was still 0.6 | 18:50 |
| lifeless | it breaks if it does | 18:50 |
| lifeless | mriedem: I linked the bug which documents why it breaks in my 1.0 thread yesterday, I can dig it up if you like) | 18:51 |
| mriedem | so short term for kilo i have to cap pbr < 0.11 in our stuff, for master we (ibm) need to update our build/test environments to use pbr 0.11 | 18:51 |
| mriedem | lifeless: that's ok, i'm in enough rabbit holes at once as it is | 18:51 |
| mriedem | *and have to leave work in an hour | 18:51 |
| zaro | clarkb: i don't see NOOP from server. | 18:51 |
| clarkb | zaro: no the worker sends it to itself | 18:52 |
| zaro | clarkb: that's probably becuase no GRAB_JOB/PREE_SLEEP | 18:52 |
| openstackgerrit | lifeless proposed openstack/requirements: Bump pbr versions. https://review.openstack.org/180271 | 18:53 |
| lifeless | mriedem: ^ | 18:53 |
| *** tnovacik has joined #openstack-infra | 18:53 | |
| *** dboik has quit IRC | 18:53 | |
| clarkb | zaro: that is what sends the GRAB_JOB | 18:54 |
| clarkb | zaro: at least in theory, but its not happening | 18:54 |
| *** dboik has joined #openstack-infra | 18:54 | |
| mriedem | lifeless: thanks, +1 | 18:55 |
| lifeless | if folk have been using 0.6 this whole time | 18:56 |
| lifeless | I can understand distros going 'omg its broken patch it out' :) | 18:56 |
| jhesketh | Morning | 18:56 |
| lifeless | jhesketh: 5am?! | 18:56 |
| pleia2 | infra meeting means early day for jhesketh | 18:56 |
| jhesketh | Pretty much | 18:57 |
| lifeless | there's eary | 18:57 |
| lifeless | and there's EARLY | 18:57 |
| *** pblaho has joined #openstack-infra | 18:57 | |
| anteaya | morning jhesketh | 18:57 |
| zaro | clarkb: i was expecting a NOOP from server after CAN_DO but that seems to not be happening. | 18:57 |
| jhesketh | lifeless: are you suggesting we move the meeting per chance? :-P | 18:57 |
| jhesketh | anteaya: o/ | 18:57 |
| anteaya | my CAN_DO graph just nosedived recently | 18:58 |
| anteaya | if I am looking at the correct graph | 18:58 |
| lifeless | jhesketh: Nope :) | 18:58 |
| zaro | clarkb: maybe NOOP from server is what's needed? | 18:58 |
| pleia2 | anteaya: here's to graph collection | 18:58 |
| pleia2 | err correction | 18:58 |
| clarkb | zaro: well in theory we are giving ourselves a NOOP to trigger the GRAB_JOB action | 18:58 |
| anteaya | pleia2: I'm trying to follow along | 18:59 |
| anteaya | it is tough | 18:59 |
| lifeless | jhesketh: since I'm not tracking tripleo so closely, the conflict doesn't bother me now, and 7am I can usually do | 18:59 |
| *** btully has joined #openstack-infra | 18:59 | |
| *** annegentle has joined #openstack-infra | 18:59 | |
| *** bknudson has quit IRC | 19:00 | |
| *** mmedvede has joined #openstack-infra | 19:00 | |
| fungi | meeting time | 19:01 |
| fungi | keystone have wrapped up | 19:01 |
| *** gokrokve has quit IRC | 19:01 | |
| *** gokrokve has joined #openstack-infra | 19:01 | |
| *** Longgeek_ has quit IRC | 19:02 | |
| *** soren_ has joined #openstack-infra | 19:02 | |
| *** soren_ has quit IRC | 19:03 | |
| *** soren_ has joined #openstack-infra | 19:03 | |
| sdake_ | hey my gate is failing with NOT_REGISTERED | 19:03 |
| sdake_ | https://review.openstack.org/#/c/178048/ | 19:03 |
| sdake_ | what does that mean? | 19:03 |
| *** soren_ is now known as soren | 19:04 | |
| anteaya | zuul is unhappy today | 19:04 |
| sdake_ | thanks so not a bug on my part | 19:04 |
| anteaya | uncertain | 19:04 |
| *** hdd has quit IRC | 19:04 | |
| anteaya | might be hard to diagnose given today's zuul illness | 19:04 |
| *** HeOS has joined #openstack-infra | 19:04 | |
| sdake_ | unfortunately it says "fiel not foudn" and there are no logs | 19:05 |
| sdake_ | and its causing all our gating to fail :( | 19:05 |
| clarkb | its not your fault, there are no logs because there was no job | 19:05 |
| fungi | clarkb: jeblair: another interesting problem... we apparently have nodes in use which still look ready in nodepool. i suspect an unrelated zmq disconnect is part of the mess here. restarting nodepool momentarily | 19:06 |
| *** baoli_ has quit IRC | 19:06 | |
| *** jistr has joined #openstack-infra | 19:08 | |
| fungi | nodepoold has been restarted. i'll keep an eye out for nodes still stuck in ready long enough to no longer be running jobs | 19:08 |
| *** shashankhegde has quit IRC | 19:09 | |
| *** ZZelle_ has joined #openstack-infra | 19:09 | |
| clarkb | zaro: I figured it out sort of | 19:11 |
| *** shivharis has joined #openstack-infra | 19:11 | |
| clarkb | zaro: will need to trace the code paths to understand if my fix is sufficient, but basic problem is that when we start work() we are already initialized so we never add the NOOP to the queue | 19:11 |
| *** dizquierdo is now known as dizquierdo_afk | 19:12 | |
| shivharis | clarkb: hi | 19:13 |
| *** gyee has quit IRC | 19:13 | |
| nibalizer | clarkb: we didn't get to it last week, but this week lets talk about your plan to break my refactor up | 19:13 |
| *** tiswanso_ has quit IRC | 19:14 | |
| *** tkelsey has joined #openstack-infra | 19:14 | |
| clarkb | shivharis: currently in a meeting so mostly focused on that | 19:14 |
| *** ayoung has quit IRC | 19:14 | |
| shivharis | clarkb: np | 19:15 |
| *** EmilienM is now known as EmilienM|afk | 19:15 | |
| *** gyee has joined #openstack-infra | 19:15 | |
| openstackgerrit | Sergey Kolekonov proposed openstack-infra/project-config: Add apps-catalog/ci group for apps-catalog project https://review.openstack.org/180201 | 19:15 |
| * waynr gets head caught in tangle of spaghetti code | 19:15 | |
| openstackgerrit | Sergey Kolekonov proposed openstack-infra/project-config: Add apps-catalog-ci group for apps-catalog project https://review.openstack.org/180201 | 19:16 |
| shivharis | fungi: hi | 19:16 |
| fungi | shivharis: hi. the meeting to which clarkb refers is the infrastreucture team meeting. we're _all_ focused on that | 19:18 |
| *** rlandy has quit IRC | 19:18 | |
| fungi | er, infrastructure | 19:18 |
| *** baoli has joined #openstack-infra | 19:18 | |
| shivharis | fungi: np | 19:19 |
| *** tkelsey has quit IRC | 19:19 | |
| *** pradk has quit IRC | 19:19 | |
| *** chriscowley has quit IRC | 19:21 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 19:22 |
| sdague | zaneb: +A | 19:22 |
| markmcclain | zaneb: was waiting on sdague to confirm | 19:23 |
| *** subscope_ has joined #openstack-infra | 19:27 | |
| *** tiswanso has joined #openstack-infra | 19:27 | |
| *** annegentle has quit IRC | 19:28 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for swift-object-server bug 1451992 https://review.openstack.org/180278 | 19:30 |
| openstack | bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/1451992 | 19:30 |
| *** rloo has joined #openstack-infra | 19:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return Bunch objects instead of plain dicts https://review.openstack.org/180269 | 19:33 |
| *** mtanino has quit IRC | 19:34 | |
| *** Ala has quit IRC | 19:35 | |
| *** Sukhdev has quit IRC | 19:35 | |
| *** yfried is now known as yfried|afk | 19:35 | |
| *** doug-fish has quit IRC | 19:36 | |
| davidlenwell | when adding a new project.. does the upstream: git repo have to have the same name as the new repo? | 19:37 |
| fungi | nope | 19:38 |
| davidlenwell | sweet .. thanks | 19:38 |
| *** weshay has quit IRC | 19:39 | |
| *** ildikov has quit IRC | 19:41 | |
| *** sdake has joined #openstack-infra | 19:42 | |
| *** emagana has quit IRC | 19:42 | |
| *** sdake has quit IRC | 19:42 | |
| *** emagana has joined #openstack-infra | 19:42 | |
| *** sdake_ has quit IRC | 19:45 | |
| *** sdake has joined #openstack-infra | 19:46 | |
| *** btully has quit IRC | 19:47 | |
| *** sdake_ has joined #openstack-infra | 19:47 | |
| *** mestery has quit IRC | 19:48 | |
| *** vhoward has joined #openstack-infra | 19:48 | |
| *** jistr has quit IRC | 19:49 | |
| zaneb | sdague, markmcclain: thanks! | 19:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update server API for get/list/search interface https://review.openstack.org/180293 | 19:51 |
| *** sdake has quit IRC | 19:51 | |
| *** weshay has joined #openstack-infra | 19:52 | |
| *** shivharis has quit IRC | 19:52 | |
| *** shivharis has joined #openstack-infra | 19:52 | |
| *** ildikov has joined #openstack-infra | 19:54 | |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: adding akanda horizon plugin to stackforge https://review.openstack.org/180294 | 19:54 |
| *** shashankhegde has joined #openstack-infra | 19:54 | |
| *** yfried|afk is now known as yfried | 19:55 | |
| *** Rockyg has joined #openstack-infra | 19:59 | |
| *** maishsk has quit IRC | 19:59 | |
| *** maishsk has joined #openstack-infra | 20:00 | |
| anteaya | fungi: no storm like a perfect storm | 20:00 |
| fungi | shivharis: what did you need? | 20:01 |
| *** sdake has joined #openstack-infra | 20:01 | |
| *** ildikov has quit IRC | 20:01 | |
| AJaeger_ | fungi, are you storm fighting now or could you help me with a review where I have a question? | 20:01 |
| jeblair | fungi, clarkb, zaro: i do still think the zuul sluggishness is due to geard sluggishness, possibly because we have so many ready nodes (even though they may not be emitting as many registration events as before) | 20:01 |
| jeblair | fungi, clarkb, zaro: hopefully having fewer ready nodes sitting in pre-sleep (and therefore generating a lot of noop traffic) should help, and also perhaps my change to drop the work_status packets. | 20:02 |
| fungi | jeblair: i'm unsure they're actually ready. i've ssh'd into some of the devstack-trusty nodes which claim to be ready for up to an hour and they had devstack running and doing stuff | 20:02 |
| zaro | jeblair: yes, that was based on last version of your change: https://review.openstack.org/#/c/174607/ | 20:02 |
| *** mriedem has quit IRC | 20:02 | |
| yolanda | here, i wanted to ask for some reviews on nodepool image testing spec, please: https://review.openstack.org/139598 | 20:03 |
| jeblair | fungi: oh, that was why you mentioned zmq earlier | 20:03 |
| fungi | yep | 20:03 |
| yolanda | and for the ton of downstream-puppet changes as well :) | 20:03 |
| fungi | AJaeger_: several fires at once. is it urgent? | 20:03 |
| zaro | jeblair: incidentally that change is .23 on review-dev. | 20:03 |
| shivharis | fungi: wanted to find out how to add members to the <projectname>-core group - did once some time back but cannot seem to find it | 20:04 |
| jeblair | zaro: ah, that splains it | 20:04 |
| anteaya | shivharis: are you currently a member of the core group? | 20:04 |
| shivharis | yes | 20:04 |
| AJaeger_ | fungi: Then continue firefighting and after you rested, please see my last comment in https://review.openstack.org/#/c/173408/ ;) | 20:04 |
| shivharis | anteaya: yes, core and admin | 20:04 |
| anteaya | if you are signed into gerrit and go to people > groups > name of group-core | 20:05 |
| *** sdake_ has quit IRC | 20:05 | |
| anteaya | you should see a text field and an add button | 20:05 |
| AJaeger_ | fungi: not urgent for me but the glusterfs guys are fighting some time already... | 20:05 |
| *** annegentle has joined #openstack-infra | 20:05 | |
| *** _nadya_ has joined #openstack-infra | 20:05 | |
| shivharis | anteaya: thanks | 20:05 |
| *** yfried is now known as yfried|afk | 20:05 | |
| anteaya | shivharis: welcome | 20:06 |
| *** doug-fish has joined #openstack-infra | 20:06 | |
| *** heyongli has quit IRC | 20:07 | |
| *** freerunner has joined #openstack-infra | 20:08 | |
| *** heyongli has joined #openstack-infra | 20:09 | |
| fungi | AJaeger_: you mean mtreinish's comment on that review? | 20:09 |
| AJaeger_ | fungi: no, my comment on name collision | 20:10 |
| *** sdake has quit IRC | 20:10 | |
| *** sdake has joined #openstack-infra | 20:10 | |
| AJaeger_ | fungi: I prefer to rename the job so that it cannot collision with the plugin one | 20:10 |
| *** kgiusti has quit IRC | 20:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering https://review.openstack.org/179988 | 20:11 |
| clarkb | I have run ~100 trivial shell jobs on the master executor using ^ with 10 clients | 20:11 |
| * AJaeger_ wonders whether I'm insisting here on too much and should just say go for it - or really insist... | 20:11 | |
| *** achanda has quit IRC | 20:11 | |
| clarkb | zaro: jeblair the needed fix was that when we start running in work() we are already initialized (which is weird but that can be a followup fix) so you have to queue the noop event first thing | 20:12 |
| AJaeger_ | fungi: really put this on the back and continue fire fighting ... | 20:12 |
| clarkb | otherwise we never queue the noop event | 20:12 |
| jeblair | clarkb: ooooh | 20:12 |
| * clarkb is going to do more manual testing of that | 20:12 | |
| fungi | AJaeger_: seems like a fine suggestion to me. should we though rename other jobs which also use a similar naming pattern so as to avoid the same sorts of conflicts in the future? | 20:13 |
| *** ildikov has joined #openstack-infra | 20:13 | |
| AJaeger_ | Like the ceph one? Yeah, I could propose that to make a point ;) | 20:14 |
| AJaeger_ | any ideas on how to rename? | 20:14 |
| ianw | fungi / AJaeger_ : has anyone ever considered arguments to macros in jjb, so we could retain templates but pass through things like the regex for skipping tests? | 20:14 |
| ianw | i can't think of a way to do that as is | 20:15 |
| fungi | ianw: you can pass parameters to a builder macro, for example | 20:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update server API for get/list/search interface https://review.openstack.org/180293 | 20:16 |
| AJaeger_ | ianw, so invoking it using: "variable: regex"? That should work today... | 20:16 |
| *** otter768 has joined #openstack-infra | 20:16 | |
| clarkb | hrm my minimal jenkins doesn't seem to support running jobs by label | 20:16 |
| *** shivharis has quit IRC | 20:16 | |
| *** shivharis has joined #openstack-infra | 20:17 | |
| fungi | AJaeger_: probably the plugins should have namespacing like ...-plugin-{plugin}{branch-designator} | 20:17 |
| clarkb | want to test that the reset abilities works properly while jobs are running | 20:17 |
| *** maishsk has quit IRC | 20:17 | |
| clarkb | I actually should be able to just add a new job | 20:17 |
| AJaeger_ | fungi: Let me propose that. thanks | 20:17 |
| harlowja | periodic ping to get someone to review https://review.openstack.org/#/c/171393/ :-P | 20:17 |
| harlowja | ^ and followups that add tests | 20:18 |
| harlowja | ^^^^^^ tests are good | 20:18 |
| clarkb | yup it seems to have done the correct thing, reset after work complete, then sent can dos then sent grab jobs | 20:18 |
| *** dizquierdo_afk is now known as dizquierdo | 20:19 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: WIP: Use shade to get the client objects https://review.openstack.org/168603 | 20:19 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Upgrade Gerrit from 2.8 to ver 2.10 on review.o.o https://review.openstack.org/155463 | 20:19 |
| greghaynes | mordred: test indicates that change doesnt work as expected :( | 20:19 |
| greghaynes | mordred: er, ^ that test | 20:20 |
| *** mtanino has joined #openstack-infra | 20:20 | |
| ianw | AJaeger_: ahh, ok. so why not setup a builder like devstack-rally-gate for plugins that we can pass arbitrary test skips through to, and avoid having to create duplicate jobs? | 20:20 |
| clarkb | zaro: jeblair: I am going to go grab lunch now that those two basic things seem to work | 20:20 |
| greghaynes | yes, food | 20:20 |
| zaro | clarkb: will take a look | 20:21 |
| clarkb | zaro: thanks, independent testing is likely also useful | 20:21 |
| clarkb | zaro: since that should use different timings and stuff | 20:21 |
| *** otter768 has quit IRC | 20:21 | |
| *** shivharis has quit IRC | 20:21 | |
| *** shivharis has joined #openstack-infra | 20:22 | |
| *** palendae has left #openstack-infra | 20:22 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Make gerritbot report changes from Gerrit 2.10.2 branch https://review.openstack.org/155450 | 20:22 |
| *** maishsk has joined #openstack-infra | 20:23 | |
| AJaeger_ | ianw: So, doing something like if [ "{regex}" != "" ] ; then export DEVSTACK_GATE_TEMPEST_REGEX="{regex}" fi ? | 20:25 |
| AJaeger_ | that would mean we would need to add to each job that uses this the regex. | 20:25 |
| AJaeger_ | Let me propose that as separate step and then we can review... | 20:26 |
| AJaeger_ | ... and discuss | 20:26 |
| ianw | AJaeger_: yeah, something like that ... we have a nice template for plugin devstack jobs, but if you want to skip anything you can't use it | 20:26 |
| ianw | AJaeger_: i'm happy to work on that. i want to clean up the centos7/fedora jobs in a similar way as well | 20:26 |
| ianw | AJaeger_: so feel free to ignore my -1 and i'll look at it soonish | 20:27 |
| *** abregman has joined #openstack-infra | 20:27 | |
| *** gyee has quit IRC | 20:28 | |
| *** ildikov has quit IRC | 20:28 | |
| ianw | i'm not sure i agree with tempest's position that it's happy to be broken for known issues with drivers, but anyway | 20:28 |
| fungi | ianw: i believe it's position is that it's willing to remove testing and general support for drivers if they're broken or untestable, but not simply add overrides/skips | 20:28 |
| fungi | though mtreinish can certainly clarify | 20:29 |
| mtreinish | fungi: yeah that's a good high level explanation | 20:29 |
| ianw | I don't pretend to know the history. it seems if there was a desire there are certainly ways you could keep skip lists in some common format within tempest itself | 20:30 |
| ianw | instead of leaving it up to devstack-gate jobs | 20:30 |
| *** AL34N1X has quit IRC | 20:31 | |
| ianw | my main point is that we should be striving for less & more templated jobs in jjb, not more | 20:31 |
| mtreinish | ianw: that exact thing was proposed maybe 2 years ago and it wasn't accepted for the same reason | 20:32 |
| mtreinish | it's not tempest's job to maintain what works with what driver, it just tests that an openstack deployment works as expected through the rest api | 20:34 |
| mtreinish | it's a layer separation thing | 20:34 |
| jlvillal | Would this be the correct channel to ask about Stackaltyics? When going here for Liberty: http://stackalytics.com/?release=liberty I don't see an Ironic project listed. | 20:34 |
| *** tiswanso has quit IRC | 20:34 | |
| mtreinish | there are already flags to say a cloud does or does not support a particular feature | 20:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename tempest-dsvm-full-{plugin} https://review.openstack.org/180308 | 20:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Parse regex to skip to tempest-dsvm-full-plugin jobs https://review.openstack.org/180309 | 20:34 |
| mtreinish | which is what should be leveraged to skip a test if a driver is broken for something | 20:34 |
| AJaeger_ | ianw, fungi here're my two patches ^ | 20:35 |
| *** tiswanso has joined #openstack-infra | 20:35 | |
| ianw | mtreinish: well, you could add a layer ontop in a yaml file or something that kept some centralised list of issues, without having to add @skip to every function. at least people would know where to look | 20:35 |
| *** achanda has joined #openstack-infra | 20:35 | |
| fungi | okay, i'm deleting the devstack-trusty images from today with the expectation that will provide short-term relief for bug 1451992 while we continue digging into the underlying cause | 20:35 |
| openstack | bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/1451992 | 20:35 |
| anteaya | bit of meeting bot lag in the meeting | 20:35 |
| anteaya | like that adds to all the other things | 20:35 |
| *** miqui has quit IRC | 20:36 | |
| mtreinish | ianw: what exactly are you trying to solve? Is it just that using a selection regex with testr doesn't report on the console that the test isn't being run? | 20:38 |
| *** e0ne has quit IRC | 20:38 | |
| *** Trozz has quit IRC | 20:39 | |
| *** Trozz has joined #openstack-infra | 20:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Optionally pass regex to skip to tempest-dsvm-full-plugin jobs https://review.openstack.org/180309 | 20:40 |
| *** persia has quit IRC | 20:40 | |
| *** _nadya_ has quit IRC | 20:40 | |
| *** ildikov has joined #openstack-infra | 20:41 | |
| *** d34dh0r53 has quit IRC | 20:41 | |
| *** AJaeger has quit IRC | 20:41 | |
| *** AJaeger_ has quit IRC | 20:41 | |
| *** fungi has quit IRC | 20:41 | |
| *** persia has joined #openstack-infra | 20:41 | |
| *** pblaho has quit IRC | 20:41 | |
| dims | fungi: cffi version mismatch? (looks like tripleo guys saw it before - https://bugs.launchpad.net/tripleo/+bug/1340776/comments/3) | 20:42 |
| openstack | Launchpad bug 1340776 in tripleo "Swift fails to start in overcloud" [High,Fix released] - Assigned to Eamonn O'Toole (eamonn-otoole) | 20:42 |
| *** mwagner_lap has quit IRC | 20:42 | |
| ianw | mtreinish: my beef, much as it is, is that devstack-gate isn't the only way to run tempest, so we know it doesn't work in various ways, but there's no way for anyone to realise that | 20:43 |
| ianw | the information about what works and what doesn't is scattered in regex's to various jobs all over the jjb config | 20:44 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add port resource methods https://review.openstack.org/179968 | 20:45 |
| mtreinish | ianw: well sort of. The issues is deeper than just tempest, if a tempest test doesn't work then that doesn't work for any cloud. How is an end user using the api supposed to know? | 20:45 |
| *** d34dh0r53 has joined #openstack-infra | 20:45 | |
| *** ivar-laz_ has quit IRC | 20:45 | |
| mtreinish | encoding that into tempest somehow doesn't solve the fundamental issue, I would argue that it would probably mask it even further | 20:45 |
| *** amitgandhinz has quit IRC | 20:46 | |
| *** ivar-lazzaro has joined #openstack-infra | 20:46 | |
| *** ivar-lazzaro has quit IRC | 20:46 | |
| *** AJaeger has joined #openstack-infra | 20:47 | |
| *** r-daneel has joined #openstack-infra | 20:47 | |
| *** fungi has joined #openstack-infra | 20:47 | |
| *** ivar-lazzaro has joined #openstack-infra | 20:47 | |
| ianw | mtreinish: well, i agree that say, hypervisor assisted snapshots don't work the same way with glusterfs and skipping the test in tempest doesn't help anyone trying to do that outside tempest | 20:48 |
| *** sabeen has joined #openstack-infra | 20:49 | |
| fungi | bad time for freenode to be breaking | 20:49 |
| *** tonytan4ever has joined #openstack-infra | 20:50 | |
| fungi | clarkb: jeblair: i need to disappear for a couple hours unfortunately, but will check back in on things when i return | 20:50 |
| clarkb | fungi before you go anything I can take over? | 20:51 |
| clarkb | or are we mostly recovering at this point? | 20:51 |
| fungi | deleting ready nodes >2 hours old is generally safe at this point i think, so i've done that | 20:51 |
| fungi | in another hour, deleting ready nodes >1 hour should be safe | 20:51 |
| clarkb | ok | 20:51 |
| clarkb | can do | 20:51 |
| *** fawadkhaliq has quit IRC | 20:51 | |
| fungi | thanks! that should get things moving faster i think | 20:52 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Create XmlBuilder class, split Builder.update_jobs https://review.openstack.org/180315 | 20:52 |
| ianw | mtreinish: but i still think that keeping this information within tempest would be better. anyway, adding arguments to the plugin template as proposed i think at least is a good way to avoid job-explosion in project-config | 20:52 |
| fungi | also check in with #-dev and see how the xattr/cffi debugging is going in case they need more infra assistance besides the image removal i did | 20:52 |
| fungi | back as soon as i can be | 20:52 |
| *** patrickeast has quit IRC | 20:52 | |
| *** aysyd has quit IRC | 20:52 | |
| *** patrickeast has joined #openstack-infra | 20:53 | |
| *** pabelanger has quit IRC | 20:53 | |
| clarkb | kk | 20:53 |
| mtreinish | ianw: btw, have you seen: http://git.openstack.org/cgit/openstack/os-testr/commit/?id=ff3dc87d6b496af625652838936288e0c3882853 that was my idea for doing what your describing | 20:53 |
| mtreinish | the plugins could then just put an exclude file in a known place in the plugin tree and the jobs would use that to exclude if present | 20:54 |
| *** gyee has joined #openstack-infra | 20:54 | |
| *** amitgandhinz has joined #openstack-infra | 20:54 | |
| openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: Status: Don't raise "abort" as error to the user https://review.openstack.org/172241 | 20:55 |
| ianw | mtreinish: that's certainly going down the right path -- keep the skip information right there alongside the plugin. i have to dash for a bit, i will take a closer look | 20:55 |
| openstackgerrit | Merged openstack-infra/release-tools: Add script to cleanup of translations https://review.openstack.org/172827 | 20:56 |
| openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: Status: Don't raise "abort" as error to the user https://review.openstack.org/172241 | 20:56 |
| *** sparkycollier has joined #openstack-infra | 20:56 | |
| *** shivharis has quit IRC | 20:58 | |
| *** frobware_ has joined #openstack-infra | 20:58 | |
| *** nelsnelson has quit IRC | 21:00 | |
| *** hdd has joined #openstack-infra | 21:02 | |
| *** shashankhegde has quit IRC | 21:03 | |
| anteaya | I am going to get a walk in | 21:06 |
| *** maishsk has quit IRC | 21:06 | |
| *** radez is now known as radez_g0n3 | 21:06 | |
| *** btully has joined #openstack-infra | 21:07 | |
| clarkb | pleia2: re https://review.openstack.org/#/c/179840/1/doc/source/testing.rst can you try also granting perms for openstack_citest@127.0.0.1 and @::1 ? | 21:07 |
| clarkb | pleia2: I bet its a localhost isn't resolved the way we expect it there issue | 21:07 |
| *** craigbr has joined #openstack-infra | 21:08 | |
| jeblair | clarkb: who else should we ask to review your change and mine? | 21:08 |
| jeblair | clarkb: or should we just merge them now? | 21:08 |
| *** hdd has quit IRC | 21:08 | |
| clarkb | jeblair: zaro, jhesketh maybe? | 21:09 |
| clarkb | jeblair: I am pretty confident in yours | 21:09 |
| clarkb | jeblair: mine is the scary one :) | 21:09 |
| jeblair | clarkb: but you actually ran it, right? :) | 21:09 |
| clarkb | jeblair: I did, but only against a local jenkins (of the same version) with a a test job and a handful of clients | 21:09 |
| pleia2 | clarkb: will do, also testing the devstack one and have a few comments for that too | 21:09 |
| clarkb | jeblair: I also added new jobs to force reregistration | 21:09 |
| clarkb | jeblair: and that all looked good | 21:09 |
| clarkb | pleia2: cool, thanks | 21:10 |
| jeblair | clarkb: that seems good to me | 21:10 |
| zaro | clarkb, jeblair : i'm reviewing, if you want to wait a little. | 21:10 |
| jeblair | zaro: excellent, will do, thanks. :) | 21:10 |
| jeblair | jhesketh: let us know if you want to review those gearman-plugin changes too | 21:11 |
| *** annegentle has quit IRC | 21:11 | |
| *** annegentle has joined #openstack-infra | 21:12 | |
| *** bswartz has quit IRC | 21:12 | |
| dstufft | wooo, cffi 1.0 is gonna be awesome | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| zaro | clarkb: i don't see a GRAB_JOB after register and CAN_DO? | 21:16 |
| clarkb | zaro: with the latest patchset? | 21:16 |
| zaro | clarkb: yeah | 21:16 |
| zaro | clarkb: I guess we want NOOP right? | 21:17 |
| clarkb | zaro: no you want GET_JOB | 21:17 |
| clarkb | er GRAB_JOB | 21:17 |
| *** dkranz has quit IRC | 21:18 | |
| *** dprince has quit IRC | 21:18 | |
| zaro | clarkb: so i'm doing set node offline, SUBMIT_JOB, bring node back online | 21:18 |
| zaro | clarkb: register happens, but not GRAB_JOB | 21:19 |
| clarkb | zaro: so that may be a code path I didn't cover, I did check the temporarily offline then online case | 21:19 |
| clarkb | is that what you mean by offline or are you disconnecting? | 21:19 |
| *** claudiub has joined #openstack-infra | 21:20 | |
| zaro | clarkb: temp offline/online _not_ disconnect | 21:20 |
| *** panda is now known as panda|Zz | 21:20 | |
| clarkb | ok, I did test that, let me try again | 21:20 |
| *** tiswanso has quit IRC | 21:21 | |
| clarkb | zaro: huh this time I see what you see | 21:21 |
| clarkb | let me dig in | 21:22 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add keystone service resource methods https://review.openstack.org/177621 | 21:22 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add keystone endpoint resource methods https://review.openstack.org/180328 | 21:22 |
| *** jtriley has quit IRC | 21:22 | |
| *** baoli has quit IRC | 21:23 | |
| clarkb | gah I think that is because we call register jobs directly from that jenkins event listener thing | 21:24 |
| *** baoli_ has joined #openstack-infra | 21:24 | |
| clarkb | I see the bug I think | 21:27 |
| *** gyee has quit IRC | 21:27 | |
| *** boris-42 has quit IRC | 21:28 | |
| clarkb | its a logic bug around me trying not to send extra NOOPs | 21:28 |
| clarkb | but this works around my serialization of register jobs and grab jobs | 21:29 |
| *** dizquierdo has quit IRC | 21:29 | |
| clarkb | might be a good follow up fix to deal with that separately | 21:30 |
| *** doug-fish has quit IRC | 21:30 | |
| *** shardy is now known as shardy_z | 21:31 | |
| *** ociuhandu has quit IRC | 21:32 | |
| clarkb | we should be calling setFunctions instead | 21:33 |
| *** sparkycollier has quit IRC | 21:33 | |
| *** dguerri is now known as _dguerri | 21:33 | |
| *** _dguerri is now known as dguerri | 21:33 | |
| *** dustins has quit IRC | 21:33 | |
| *** sdake_ has joined #openstack-infra | 21:35 | |
| mordred | clarkb, jeblair: I support you landing your gearman plugin changes | 21:36 |
| clarkb | ok I have tracked this down we have two different registerJobs() methods that do different things | 21:37 |
| clarkb | we have the one in the thread which sets setFunctions() on the worker whihc does the right thing | 21:37 |
| clarkb | but you cannot call registerJobs() on the worker from an arbitrary thread | 21:38 |
| jeblair | mordred, clarkb, fungi, zaro: i believe gear on zuul.o.o has regressed which may explain the performance degredation there | 21:38 |
| clarkb | I think we are stuck with this because we implemented some interfaces | 21:38 |
| *** dguerri is now known as _dguerri | 21:38 | |
| *** _dguerri is now known as dguerri | 21:38 | |
| *** sdake has quit IRC | 21:39 | |
| *** shashankhegde has joined #openstack-infra | 21:39 | |
| mordred | jeblair: like we landed a regression in the code? or we accidentally caused an old thing to install? | 21:40 |
| jeblair | mordred: the second | 21:40 |
| clarkb | huh | 21:40 |
| mordred | oh awesome | 21:40 |
| jeblair | maybe we fixed puppet installing things, and probably it installed the actual released version instead of a local thing | 21:40 |
| clarkb | like before our event loop performance changes? | 21:40 |
| clarkb | ah | 21:40 |
| jeblair | clarkb: yeah, it's at 0.5.5 | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering https://review.openstack.org/179988 | 21:41 |
| clarkb | zaro: ^ can you test that version, it fixes the logic bug | 21:41 |
| mordred | jeblair: so maybe we should release a gear? | 21:41 |
| jeblair | so this is probably my fault for not releasing gear and bumping zuul's dep | 21:41 |
| jeblair | mordred: sounds like a good idea :) | 21:41 |
| clarkb | +1 | 21:41 |
| jeblair | in the mean time, i would like to just fix it on server | 21:42 |
| clarkb | jeblair: sounds reasonable to me | 21:42 |
| jeblair | zuul hit 0 events so i saved the queue and restarted it | 21:43 |
| jeblair | i didn't know how long it would take to catch up again, so i took my opportunity | 21:43 |
| mordred | jeblair: ++ | 21:44 |
| zaro | clarkb: aren't those registerJobs syncronized? | 21:44 |
| clarkb | zaro: ya, they are because they are registerJobs against the thread not the worker | 21:44 |
| clarkb | zaro: which is really confusing for me but I believe its the interface we were given so meh | 21:45 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Make zaqarclient functional gate non-voting https://review.openstack.org/180332 | 21:46 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Extend Zaqar's gate jobs https://review.openstack.org/180333 | 21:46 |
| jeblair | geard is not pegging a cpu now | 21:46 |
| *** stevemar has joined #openstack-infra | 21:47 | |
| *** SumitNaiksatam has quit IRC | 21:47 | |
| zaro | clarkb: trying again now | 21:47 |
| clarkb | zaro: ty | 21:47 |
| clarkb | zaro: fwiw it did work for me locally using your test of tempoffline node, submit job, online node | 21:47 |
| *** doug-fish has joined #openstack-infra | 21:48 | |
| jeblair | gear 0.5.6 tagged | 21:49 |
| *** peristeri has quit IRC | 21:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Update to gear 0.5.6 https://review.openstack.org/180334 | 21:50 |
| *** sdake has joined #openstack-infra | 21:50 | |
| *** mriedem has joined #openstack-infra | 21:51 | |
| *** whoops has quit IRC | 21:51 | |
| jeblair | nodepool 0.1.0 tagged | 21:52 |
| jeblair | hasharMeeting: ^ | 21:52 |
| jeblair | clarkb: i think that takes the pressure off of the gearman-plugin changes | 21:53 |
| *** sdake__ has joined #openstack-infra | 21:54 | |
| clarkb | jeblair: ok | 21:54 |
| *** pcm_ has quit IRC | 21:54 | |
| *** sdake_ has quit IRC | 21:54 | |
| jeblair | clarkb: after the jjb change merges and we manually run it on all the jenkins masters, we should not see big ready node problems unless we merge another change that touches all jobs | 21:54 |
| jeblair | so we just need to get it fixed before we do that :) | 21:55 |
| clarkb | roger, that gets us back to where we were with the existing race in gearman plugin | 21:55 |
| clarkb | which while annoying wasn't nearly world shattering | 21:55 |
| *** ptoohill is now known as pothole | 21:55 | |
| *** tonytan4ever has quit IRC | 21:55 | |
| *** annegent_ has joined #openstack-infra | 21:55 | |
| *** mrmartin has quit IRC | 21:55 | |
| jeblair | yeah, we had a nice perfect storm of 3 distinct problems :) | 21:55 |
| jeblair | (jjb deleting jobs, gearman-plugin racing, and geard regressing) | 21:56 |
| *** cdent has quit IRC | 21:57 | |
| *** asettle has joined #openstack-infra | 21:58 | |
| *** sdake has quit IRC | 21:58 | |
| *** asettle has quit IRC | 21:58 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:58 | |
| *** esker has quit IRC | 21:58 | |
| pleia2 | clarkb: no luck with @127.0.0.1 or @::1 (have those and @localhost in the database now), it's odd since it seems to create them fine (I have about a billion mysql databases now ;)), but some part it gets to where it gets permission denied for them | 21:59 |
| * pleia2 confirms no typos and usernames/passwords | 21:59 | |
| *** annegentle has quit IRC | 22:00 | |
| clarkb | pleia2: oh, maybe its failing to clean them up | 22:00 |
| clarkb | pleia2: did you do the grant ALL on *.*? | 22:00 |
| *** Krinkle is now known as Krinkle|detached | 22:00 | |
| pleia2 | clarkb: yeah, and I thought it was a flush privileges issue at first, but that didn't help either | 22:01 |
| clarkb | pleia2: hrm | 22:01 |
| clarkb | let me look at my existing db | 22:01 |
| *** Swami has quit IRC | 22:03 | |
| *** SumitNaiksatam has quit IRC | 22:03 | |
| BobBall | Any suggestions for the best way to enable a local login for nodes built by nodepool with modified standard scripts? Got an issue where a large proportion of images don't get their networking when built by nodepool and I need to login using the console to investigate as I can't ssh | 22:03 |
| *** changbl has quit IRC | 22:04 | |
| clarkb | BobBall: add an adduser to your build scripts | 22:04 |
| clarkb | BobBall: have it set a password for some dummy user | 22:05 |
| clarkb | and possibly add it to the sudo group at the same time | 22:05 |
| clarkb | or make root have a password | 22:05 |
| BobBall | heh... yes... sorry I am clearly being thick... I was thinking the puppet scripts reset the root password | 22:05 |
| BobBall | but clearly I could just add a new user *sigh* | 22:05 |
| *** bnemec has quit IRC | 22:05 | |
| clarkb | I don't think puppet touches root | 22:05 |
| BobBall | it's 11pm and I'm tired. That's my excuse. | 22:05 |
| clarkb | but it may, would have to double check | 22:05 |
| BobBall | Oh... I could have sworn something was clearing my root password | 22:06 |
| *** bnemec has joined #openstack-infra | 22:08 | |
| clarkb | pleia2: my mysql user table also has openstack_citest@% which is wildcard match I think | 22:09 |
| clarkb | pleia2: maybe give that a go? looking at perms now | 22:09 |
| clarkb | also I seem to have thousands of rows in the user table which has me worried :/ | 22:09 |
| clarkb | oh they are all randomized users? is nodepool creating those? | 22:11 |
| pleia2 | I only have 10, so it's not doing something crazy | 22:11 |
| pleia2 | randomized database names for me | 22:11 |
| pleia2 | actually | localhost | iHiDbXEd_3672 | 22:11 |
| sdake__ | willa a recheck fix a "Need Verified"state? | 22:11 |
| pleia2 | have a couple of those | 22:11 |
| * pleia2 headscratch | 22:12 | |
| clarkb | pleia2: ya thats what I have too, I think maybe if it doesn't clean up properly then we may leak the users too? | 22:12 |
| *** packet has quit IRC | 22:12 | |
| clarkb | sdake__: if the verified state is -2 or -1 | 22:12 |
| *** jtriley has joined #openstack-infra | 22:12 | |
| sdake__ | the verified state is +1 | 22:12 |
| pleia2 | clarkb: yeah, it only seems to happen sometimes for me, I have dozens of databases but only ended up with 2 randomized users | 22:12 |
| sdake__ | but its been in needs verified for 8+ hours | 22:12 |
| *** amitgandhinz has quit IRC | 22:12 | |
| sdake__ | https://review.openstack.org/#/c/180021/ | 22:13 |
| sdake__ | our gate takes about 2 minutes to run :( | 22:13 |
| clarkb | pleia2: so ya my accounts have all the permissions, make sure you do the grant for all of them and maybe try the wildcard %? | 22:13 |
| *** erikmwilson has quit IRC | 22:14 | |
| pleia2 | clarkb: I'll give that a go | 22:14 |
| clarkb | sdake__: what you can do is reapprove it | 22:14 |
| sdake__ | thanks | 22:15 |
| sdake__ | i'll try that | 22:15 |
| *** soren has quit IRC | 22:15 | |
| clarkb | sdake__: the problem is since its already +1 verified the check queue is going to come back as +1 which is not a delta to the old vote so gerrit won't emit an event with the +1 | 22:15 |
| *** hodos has quit IRC | 22:15 | |
| clarkb | zaro: ^ any progress with your chnage to gerrit to fix that? | 22:15 |
| sdake__ | clarkb thanks | 22:16 |
| sdake__ | clarkb now the gate says "NOT REGISTERED" | 22:17 |
| *** otter768 has joined #openstack-infra | 22:17 | |
| clarkb | sdake__: thats related to the current world is upside down issues we are having | 22:17 |
| sdake__ | and verified went to -2 | 22:17 |
| clarkb | sdake__: we have no f21 nodes | 22:17 |
| sdake__ | oh thats still a problem | 22:18 |
| clarkb | (apparently) | 22:18 |
| sdake__ | f21 was in the gate previosuly wasn't it? | 22:18 |
| clarkb | f21 has never been in the gate iirc | 22:18 |
| sdake__ | the gate was working for ages prior with f21 | 22:18 |
| clarkb | we have run jobs on f21 but not in the integrated gate iirc | 22:18 |
| zaro | clarkb: ohh yeah, it's looking good, just wanted to try with jjb. | 22:18 |
| clarkb | sdake__: anyways I will see what I can do about adding some nodes, thats what the issue is | 22:19 |
| *** freerunner has quit IRC | 22:19 | |
| clarkb | sdake__: right now gearman knows of no f21 hosts so says go away | 22:19 |
| clarkb | usually this isn't an issue because typicaly we have had at least one f21 host attached to gearman at some point | 22:20 |
| clarkb | its just right now with the geard restarts and nodepool losing track of nodes things are silly | 22:20 |
| sdake__ | oh so this problem will be resolved when gearman eventually restarts :) | 22:21 |
| clarkb | sdake__: no it will be resolved as soon as we build an f21 host and attach it to gearman | 22:21 |
| sdake__ | ok, so that probably sin't goin to happen immediatey, shoudl I just revert the gate patch temporarily? | 22:21 |
| *** gordc has quit IRC | 22:22 | |
| clarkb | sdake__: what gate patch? | 22:22 |
| sdake__ | the one where we added f21 gating to kolla | 22:22 |
| *** otter768 has quit IRC | 22:22 | |
| clarkb | is that new? | 22:22 |
| clarkb | (I totally missed that if that is the case) | 22:22 |
| sdake__ | 3 months ago | 22:22 |
| *** freerunner has joined #openstack-infra | 22:23 | |
| jeblair | actually should happen real soon | 22:23 |
| pleia2 | clarkb: aha, I think we need 'with grant option' at the end of the GRANT ALL line | 22:23 |
| *** stevemar has quit IRC | 22:24 | |
| pleia2 | py27: commands succeeded | 22:24 |
| pleia2 | congratulations :) | 22:24 |
| pleia2 | w00t | 22:24 |
| sdake__ | jeblair should be rectified today then? | 22:24 |
| pleia2 | I'll run through these again with a clean database to be sure | 22:25 |
| clarkb | pleia2: aha! | 22:25 |
| jeblair | sdake__: yes, probably within an hour, possibly much sooner | 22:25 |
| *** haypo has joined #openstack-infra | 22:25 | |
| haypo | hey. i'm investigating https://bugs.launchpad.net/swift/+bug/1451992 -- i just commented the issue | 22:26 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 22:26 |
| *** Krinkle|detached is now known as Krinkle | 22:26 | |
| haypo | it looks like the issue is that DevStack builds a local wheel package for xattr, whereas building a wheel package for cffi extensions doesn't work right now | 22:26 |
| clarkb | haypo: you should talk to clayg and greghaynes and jogo in openstack-dev | 22:26 |
| haypo | is anyone aware of a recent change in devstack or something else related to building wheel packages? | 22:26 |
| clarkb | haypo: they have mostly figured it out I think | 22:26 |
| *** anthonyper_ has joined #openstack-infra | 22:27 | |
| sdake__ | thanks jeblair | 22:27 |
| *** jcooley_ has joined #openstack-infra | 22:27 | |
| clarkb | jeblair: are you currently updating jobs with jjb? | 22:27 |
| *** baoli_ has quit IRC | 22:27 | |
| * clarkb is trying to keep up | 22:28 | |
| jeblair | clarkb: no | 22:28 |
| jeblair | clarkb: waiting for jjb change to merge so i don't have to install it manually on masters | 22:28 |
| *** jgrimm_ has joined #openstack-infra | 22:28 | |
| clarkb | jeblair: gotcha | 22:28 |
| jeblair | (or risk it reverting, etc) | 22:28 |
| *** doude_ has joined #openstack-infra | 22:29 | |
| *** sweston_ has joined #openstack-infra | 22:29 | |
| *** rbradfor has quit IRC | 22:29 | |
| *** neoice_ has joined #openstack-infra | 22:29 | |
| *** darvon_ has joined #openstack-infra | 22:29 | |
| *** anthonyper has quit IRC | 22:30 | |
| *** jcooley has quit IRC | 22:30 | |
| *** jgrimm has quit IRC | 22:30 | |
| *** kun_huang has quit IRC | 22:30 | |
| *** darvon has quit IRC | 22:30 | |
| *** neoice has quit IRC | 22:30 | |
| *** tteggel has quit IRC | 22:30 | |
| *** JayF has quit IRC | 22:30 | |
| *** rwsu_ has quit IRC | 22:30 | |
| *** edwarnicke has quit IRC | 22:30 | |
| *** doude has quit IRC | 22:30 | |
| *** sweston has quit IRC | 22:30 | |
| *** morgabra has quit IRC | 22:30 | |
| *** dmellado has quit IRC | 22:30 | |
| *** jcooley_ is now known as jcooley | 22:30 | |
| *** Rockyg has quit IRC | 22:30 | |
| *** anthonyper_ is now known as anthonyper | 22:30 | |
| *** morgabra has joined #openstack-infra | 22:30 | |
| *** darvon_ is now known as darvon | 22:30 | |
| *** kun_huang has joined #openstack-infra | 22:30 | |
| *** tteggel has joined #openstack-infra | 22:30 | |
| *** JayF has joined #openstack-infra | 22:30 | |
| *** rwsu_ has joined #openstack-infra | 22:30 | |
| *** dboik_ has joined #openstack-infra | 22:31 | |
| *** dmellado has joined #openstack-infra | 22:31 | |
| *** sweston_ is now known as sweston | 22:31 | |
| *** edwarnicke has joined #openstack-infra | 22:31 | |
| mordred | jeblair: wow. nodepool is up to 0.1.0 | 22:31 |
| *** unicell1 has joined #openstack-infra | 22:31 | |
| *** unicell has quit IRC | 22:31 | |
| shashankhegde | Hi | 22:32 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to convert append only tables to MyISAM https://review.openstack.org/179964 | 22:32 |
| sdake__ | jeblair can you ping me when you think I hsould try again | 22:32 |
| shashankhegde | I noticed that one of my patches was not merged even though gerrit says that it is | 22:32 |
| sdake__ | or say something gerneral in the channel i'll be watching :) | 22:33 |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to compute-hyperv https://review.openstack.org/180187 | 22:33 |
| shashankhegde | Any idea what could be going on? | 22:33 |
| shashankhegde | The patch in question is: https://review.openstack.org/#/c/177061/ | 22:33 |
| *** subscope_ has quit IRC | 22:34 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to networking-hyperv https://review.openstack.org/180160 | 22:34 |
| *** dboik has quit IRC | 22:35 | |
| *** erikmwilson_ is now known as erikmwilson | 22:35 | |
| *** dboik_ has quit IRC | 22:35 | |
| *** Krinkle is now known as Krinkle|detached | 22:35 | |
| *** Sukhdev has joined #openstack-infra | 22:36 | |
| *** ddieterly has quit IRC | 22:36 | |
| *** tkelsey has joined #openstack-infra | 22:39 | |
| *** craigbr has quit IRC | 22:41 | |
| zaro | clarkb: for some the rest api on my local instance of jenkins doesn't seem to work. i'm guessing it's because i've started with maven or something. | 22:42 |
| *** tkelsey has quit IRC | 22:43 | |
| zaro | clarkb: i can't seem to test with jjb right now and I gotta take off. can test more later tonight if you want to wait otherwise i have no problem if you want to get it in. | 22:43 |
| clarkb | zaro: ok thanks | 22:43 |
| zaro | clarkb: most of everthing i've look at looks good | 22:43 |
| clarkb | I can probably update my local testing to do some jjb pretty easily | 22:43 |
| *** salv-orlando has quit IRC | 22:43 | |
| *** craigbr has joined #openstack-infra | 22:44 | |
| clarkb | btu trying to make sure the other issues get addressed since they seem more pressing right now | 22:44 |
| zaro | yeah, i would just check gearman-plugin continues to work with changes from jjb. we had issues with that before. | 22:44 |
| zaro | yeah, i checked online/offline of slaves and looks good | 22:45 |
| *** craigbr has quit IRC | 22:45 | |
| zaro | did lots of stuff with re-registering of jobs and looks good too. | 22:45 |
| *** craigbr has joined #openstack-infra | 22:47 | |
| shashankhegde | So anyone has an idea why my patch was not actually merged even though jenkins says it has? | 22:49 |
| *** alex_xu has quit IRC | 22:49 | |
| clarkb | shashankhegde: can you expand on that? jenkins shouldn't claim mergedness | 22:50 |
| clarkb | shashankhegde: maybe point at the change? | 22:50 |
| shashankhegde | Hi clarkb , here is the review: https://review.openstack.org/#/c/177061/ | 22:50 |
| *** alex_xu has joined #openstack-infra | 22:51 | |
| shashankhegde | Its a stackforge repository | 22:51 |
| clarkb | shashankhegde: so gerrit says it has merged, can you point out whee it has not merged? | 22:51 |
| shashankhegde | The patch should have merged here: https://github.com/stackforge/networking-arista/blob/stable/kilo/requirements.txt | 22:52 |
| *** EmilienM|afk is now known as EmilienM | 22:52 | |
| tchaypo | I cloned the repo, and the tip looks like | 22:52 |
| tchaypo | * 6f62ae6 2015-04-24 Merge "Adding jsonrpclib to requirements.txt" (HEAD, refs/remotes/origin/master, refs/remotes/origin/HEAD, refs/heads/master) <Jenkins> | 22:52 |
| tchaypo | |\ | 22:52 |
| tchaypo | | * a8788ac 2015-04-23 Adding jsonrpclib to requirements.txt <Shashank Hegde> | 22:52 |
| tchaypo | it looks fairly merged to me | 22:52 |
| tchaypo | Github is not where the repo lives | 22:52 |
| tchaypo | github is a convenience mirror that gets updated sometimes | 22:52 |
| shashankhegde | hmmm... I actually branched from stable/kilo branch and submitted my review from there | 22:53 |
| shashankhegde | Any idea why it merged to master? | 22:53 |
| shashankhegde | tchaypo: I see | 22:53 |
| *** sarob has quit IRC | 22:53 | |
| tchaypo | If you look up the top of the Gerrit page, it has a “project” heading; if you click on that it will take you to https://review.openstack.org/#/admin/projects/stackforge/networking-arista which says.. | 22:53 |
| clarkb | shashankhegde: your chage was against master | 22:53 |
| tchaypo | actually it says to clone from review.openstack.org and I don’t think that’s right either | 22:54 |
| tchaypo | but it any case http://git.openstack.org/cgit/stackforge/networking-arista/log/ shows the merge as well | 22:54 |
| shashankhegde | clarkb: If I checkout "stable/kilo" and submit a review, shouldn't the change merge to the "stable/kilo" branch? | 22:55 |
| clarkb | shashankhegde: not necessarily, it depends on what branch you pushed the chagne against | 22:55 |
| clarkb | in this case the chagne was proposed against master as the gerrit change screen shows | 22:55 |
| jeblair | sdake__: there is an f21 node ready now | 22:55 |
| tchaypo | the parent commit is the one at the tip of the stable/kilo branch | 22:55 |
| tchaypo | but that commit is also part of the master branch | 22:56 |
| fungi | skimmed scrollback... i _think_ things look more under control | 22:56 |
| tchaypo | shashankhegde: FWIW, if you’d submitted to stable/kilo it should (I think) have been rejected anyway | 22:57 |
| *** emagana has quit IRC | 22:57 | |
| tchaypo | assuming that project follows the guidelines at https://wiki.openstack.org/wiki/StableBranch | 22:57 |
| shashankhegde | clarkb:, tchaypo, sorry, I am following http://docs.openstack.org/infra/manual/developers.html#committing-a-change and I can't seem to find how to propose change against a different branch | 22:57 |
| *** Swami has joined #openstack-infra | 22:57 | |
| *** ZZelle_ has quit IRC | 22:58 | |
| jeblair | tchaypo, clarkb, shashankhegde: i think we're missing info in infra-manual about submitting changes to alternate branches | 22:58 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Revert "Added parallelization options" https://review.openstack.org/180258 | 22:58 |
| *** emagana has joined #openstack-infra | 22:58 | |
| clarkb | jeblair: appears to e the case | 22:58 |
| jeblair | shashankhegde: the short version is "git review stable/kilo", sorry that's not in the manual | 22:58 |
| tchaypo | jeblair: we definitely don’t have anything in infra-manual, I was just hunting for it. But my understanding is that people should be submitting (except for cherry-picks) to stable branches anway? | 22:58 |
| shashankhegde | jeblair, thanks! | 22:59 |
| jeblair | tchaypo: i believe that's generally the case | 22:59 |
| tchaypo | so perhaps we need a mention of how to push to an alternate branch, with a note that most projects won’t accept new code on a branch named “stable/*”? | 22:59 |
| shashankhegde | jeblair, tchaypo, The change is basically to back port something in master to kilo | 22:59 |
| *** IAX has joined #openstack-infra | 22:59 | |
| shashankhegde | So how do I go about cherrypicking that change? | 23:00 |
| jeblair | tchaypo: yeah, we should document how the tool works but be careful not to overly document the workflow in infra-manual | 23:00 |
| tchaypo | or even better, a pointer to “If you’re submitting to a stable branch, please follow the process at https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes” | 23:00 |
| jeblair | yup | 23:00 |
| tchaypo | shashankhegde: can you take a look at https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes and let us know if it’s helpful? | 23:01 |
| *** IAX has quit IRC | 23:01 | |
| tchaypo | Getting someone unfamiliar with the process is usually a good way to get useful feedback :) | 23:01 |
| tchaypo | jeblair: thanks for putting the conversation back on track | 23:01 |
| jeblair | fungi: indeed. i found that geard had reverted to an old version on zuul.o.o; i fixed that manually and proposed a change to zuul to fix it more permanently. that has resolved the geard load issues. | 23:01 |
| *** bswartz has joined #openstack-infra | 23:02 | |
| jeblair | fungi: once the jjb change lands, i think we can continue our work of cleaning up the jobs on masters | 23:02 |
| *** annegent_ has quit IRC | 23:02 | |
| shashankhegde | tchaypo: Give me 5 mins , I'll let you know | 23:02 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 23:02 |
| fungi | interested to know which puppet change caused that | 23:02 |
| *** emagana has quit IRC | 23:02 | |
| jeblair | fungi: so there's no rush for clarkb's change, just we should probably get it into prod before the next 'change every job' change | 23:02 |
| clarkb | ya I can take a stab at rolling it onto jenkins07 as soon as everything else stabilizes | 23:03 |
| jeblair | (in the interim, we might leak a few ready nodes, but we'll be back at our background level for that, we can live with that for a little while longer) | 23:03 |
| jeblair | fungi: i have not been deleting old ready nodes though, sorry. | 23:04 |
| jeblair | oh looks like the jjb change has landed | 23:05 |
| ianw | so i'm not quite keeping up, sorry ... the NOT_REGISTERED issues for jobs are still happening? | 23:05 |
| clarkb | I deleted a few but not the entirety | 23:06 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: adding akanda horizon plugin to stackforge https://review.openstack.org/180294 | 23:06 |
| *** btully has quit IRC | 23:06 | |
| jeblair | ianw: should probably be settled out at this point for newly enqueued changes | 23:06 |
| jeblair | apparently that's not true for a change enqueued 6 mins ago though | 23:06 |
| ianw | ok so my 9:02 change just above got NOT_REGISTERED for centos7 ... let me try again | 23:06 |
| *** camunoz has joined #openstack-infra | 23:06 | |
| jeblair | ah, old ready nodes there | 23:07 |
| jeblair | i will delete them | 23:07 |
| jeblair | ianw: that should correct in 15-30 mins | 23:07 |
| shashankhegde | tchaypo, jeblair, clarkb: Thanks guys! it helped | 23:07 |
| fungi | i'll check for more stale nodes | 23:08 |
| ianw | jeblair: ok, thanks | 23:08 |
| *** wenlock_ has quit IRC | 23:09 | |
| jeblair | i just cleared out the bare-precise nodes too | 23:09 |
| *** emagana has joined #openstack-infra | 23:09 | |
| jeblair | (basically stuck-ready nodes + zuul restart means low-traffic nodes don't end up registering; it's an edge case for the problems we're seeing and should be resolved by the solutions in progress) | 23:09 |
| fungi | ianw: also puppet agent is probably still stopped on the jenkins masters. i think i need to circle back around and finish jjb runs on 3 of them | 23:10 |
| *** alex_xu has quit IRC | 23:10 | |
| fungi | so if that's a new job, it really may not be registered yet | 23:10 |
| ianw | fungi: no, it's just the regular job. sounds like the stuck nodes issue | 23:10 |
| *** alex_xu has joined #openstack-infra | 23:11 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for swift-object-server bug 1451992 https://review.openstack.org/180278 | 23:11 |
| openstack | bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/1451992 | 23:11 |
| tchaypo | shashankhegde: woo :) | 23:11 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config: puppet: run beaker on both trusty & centos7 https://review.openstack.org/175434 | 23:12 |
| *** jtriley has quit IRC | 23:12 | |
| jeblair | fungi: i think we have inconsistent versions of jjb installed on masters :/ | 23:13 |
| *** emagana has quit IRC | 23:14 | |
| timrc | jeblair, Is running multiple jenkins masters really as simple as bringing up a new Jenkins server, installing the gearman plugin, and pointing it at the gearman host? | 23:15 |
| timrc | gearman server host | 23:16 |
| clarkb | timrc: yes | 23:16 |
| jeblair | timrc: there's a tarball of configuration that we have not automated that we copy onto the new masters | 23:16 |
| clarkb | timrc: just keep in mind that when you do so things like periodic jobs need to go through zuultoo | 23:16 |
| jeblair | clarkb: i think that's still the case ^ ? | 23:16 |
| clarkb | jeblair: yup | 23:16 |
| fungi | jeblair: that also doesn't entirely surprise me | 23:16 |
| jeblair | timrc: but in spirit, yes :) | 23:16 |
| jeblair | fungi: i will ansible them into shape | 23:16 |
| fungi | thanks | 23:17 |
| fungi | since gear is reupgraded, no performance concerns with me kicking off full jjb updates on 02, 04 and 07 once jjb is upgraded everywhere? | 23:18 |
| *** panda|Zz has quit IRC | 23:18 | |
| jeblair | fungi: nope, i think we'll just expect some more stuck ready nodes | 23:18 |
| clarkb | fungi: I don't think so | 23:18 |
| *** panda|Zz has joined #openstack-infra | 23:19 | |
| jeblair | how do i ansible on jenkins, jenkins0* but not jenkins-dev? | 23:19 |
| mordred | jeblair: !jenkins-dev | 23:19 |
| jeblair | mordred: how do i combine that with the specifier for jenkins*? | 23:19 |
| clarkb | jeblair: its a regex if you prepend with ~ | 23:19 |
| mordred | jeblair: one sec | 23:19 |
| fungi | ~ totally says "regex" to me | 23:19 |
| clarkb | so ~jenkins[0-9]*.openstack.org I think | 23:20 |
| mordred | "!review.openstack.org:!git0*:!afs*" | 23:20 |
| clarkb | er ~jenkins[0-9]*.openstack\.org | 23:20 |
| jeblair | oh : | 23:20 |
| clarkb | first . needs \ too :/ | 23:20 |
| jeblair | okay, i regexed it this time | 23:20 |
| mordred | so, "jenkins.openstack.org:jenkins01*:!jenkins-dev*" | 23:20 |
| mordred | col | 23:20 |
| mordred | cool | 23:21 |
| jeblair | but good to know both things | 23:21 |
| fungi | i already knew half a dozen distinct regex grammars. what's one more? | 23:21 |
| jeblair | ~ instead of ^. i tried "," instead of ":". :) | 23:21 |
| *** shashankhegde has quit IRC | 23:21 | |
| jeblair | so apparently, the ansible authors and i don't quite see eye to eye on hostnames. | 23:21 |
| *** heyongli has quit IRC | 23:21 | |
| * fungi needs to remember to make a new kind of regex for whatever his next project is. just to keep people on their toes | 23:21 | |
| * jeblair mumbles something about ipv6 and colons | 23:21 | |
| fungi | yeah, um, don't regex ipv6 addys in ansible i guess | 23:22 |
| *** heyongli has joined #openstack-infra | 23:22 | |
| mordred | I don't think they expect the ipv6 address to be the hostname | 23:22 |
| mordred | or the name of the group in which the host is defined to be | 23:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 23:22 |
| mordred | jeblair: in theory, we should perhaps have a "jenkins-prod" group | 23:22 |
| fungi | best news of the day... zuul seems to still be catching up. nearly down to 500 jobs waiting now | 23:23 |
| *** abregman has quit IRC | 23:23 | |
| mordred | \o/ | 23:23 |
| jeblair | jenkins-job-builder==1.2.0.47.g5be7b76 # git sha 5be7b76 | 23:23 |
| jeblair | jenkins-job-builder==1.2.0.post47 # git sha 5be7b76 | 23:23 |
| jeblair | jenkins-job-builder==1.2.1.dev40 # git sha 5be7b76 | 23:24 |
| mordred | jeblair: taht's spectacular | 23:24 |
| jeblair | from 3 different hosts :( | 23:24 |
| clarkb | thats different pbrs | 23:24 |
| mordred | jeblair: I believe we want to upgrade pbr to the latest everywhere | 23:24 |
| *** signed8b_ is now known as signed8bit_ZZZzz | 23:24 | |
| clarkb | mordred: yup | 23:24 |
| *** Krinkle|detached is now known as Krinkle | 23:25 | |
| jeblair | 'pip install -U pbr' does not accomplish that; do i need to give it a min? | 23:25 |
| timrc | clarkb, Periodic jobs going through zuul as opposed to running directly on the Jenkins server? Just want to be clear on that point. AFAIK all of our periodic jobs are configured through zuul. | 23:26 |
| *** jtriley has joined #openstack-infra | 23:26 | |
| jeblair | clarkb, mordred: oh, pbr is latest everywhere | 23:26 |
| jeblair | clarkb, mordred: maybe it wasn't latest when jjb was installed though? | 23:27 |
| mordred | jeblair: yah. that would be it | 23:27 |
| clarkb | timrc: yes, because otherwise you get mulitple copiues of the jobs running across your masters | 23:27 |
| jeblair | anyway, i think it's not important since the git shas are all the same | 23:27 |
| clarkb | timrc: using zuul gets you a single set | 23:27 |
| jeblair | so let's call it done | 23:27 |
| jeblair | fungi: i think you're clear to proceed with jjb runs | 23:27 |
| Krinkle | jeblair: FYI the failure on https://review.openstack.org/#/c/172241/ seems a false positive since the file I modified afaik doesn't influence tests of any kind. | 23:28 |
| *** Krinkle is now known as Krinkle|detached | 23:29 | |
| fungi | jeblair: thanks, firing up | 23:29 |
| timrc | clarkb, Cool. Thanks. | 23:29 |
| fungi | pkg_resources.DistributionNotFound: ordereddict | 23:30 |
| fungi | once again pip fails us | 23:30 |
| *** shashankhegde has joined #openstack-infra | 23:30 | |
| fungi | seems to be running okay on 02 and 04 but the jjb install is broken on 07 | 23:31 |
| fungi | rerunning `sudo pip install -U .` in /opt/jenkins_job_builder seems to have fixed it | 23:32 |
| jeblair | Krinkle|detached: quite likely, thanks | 23:33 |
| *** ashleighfarnham has quit IRC | 23:33 | |
| fungi | once these jjb runs complete, i'll reenable puppet agent on the jenkins masters | 23:35 |
| *** wenlock has quit IRC | 23:35 | |
| *** abregman has joined #openstack-infra | 23:38 | |
| *** Kennan2 is now known as Kennan | 23:38 | |
| *** craigbr has quit IRC | 23:39 | |
| Kennan | hi :fungi | 23:39 |
| *** apmelton is now known as apmelton_away | 23:40 | |
| *** IAX has joined #openstack-infra | 23:40 | |
| Kennan | just short question, if we have a branched named kilo-stable , is it the devstack gatejob knows it would run against kilo branch ? | 23:40 |
| *** abregman has quit IRC | 23:40 | |
| Kennan | we(our project) | 23:41 |
| *** _nadya_ has joined #openstack-infra | 23:41 | |
| *** ajmiller has quit IRC | 23:41 | |
| IAX | hi guys, just a quick question, do you have configured a bash shell lint job in jenkins? if so, what do you use? | 23:42 |
| *** sdake__ has quit IRC | 23:42 | |
| clarkb | IAX: we use bashate | 23:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 23:43 |
| fungi | Kennan: devstack-gate will try to test a change against branches of the same name in other projects | 23:44 |
| *** salv-orlando has joined #openstack-infra | 23:44 | |
| fungi | Kennan: so if your branch is "kilo-stable" then devstack-gate isn't going to magically figure out that it should test that with branches named "stable/kilo" in other projects, no | 23:45 |
| fungi | Kennan: you should probably have a "stable/kilo" | 23:45 |
| fungi | like other projects do | 23:45 |
| Kennan | :fungi ok Thanks | 23:45 |
| *** _nadya_ has quit IRC | 23:46 | |
| fungi | Kennan: you're welcome | 23:46 |
| IAX | awesome, thank you clarkb! | 23:46 |
| *** annegentle has joined #openstack-infra | 23:47 | |
| pleia2 | clarkb: any thoughts on my traceback re: https://review.openstack.org/#/c/179499/ ? | 23:47 |
| *** annegentle has quit IRC | 23:47 | |
| pleia2 | (still on these docs testing, need this dependencies) | 23:48 |
| pleia2 | er, dependency | 23:48 |
| Kennan | :fungi | 23:50 |
| Kennan | it seems different project can have differnt names | 23:50 |
| Kennan | it need self config | 23:50 |
| Kennan | check this https://review.openstack.org/#/c/176907/ | 23:50 |
| Kennan | you reviewed this before | 23:51 |
| Kennan | :) | 23:51 |
| *** salv-orlando has quit IRC | 23:51 | |
| *** rfolco has quit IRC | 23:52 | |
| fungi | Kennan: yes, you have to manually define a mapping. like in that configuration you're linking. devstack-gate won't simply figure it out unless it matches (which is preferable, as that's automatic) | 23:52 |
| cinerama | pleia2: zanata folks marked the percentage bug with high severity, cool | 23:52 |
| *** IAX has quit IRC | 23:53 | |
| pleia2 | cinerama: yay! | 23:53 |
| *** dangers is now known as dangers_away | 23:56 | |
| clarkb | pleia2: only that maybe your nodepool code is broken? | 23:59 |
| clarkb | pleia2: that is odd | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!