| *** reed has joined #openstack-infra | 00:04 | |
| *** alkari has quit IRC | 00:06 | |
| *** zehicle_at_dell has joined #openstack-infra | 00:07 | |
| *** rmcall has quit IRC | 00:08 | |
| *** lcheng_ has joined #openstack-infra | 00:08 | |
| *** gokrokve has quit IRC | 00:11 | |
| *** dmellado has quit IRC | 00:11 | |
| *** armax has quit IRC | 00:13 | |
| *** dmellado has joined #openstack-infra | 00:13 | |
| arosen | Do you guys ever see images that nodepools creates just hanging on grub same image not found? | 00:13 |
|---|---|---|
| * arosen Booting from Hard Disk... error: file not found. | 00:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 00:14 |
| clarkb | arosen: no, I don't think we have had grub errors. we do update grub to remove the 5 second boot timeout thing that it does by default on all our images | 00:16 |
| clarkb | arosen: you can probably remove that to work around it. Otherwise your base image may need whatever grub uses to rebuild itself and set up the ramdisk thingy? | 00:16 |
| clarkb | also if anyone that builds cloud images is following along, you should probably have grub not wait 5 seconds. it doesn't help much when you have remote console latency | 00:17 |
| arosen | clarkb: hrm i think it might be an infra issue we are having in our cloud | 00:18 |
| arosen | any idea about this? http://paste.openstack.org/show/109867/ | 00:18 |
| *** gyee has quit IRC | 00:18 | |
| *** tomoe has joined #openstack-infra | 00:19 | |
| clarkb | no, but that looks like the same issue you had before. Are you running latest nodepool? | 00:19 |
| arosen | i am | 00:19 |
| arosen | last commit: i have is https://review.openstack.org/#/c/119256/ | 00:20 |
| openstackgerrit | A change was merged to openstack/requirements: add header warning against sorting requirements https://review.openstack.org/119658 | 00:23 |
| clarkb | arosen: I think that is a bug. one sec and I will push a potential fix | 00:23 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/120628 | 00:24 |
| *** harlowja_ has quit IRC | 00:25 | |
| *** cdent has quit IRC | 00:25 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Delete correct item in image delete command. https://review.openstack.org/120639 | 00:27 |
| clarkb | arosen: ^ completely untested | 00:27 |
| clarkb | arosen: but worth a shot | 00:27 |
| *** harlowja has joined #openstack-infra | 00:28 | |
| *** armax has joined #openstack-infra | 00:28 | |
| arosen | clarkb: grr no now i'm in this bad state again. I cherry-picked your patch and ran: sudo python setup.py develop; ' | 00:29 |
| arosen | http://paste.openstack.org/show/109875/ | 00:29 |
| *** yamamoto has joined #openstack-infra | 00:29 | |
| clarkb | try pip install -e? | 00:29 |
| arosen | http://paste.openstack.org/show/109876/ | 00:30 |
| clarkb | also if you develop or pip install -e you shouldn't have ot reinstall after updating the code | 00:30 |
| *** weshay has quit IRC | 00:30 | |
| arosen | http://paste.openstack.org/show/109877/ | 00:30 |
| arosen | i used asselin's installer for this which does python setup.py develop i believe. | 00:30 |
| clarkb | mordred: ^ I have no idea what is going on there | 00:31 |
| clarkb | mordred: also can we have setup.py figure out if pip called it and fail otherwise? | 00:31 |
| *** julim has quit IRC | 00:32 | |
| *** yamamoto has quit IRC | 00:34 | |
| *** hdd has joined #openstack-infra | 00:34 | |
| clarkb | jogo: looking at er, 1254890 and 1357476 appear to be the same bug | 00:34 |
| clarkb | jogo: the graphs are almost identical | 00:34 |
| *** julim has joined #openstack-infra | 00:35 | |
| asselin | arosen, clarkb I've seen this issue too, but managed to clean up my nodepool env and got past it....http://paste.openstack.org/show/109867/ | 00:35 |
| clarkb | asselin: you should not setup.py develop | 00:36 |
| arosen | sorry guys gotta run for the train bbl | 00:36 |
| asselin | clarkb, no I don't think I am. | 00:36 |
| *** julim has quit IRC | 00:36 | |
| asselin | clarkb, arosen I setup with nodepool puppet: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/manifests/master.pp#L345 | 00:37 |
| clarkb | jogo: I am going to delete H803 with fire and acid and all the things | 00:38 |
| asselin | I manually start though (in screen): sudo su - nodepool; nodepoold -d | 00:38 |
| *** daya_k has quit IRC | 00:39 | |
| *** armax has quit IRC | 00:45 | |
| clarkb | ideas on how to deal with my comment on https://review.openstack.org/#/c/120228/ are much appreciated | 00:48 |
| *** yaguang has joined #openstack-infra | 00:49 | |
| clarkb | may have to just switch on pthon version and do level checking based on that | 00:49 |
| asselin | is this new? Parameter name failed on Database_user[@]: Invalid database user @ at /etc/puppet/modules/mysql/manifests/server/account_security.pp:8 | 00:49 |
| clarkb | Alex_Gaynor: ^ thoughts? | 00:49 |
| *** armax has joined #openstack-infra | 00:49 | |
| fungi | oh, ew | 00:49 |
| *** dimtruck is now known as zz_dimtruck | 00:49 | |
| clarkb | asselin: no I think that means you don't have a working $fqdn fact | 00:49 |
| clarkb | asselin: you can confirm with `facter fqdn` | 00:49 |
| clarkb | fungi: ya... | 00:49 |
| fungi | clarkb: well, it's said that supporting 2.6 and 3.x is fruitless | 00:50 |
| clarkb | I mean we can drop 2.6 in kilo right? | 00:50 |
| asselin | clarkb, no output.... | 00:50 |
| clarkb | but 3.3 works today... thats what makes this frustrating | 00:50 |
| clarkb | asselin: yup that confirms it | 00:50 |
| *** jpich has quit IRC | 00:51 | |
| *** bdpayne has quit IRC | 00:51 | |
| JayF | clarkb: a visual check says alex isn't here anymore (we work in the same office) | 00:51 |
| clarkb | JayF: thanks | 00:51 |
| fungi | clearly the spread from 2.6 to 3.4 is untenable | 00:51 |
| lifeless | fungi: for python? | 00:51 |
| *** marcoemorais has quit IRC | 00:51 | |
| lifeless | fungi: how so? | 00:51 |
| fungi | JayF: when he is, i've also got a pypy bug to consult with him on | 00:51 |
| clarkb | lifeless: Alex_Gaynor wrote a change that broke backward compatiblity of some public interfaces in python logging module | 00:51 |
| lifeless | clarkb: in 2.7? | 00:51 |
| *** yamahata has joined #openstack-infra | 00:51 | |
| fungi | lifeless: in 3.4 | 00:52 |
| lifeless | ah | 00:52 |
| clarkb | lifeless: you can now no longer map from string to logging level in 3.4 with public interfaces | 00:52 |
| clarkb | you can only map from level to string | 00:52 |
| lifeless | I thought guido had said that there were to be no such breaks in 3.3+ ? | 00:52 |
| clarkb | lifeless: well the change was a "refactor" I think it was just a mistake | 00:52 |
| clarkb | but at this point we may have to live with it | 00:52 |
| lifeless | so we fix and do a point release of 3.4 | 00:52 |
| clarkb | lifeless: http://hg.python.org/cpython/rev/5629bf4c6bba?revcount=60 is the change | 00:53 |
| clarkb | lifeless: notice that the dict getLevelName uses only maps one direction now | 00:53 |
| fungi | i think it's being argued that the change is purely cosmetic, but it definitely changes the return type of the method completely | 00:53 |
| clarkb | lifeless: do you think we should file a bug? not sure what upstream python bug reporting process is | 00:53 |
| lifeless | clarkb: bugs.python.org | 00:55 |
| fungi | clearly the values were inverted in the tests for logging too, which should have been a tip off | 00:55 |
| lifeless | what does Alex_Gaynor think ? | 00:55 |
| clarkb | lifeless: yesterday he felt we were stuck with it | 00:55 |
| *** lcheng_ has quit IRC | 00:55 | |
| clarkb | but filing a bug to get greater python consensus won't hurt /me does this | 00:55 |
| Alex_Gaynor | (I'm on an airplane so not super responsive), the docs only describe the behavior when you pass a string, AFAICT the string -> int usage was purely emergent and unintended | 00:56 |
| Alex_Gaynor | err, when you pass an *int* | 00:56 |
| fungi | though present from at least 2.6 through 3.3 | 00:56 |
| Alex_Gaynor | Yeah, filing a bug can't hurt | 00:57 |
| fungi | at the moment it's proving challenging to test its behavior with one piece of code on all of 2.6, 2.7 and 3.4 as a result | 00:57 |
| fungi | clarkb: instead of a single assertEqual could we cascade a conditional fallback to catch the earlier behavior? | 00:59 |
| clarkb | I think the biggest problem is there is now no public interface to get that mapping | 00:59 |
| clarkb | fungi: yeah so we could have wrapper.py check in both ways | 00:59 |
| clarkb | fungi: and since we get a ValueError each time the test doesn't really need changing | 00:59 |
| *** SumitNaiksatam has quit IRC | 00:59 | |
| lifeless | Alex_Gaynor: so I'm sure it was unintended; I think there's a strong argument that we need to suck it up and deprecate gently, since I doubt openstack is the only culprit here | 01:00 |
| clarkb | is a pypi account useable on bugs.python.org? | 01:00 |
| * clarkb uses lp openid | 01:00 | |
| Alex_Gaynor | Sure, I think the fact that there's no other API for this makes me think they might be more receptive to adding that behavior back | 01:00 |
| Alex_Gaynor | clarkb: no, seperate account systems :-( | 01:00 |
| fungi | Alex_Gaynor: while you happen to be semi-around, any feeling on whether https://bitbucket.org/pypy/pypy/issue/1694/ might be present in the pypy 2.2.1 on ubuntu trusty? looks like zaqar's unit tests are hitting similar behavior | 01:02 |
| fungi | trying to get a feel for how likely it would be taht we'd be able to convince the package maintainer to sru a fix for that | 01:02 |
| clarkb | would type be Library (Lib)? | 01:03 |
| clarkb | sorry component? | 01:03 |
| Alex_Gaynor | clarkb: yeah | 01:03 |
| Alex_Gaynor | fungi: seems likely, and I think stefano would be receptive, assuming ubuntu polict allows it | 01:03 |
| fungi | Alex_Gaynor: okay, i'll try to confirm it really is that bug and open a bug on lp for ubuntu's package | 01:04 |
| fungi | thanks! | 01:04 |
| *** jpich has joined #openstack-infra | 01:07 | |
| *** gokrokve has joined #openstack-infra | 01:09 | |
| *** baoli has quit IRC | 01:10 | |
| clarkb | Alex_Gaynor: lifeless http://bugs.python.org/issue22386 | 01:10 |
| *** baoli has joined #openstack-infra | 01:10 | |
| clarkb | and with that /me calls it a night | 01:12 |
| *** annegentle has joined #openstack-infra | 01:15 | |
| *** dane_leblanc has joined #openstack-infra | 01:15 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:16 | |
| *** daneleblanc has joined #openstack-infra | 01:18 | |
| *** Daisy has joined #openstack-infra | 01:19 | |
| *** dane_leblanc has quit IRC | 01:21 | |
| *** zz_dimtruck is now known as dimtruck | 01:24 | |
| *** annegentle has quit IRC | 01:26 | |
| *** yamamoto_ has joined #openstack-infra | 01:29 | |
| *** annegentle has joined #openstack-infra | 01:29 | |
| *** Ryan_Lane has quit IRC | 01:30 | |
| *** sandywalsh has quit IRC | 01:31 | |
| *** dims_ has joined #openstack-infra | 01:32 | |
| *** nelsnels_ has quit IRC | 01:32 | |
| *** Hefeweizen has quit IRC | 01:33 | |
| *** amotoki has joined #openstack-infra | 01:34 | |
| *** yamamoto_ has quit IRC | 01:34 | |
| *** oomichi has joined #openstack-infra | 01:37 | |
| *** TravT has quit IRC | 01:39 | |
| *** Sukhdev has joined #openstack-infra | 01:40 | |
| *** otter768 has joined #openstack-infra | 01:41 | |
| *** rainya has quit IRC | 01:42 | |
| *** Sukhdev has quit IRC | 01:42 | |
| *** Sukhdev has joined #openstack-infra | 01:43 | |
| *** rainya has joined #openstack-infra | 01:43 | |
| alexpilotti | jeblair: hi | 01:44 |
| *** annegentle has quit IRC | 01:45 | |
| *** nosnos has joined #openstack-infra | 01:47 | |
| *** Sukhdev has quit IRC | 01:47 | |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Create repo for keystoneclient-federation https://review.openstack.org/120670 | 01:55 |
| *** jaypipes has quit IRC | 01:58 | |
| *** alexpilotti has quit IRC | 02:01 | |
| *** markmcclain has quit IRC | 02:05 | |
| *** jpich has quit IRC | 02:06 | |
| *** yamamoto_ has joined #openstack-infra | 02:08 | |
| *** jyuso has joined #openstack-infra | 02:08 | |
| *** ramashri has quit IRC | 02:10 | |
| *** amotoki_ has joined #openstack-infra | 02:10 | |
| *** amotoki has quit IRC | 02:11 | |
| *** amotoki_ is now known as amotoki | 02:11 | |
| *** wenlock has joined #openstack-infra | 02:13 | |
| *** zehicle_at_dell has quit IRC | 02:15 | |
| *** annegentle has joined #openstack-infra | 02:15 | |
| *** dkranz has joined #openstack-infra | 02:16 | |
| *** yamamoto_ has quit IRC | 02:18 | |
| *** jpich has joined #openstack-infra | 02:19 | |
| *** annegentle has quit IRC | 02:21 | |
| *** amcrn has quit IRC | 02:22 | |
| *** gokrokve has quit IRC | 02:25 | |
| *** patrickeast has quit IRC | 02:26 | |
| *** markmcclain has joined #openstack-infra | 02:26 | |
| *** rainya has quit IRC | 02:31 | |
| *** annegentle has joined #openstack-infra | 02:32 | |
| *** rainya has joined #openstack-infra | 02:36 | |
| *** armax has quit IRC | 02:37 | |
| *** JadnG has quit IRC | 02:38 | |
| *** daneleblanc_ has joined #openstack-infra | 02:42 | |
| *** daneleblanc has quit IRC | 02:42 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Tricircle project https://review.openstack.org/117939 | 02:44 |
| *** sileht has quit IRC | 02:46 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Bump up taskflow to its newest version https://review.openstack.org/119985 | 02:47 |
| *** praneshp has quit IRC | 02:48 | |
| *** koolhead17 has joined #openstack-infra | 02:48 | |
| *** daneleblanc_ has quit IRC | 02:48 | |
| *** dims_ has quit IRC | 02:49 | |
| *** dims_ has joined #openstack-infra | 02:49 | |
| *** dmsimard_away is now known as dmsimard | 02:49 | |
| *** lcheng_ has joined #openstack-infra | 02:52 | |
| *** dmsimard is now known as dmsimard_away | 02:52 | |
| *** rushiagr_away is now known as rushiagr | 02:53 | |
| *** chenglch has joined #openstack-infra | 02:54 | |
| *** dims_ has quit IRC | 02:54 | |
| *** yamamoto_ has joined #openstack-infra | 02:54 | |
| *** arnaud has quit IRC | 03:01 | |
| *** KanagarajM has joined #openstack-infra | 03:03 | |
| *** adalbas has quit IRC | 03:05 | |
| *** daya_k has joined #openstack-infra | 03:08 | |
| *** rushiagr is now known as rushiagr_away | 03:14 | |
| *** hdd has quit IRC | 03:18 | |
| *** annegentle has quit IRC | 03:21 | |
| SpamapS | fungi: hey, at one point you were telling me about a standard way for ssh host keys to be authenticated that didn't involve previously exchanged fingerprints. Or am I crazy? | 03:22 |
| clarkb | SpamapS: you can use a cert authority | 03:22 |
| clarkb | and have signed key data | 03:22 |
| SpamapS | So thinking in an automation sense.. if I had a CA that had an API, and I posted my public key to that API with some pre-arranged nonce.. it could spit me back a signed key, that clients who trust that CA, would trust? | 03:23 |
| clarkb | ya. also sshfp records sort of solve this problem too | 03:24 |
| *** bhuvan has joined #openstack-infra | 03:25 | |
| clarkb | sshfp may be a better option if yuo can convince dns to do it right | 03:26 |
| SpamapS | clarkb: seems like something Designate might be suited for | 03:26 |
| clarkb | that would actually be a neat feature. have designate and noav automagically create those records for you | 03:27 |
| SpamapS | clarkb: it would be an amazingly great feature | 03:28 |
| SpamapS | No more blind accept | 03:28 |
| *** rfolco has joined #openstack-infra | 03:29 | |
| *** otter768 has quit IRC | 03:29 | |
| *** Longgeek has joined #openstack-infra | 03:29 | |
| *** Longgeek_ has joined #openstack-infra | 03:32 | |
| *** nibalizer has quit IRC | 03:32 | |
| *** Longgeek has quit IRC | 03:36 | |
| *** yaguang has quit IRC | 03:36 | |
| *** nibalizer has joined #openstack-infra | 03:37 | |
| *** SumitNaiksatam has quit IRC | 03:39 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Bump up taskflow to its newest version https://review.openstack.org/119985 | 03:40 |
| *** SumitNaiksatam has joined #openstack-infra | 03:40 | |
| *** SumitNaiksatam has quit IRC | 03:42 | |
| *** dimtruck is now known as zz_dimtruck | 03:42 | |
| *** Ryan_Lane has joined #openstack-infra | 03:43 | |
| *** krotscheck has quit IRC | 03:44 | |
| *** [HeOS] has joined #openstack-infra | 03:44 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:44 | |
| *** HeOS has quit IRC | 03:46 | |
| *** rushiagr_away is now known as rushiagr | 03:46 | |
| *** krotscheck has joined #openstack-infra | 03:47 | |
| *** hdd has joined #openstack-infra | 03:48 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:53 | |
| *** praneshp has joined #openstack-infra | 03:54 | |
| *** praneshp_ has joined #openstack-infra | 03:57 | |
| *** mmaglana has quit IRC | 03:57 | |
| *** jpich has quit IRC | 03:58 | |
| *** praneshp has quit IRC | 03:58 | |
| *** praneshp_ is now known as praneshp | 03:58 | |
| *** ramashri has joined #openstack-infra | 04:01 | |
| *** zz_dimtruck is now known as dimtruck | 04:02 | |
| *** Daisy has quit IRC | 04:02 | |
| *** harlowja is now known as harlowja_away | 04:05 | |
| *** ramashri has quit IRC | 04:08 | |
| *** ramashri has joined #openstack-infra | 04:09 | |
| *** jpich has joined #openstack-infra | 04:10 | |
| *** krtaylor has joined #openstack-infra | 04:10 | |
| *** lcheng__ has joined #openstack-infra | 04:11 | |
| *** annegentle has joined #openstack-infra | 04:11 | |
| *** dimtruck is now known as zz_dimtruck | 04:11 | |
| *** baoli has quit IRC | 04:12 | |
| *** lcheng_ has quit IRC | 04:13 | |
| *** Ryan_Lane has quit IRC | 04:22 | |
| *** rainya has quit IRC | 04:29 | |
| *** koolhead_ has joined #openstack-infra | 04:29 | |
| *** gargola has quit IRC | 04:30 | |
| *** MarkAtwood has joined #openstack-infra | 04:30 | |
| *** IAX has quit IRC | 04:30 | |
| *** iax7 has quit IRC | 04:30 | |
| *** gargola has joined #openstack-infra | 04:31 | |
| *** IAX has joined #openstack-infra | 04:31 | |
| *** IAX7 has joined #openstack-infra | 04:31 | |
| *** koolhead17 has quit IRC | 04:31 | |
| *** markmcclain has quit IRC | 04:34 | |
| *** mbacchi has quit IRC | 04:36 | |
| *** gokrokve has joined #openstack-infra | 04:37 | |
| *** harlowja_at_home has joined #openstack-infra | 04:39 | |
| *** HenryG is now known as HenryG_zzzz | 04:40 | |
| *** MarkAtwood has quit IRC | 04:42 | |
| *** Sukhdev_ has joined #openstack-infra | 04:43 | |
| *** yamahata has quit IRC | 04:46 | |
| *** annegentle has quit IRC | 04:47 | |
| *** yamahata has joined #openstack-infra | 04:47 | |
| *** ppai has joined #openstack-infra | 04:49 | |
| *** jaypipes has joined #openstack-infra | 04:54 | |
| *** koolhead_ has quit IRC | 04:57 | |
| *** mrmartin has joined #openstack-infra | 05:04 | |
| *** stevemar has quit IRC | 05:05 | |
| *** Daisy has joined #openstack-infra | 05:05 | |
| *** torandu has quit IRC | 05:05 | |
| *** pcrews has quit IRC | 05:07 | |
| *** sabeen has quit IRC | 05:09 | |
| openstackgerrit | David Pursehouse proposed a change to stackforge/gertty: Use owner's username or email if display name is not set https://review.openstack.org/120684 | 05:10 |
| *** gargola has quit IRC | 05:12 | |
| *** IAX has quit IRC | 05:12 | |
| *** IAX7 has quit IRC | 05:12 | |
| *** nelsnelson has joined #openstack-infra | 05:12 | |
| *** IAX has joined #openstack-infra | 05:12 | |
| *** gargola has joined #openstack-infra | 05:12 | |
| *** IAX7 has joined #openstack-infra | 05:12 | |
| *** gargola has quit IRC | 05:17 | |
| *** IAX has quit IRC | 05:17 | |
| *** IAX7 has quit IRC | 05:17 | |
| *** IAX has joined #openstack-infra | 05:17 | |
| *** iax7 has joined #openstack-infra | 05:17 | |
| *** gargola has joined #openstack-infra | 05:18 | |
| *** mrmartin has quit IRC | 05:20 | |
| *** lcheng__ has quit IRC | 05:21 | |
| *** reed has quit IRC | 05:22 | |
| *** chuckC has joined #openstack-infra | 05:27 | |
| *** koolhead17 has joined #openstack-infra | 05:30 | |
| *** koolhead17 has joined #openstack-infra | 05:30 | |
| *** hdd has quit IRC | 05:36 | |
| *** alkari has joined #openstack-infra | 05:37 | |
| *** bswartz has quit IRC | 05:42 | |
| *** yfried has joined #openstack-infra | 05:44 | |
| *** amcrn has joined #openstack-infra | 05:44 | |
| *** nelsnelson has quit IRC | 05:47 | |
| *** bswartz has joined #openstack-infra | 05:49 | |
| *** SumitNaiksatam has quit IRC | 05:52 | |
| *** rkukura has quit IRC | 05:54 | |
| *** praneshp has quit IRC | 05:54 | |
| *** rkukura_ has joined #openstack-infra | 05:54 | |
| *** flaper87|afk is now known as flaper87 | 05:58 | |
| *** Ryan_Lane has joined #openstack-infra | 05:58 | |
| *** jcoufal has joined #openstack-infra | 05:59 | |
| *** vigneshvar has joined #openstack-infra | 05:59 | |
| *** vigneshvar_ has joined #openstack-infra | 06:01 | |
| *** vigneshvar_ has quit IRC | 06:03 | |
| *** oomichi_ has joined #openstack-infra | 06:03 | |
| *** oomichi has quit IRC | 06:05 | |
| *** zz_dimtruck is now known as dimtruck | 06:10 | |
| *** mpaolino has joined #openstack-infra | 06:12 | |
| *** andreaf has joined #openstack-infra | 06:13 | |
| *** Longgeek_ has quit IRC | 06:17 | |
| *** Longgeek has joined #openstack-infra | 06:18 | |
| *** dimtruck is now known as zz_dimtruck | 06:20 | |
| *** Longgeek has quit IRC | 06:20 | |
| *** che-arne has quit IRC | 06:20 | |
| *** Sukhdev_ has quit IRC | 06:23 | |
| *** _nadya_ has joined #openstack-infra | 06:27 | |
| *** dkranz has quit IRC | 06:28 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/config: Add a job for functional tests in zaqarclient https://review.openstack.org/120437 | 06:28 |
| *** gokrokve has quit IRC | 06:28 | |
| *** dkranz has joined #openstack-infra | 06:28 | |
| *** gokrokve has joined #openstack-infra | 06:28 | |
| *** wenlock has quit IRC | 06:30 | |
| *** jamespage_ has joined #openstack-infra | 06:32 | |
| *** gokrokve has quit IRC | 06:33 | |
| *** dkranz has quit IRC | 06:33 | |
| *** gokrokve has joined #openstack-infra | 06:34 | |
| *** dkranz has joined #openstack-infra | 06:34 | |
| *** k4n0 has joined #openstack-infra | 06:34 | |
| *** afazekas is now known as __afazekas | 06:34 | |
| *** jamespage_ has quit IRC | 06:35 | |
| *** loki184 has joined #openstack-infra | 06:36 | |
| *** gokrokve has quit IRC | 06:38 | |
| *** ramashri has quit IRC | 06:40 | |
| *** ramashri has joined #openstack-infra | 06:40 | |
| *** ashaeron has joined #openstack-infra | 06:41 | |
| *** ccit has joined #openstack-infra | 06:41 | |
| *** rhsu has quit IRC | 06:49 | |
| *** _nadya_ has quit IRC | 06:50 | |
| *** rfolco has quit IRC | 06:51 | |
| *** Longgeek has joined #openstack-infra | 06:52 | |
| *** harlowja_at_home has quit IRC | 06:56 | |
| *** mrodden has quit IRC | 06:56 | |
| *** [HeOS] has quit IRC | 06:56 | |
| *** ramashri has quit IRC | 06:57 | |
| *** mrodden has joined #openstack-infra | 06:58 | |
| *** yamamoto_ has quit IRC | 06:59 | |
| *** jlibosva has joined #openstack-infra | 06:59 | |
| *** Daisy has quit IRC | 07:00 | |
| *** ramashri has joined #openstack-infra | 07:02 | |
| *** AJaeger has quit IRC | 07:02 | |
| *** _nadya_ has joined #openstack-infra | 07:03 | |
| *** sileht has joined #openstack-infra | 07:03 | |
| *** gokrokve has joined #openstack-infra | 07:04 | |
| *** michchap_ has joined #openstack-infra | 07:05 | |
| *** gokrokve has quit IRC | 07:05 | |
| *** gokrokve has joined #openstack-infra | 07:06 | |
| *** michchap has quit IRC | 07:08 | |
| *** arxcruz has joined #openstack-infra | 07:08 | |
| *** ildikov has joined #openstack-infra | 07:09 | |
| *** ccit has quit IRC | 07:09 | |
| mika | jeblair: http://paste.grml.org/hidden/1716b56d is the full output of the curl command | 07:10 |
| *** gokrokve has quit IRC | 07:10 | |
| *** camunoz has quit IRC | 07:11 | |
| *** dtantsur|afk is now known as dtantsur | 07:12 | |
| *** todd_dsm has joined #openstack-infra | 07:12 | |
| *** Daisy has joined #openstack-infra | 07:14 | |
| *** omr has joined #openstack-infra | 07:14 | |
| omr | What is the best solution for aborted job in Jenkins? | 07:14 |
| *** Guest1245 has joined #openstack-infra | 07:15 | |
| omr | When I got aborted job (timeout) I do not have logs..so I wonder what is the best way to collect the logs in jenkins when the timeout expired in my Job | 07:15 |
| *** ramashri has quit IRC | 07:15 | |
| *** dizquierdo has joined #openstack-infra | 07:18 | |
| *** dkranz has quit IRC | 07:18 | |
| *** AJaeger has joined #openstack-infra | 07:19 | |
| *** jpich has quit IRC | 07:20 | |
| *** dkranz has joined #openstack-infra | 07:20 | |
| *** Daisy_ has joined #openstack-infra | 07:22 | |
| *** amcrn has quit IRC | 07:23 | |
| *** Daisy has quit IRC | 07:26 | |
| *** Daisy_ is now known as Daisy | 07:26 | |
| *** mrmartin has joined #openstack-infra | 07:26 | |
| *** sileht has quit IRC | 07:29 | |
| *** sileht has joined #openstack-infra | 07:29 | |
| *** berendt has joined #openstack-infra | 07:29 | |
| *** dizquierdo has quit IRC | 07:32 | |
| *** e0ne has joined #openstack-infra | 07:32 | |
| *** gokrokve has joined #openstack-infra | 07:34 | |
| *** dkranz has quit IRC | 07:34 | |
| *** dkranz has joined #openstack-infra | 07:34 | |
| *** gokrokve has quit IRC | 07:39 | |
| *** jistr has joined #openstack-infra | 07:39 | |
| *** yamamoto_ has joined #openstack-infra | 07:41 | |
| *** MaxV has joined #openstack-infra | 07:41 | |
| *** oomichi_ has quit IRC | 07:45 | |
| *** jcoufal has quit IRC | 07:46 | |
| *** pkoniszewski has joined #openstack-infra | 07:47 | |
| *** yamamoto_ has quit IRC | 07:48 | |
| *** dkranz has quit IRC | 07:50 | |
| *** dizquierdo has joined #openstack-infra | 07:51 | |
| *** dkranz has joined #openstack-infra | 07:52 | |
| *** andreykurilin has joined #openstack-infra | 07:56 | |
| omr | When I got aborted job (timeout) I do not have logs..so I wonder what is the best way to collect the logs in jenkins when the timeout expired in my Job? | 07:57 |
| *** jp_at_hp has joined #openstack-infra | 08:00 | |
| *** cnesa9 has joined #openstack-infra | 08:00 | |
| *** [HeOS] has joined #openstack-infra | 08:05 | |
| *** afazekas_ has joined #openstack-infra | 08:05 | |
| *** Ryan_Lane has quit IRC | 08:07 | |
| *** vigneshvar has quit IRC | 08:11 | |
| *** ihrachyshka has joined #openstack-infra | 08:13 | |
| *** ianw has quit IRC | 08:18 | |
| *** ianw has joined #openstack-infra | 08:19 | |
| *** amuller has joined #openstack-infra | 08:20 | |
| *** vigneshvar has joined #openstack-infra | 08:24 | |
| *** yamamoto has joined #openstack-infra | 08:24 | |
| *** dkranz has quit IRC | 08:25 | |
| *** dkranz has joined #openstack-infra | 08:26 | |
| *** pblaho has joined #openstack-infra | 08:29 | |
| *** derekh has joined #openstack-infra | 08:29 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:34 | |
| *** dkranz has quit IRC | 08:36 | |
| *** dkranz has joined #openstack-infra | 08:36 | |
| *** gokrokve has joined #openstack-infra | 08:37 | |
| *** unicell has quit IRC | 08:39 | |
| *** gokrokve has quit IRC | 08:41 | |
| *** ianw has quit IRC | 08:41 | |
| *** ianw has joined #openstack-infra | 08:42 | |
| *** tomoe has quit IRC | 08:44 | |
| *** tomoe has joined #openstack-infra | 08:44 | |
| *** sunrenjie6 has joined #openstack-infra | 08:46 | |
| *** andreaf has quit IRC | 08:47 | |
| *** tomoe has quit IRC | 08:48 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix zuul javascript cookie load https://review.openstack.org/115840 | 08:49 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix status page overlap https://review.openstack.org/115841 | 08:50 |
| *** oomichi has joined #openstack-infra | 08:52 | |
| *** unicell has joined #openstack-infra | 08:52 | |
| *** pkoniszewski has quit IRC | 08:54 | |
| *** andreykurilin has quit IRC | 08:55 | |
| *** e0ne has quit IRC | 08:55 | |
| *** yamamoto has quit IRC | 08:57 | |
| *** Daisy has quit IRC | 09:00 | |
| *** e0ne has joined #openstack-infra | 09:01 | |
| *** katyafervent is now known as katyafervent_awa | 09:06 | |
| *** dtantsur is now known as dtantsur|afk | 09:07 | |
| *** bookwar has quit IRC | 09:07 | |
| *** andreykurilin has joined #openstack-infra | 09:08 | |
| *** oomichi has quit IRC | 09:10 | |
| *** katyafervent_awa is now known as katyafervent | 09:10 | |
| *** dkranz has quit IRC | 09:11 | |
| *** dkranz has joined #openstack-infra | 09:11 | |
| *** _buzz has quit IRC | 09:13 | |
| *** _buzz has joined #openstack-infra | 09:14 | |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** jcoufal has joined #openstack-infra | 09:20 | |
| *** [HeOS] is now known as HeOS | 09:21 | |
| *** michchap_ has quit IRC | 09:21 | |
| *** michchap has joined #openstack-infra | 09:22 | |
| *** rfolco has joined #openstack-infra | 09:23 | |
| *** che-arne has joined #openstack-infra | 09:23 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: multi-node dg setup https://review.openstack.org/106043 | 09:27 |
| *** dkranz has quit IRC | 09:27 | |
| *** dkranz has joined #openstack-infra | 09:30 | |
| *** mpaolino has quit IRC | 09:34 | |
| *** gokrokve has joined #openstack-infra | 09:34 | |
| *** sunrenjie6 has quit IRC | 09:35 | |
| *** yamamoto has quit IRC | 09:36 | |
| *** rlandy has joined #openstack-infra | 09:37 | |
| *** gokrokve has quit IRC | 09:39 | |
| *** dkranz has quit IRC | 09:45 | |
| *** dkranz has joined #openstack-infra | 09:46 | |
| *** dims_ has joined #openstack-infra | 09:51 | |
| *** pelix has joined #openstack-infra | 09:52 | |
| *** amuller has quit IRC | 09:58 | |
| andreykurilin | sdague, hi! | 09:59 |
| *** dims_ has quit IRC | 10:00 | |
| *** dims_ has joined #openstack-infra | 10:01 | |
| *** mpaolino has joined #openstack-infra | 10:03 | |
| *** jgallard has joined #openstack-infra | 10:06 | |
| *** dims_ has quit IRC | 10:06 | |
| *** d0ugal has quit IRC | 10:08 | |
| *** d0ugal has joined #openstack-infra | 10:08 | |
| *** yamamoto has joined #openstack-infra | 10:09 | |
| *** Guest1245 has quit IRC | 10:12 | |
| *** chenglch has quit IRC | 10:20 | |
| *** jgallard has quit IRC | 10:22 | |
| *** marun_afk has joined #openstack-infra | 10:25 | |
| *** bhuvan has quit IRC | 10:29 | |
| *** yamahata has quit IRC | 10:31 | |
| *** gokrokve has joined #openstack-infra | 10:34 | |
| *** pcm_ has joined #openstack-infra | 10:35 | |
| *** pblaho is now known as pblaho|afk | 10:36 | |
| *** pcm_ has quit IRC | 10:37 | |
| *** e0ne has quit IRC | 10:37 | |
| *** pcm_ has joined #openstack-infra | 10:37 | |
| *** gokrokve has quit IRC | 10:39 | |
| *** markmc has joined #openstack-infra | 10:43 | |
| *** tomoe has joined #openstack-infra | 10:45 | |
| *** jyuso has quit IRC | 10:49 | |
| *** tomoe has quit IRC | 10:50 | |
| *** michchap_ has joined #openstack-infra | 10:53 | |
| *** marun_afk is now known as marun | 10:56 | |
| *** michchap has quit IRC | 10:56 | |
| *** Vega has joined #openstack-infra | 10:58 | |
| *** Vega is now known as Guest81378 | 10:58 | |
| Guest81378 | \nick Zhiyuan | 10:59 |
| *** Guest81378 is now known as Zhiyuan | 11:00 | |
| *** Zhiyuan has left #openstack-infra | 11:01 | |
| *** Zhiyuan has joined #openstack-infra | 11:01 | |
| *** Zhiyuan has quit IRC | 11:01 | |
| *** LuckyVega has joined #openstack-infra | 11:03 | |
| *** e0ne has joined #openstack-infra | 11:10 | |
| LuckyVega | Hi, folks. Today our CI failed to connect to the review server. Zuul log shows that "AuthenticationException" is raised. And when I try "ssh -p 29418 huawei-ci@review.openstack.org gerrit stream-events", the command returns "Permission denied(publickey)" | 11:10 |
| LuckyVega | Is there something wrong with our CI account? | 11:10 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 11:12 | |
| *** penguinRaider has joined #openstack-infra | 11:15 | |
| *** marun has quit IRC | 11:15 | |
| *** dims_ has joined #openstack-infra | 11:17 | |
| *** mbacchi has joined #openstack-infra | 11:18 | |
| *** MaxV has quit IRC | 11:20 | |
| *** MaxV has joined #openstack-infra | 11:20 | |
| *** pblaho|afk is now known as pblaho | 11:24 | |
| *** MaxV has quit IRC | 11:25 | |
| *** dtantsur|afk is now known as dtantsur | 11:26 | |
| *** MaxV has joined #openstack-infra | 11:27 | |
| openstackgerrit | Andrey Nikitin proposed a change to openstack-infra/gerritlib: Update listGroups function in gerrit.py https://review.openstack.org/120747 | 11:29 |
| *** baoli has joined #openstack-infra | 11:30 | |
| *** dims_ has quit IRC | 11:30 | |
| *** Guest1245 has joined #openstack-infra | 11:31 | |
| *** dims_ has joined #openstack-infra | 11:31 | |
| *** dkranz has quit IRC | 11:32 | |
| *** gokrokve has joined #openstack-infra | 11:34 | |
| *** dims__ has joined #openstack-infra | 11:34 | |
| *** loki184 has quit IRC | 11:35 | |
| *** IvanBerezovskiy has joined #openstack-infra | 11:35 | |
| *** dims_ has quit IRC | 11:35 | |
| *** gokrokve has quit IRC | 11:39 | |
| *** lttrl has joined #openstack-infra | 11:39 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Introduce tripleo-puppet-elements into Stackforge https://review.openstack.org/120285 | 11:39 |
| *** cnesa9 has quit IRC | 11:43 | |
| *** _nadya_ has quit IRC | 11:46 | |
| omr | Hi, When I got aborted job (timeout) I do not have logs..so I wonder what is the best way to collect the logs in jenkins when the timeout expired in my Job? | 11:49 |
| *** pblaho is now known as pblaho|afk | 11:51 | |
| *** penguinRaider has quit IRC | 11:51 | |
| *** enikanorov_ has joined #openstack-infra | 11:54 | |
| *** enikanorov__ has quit IRC | 11:54 | |
| *** andreaf has joined #openstack-infra | 11:54 | |
| *** dane_leblanc has joined #openstack-infra | 11:55 | |
| HeOS | Can anyone review this change ? https://review.openstack.org/116936, https://review.openstack.org/117199, https://review.openstack.org/120747. Can this change https://review.openstack.org/116936 be submit? | 11:57 |
| *** adalbas has joined #openstack-infra | 11:57 | |
| *** mwagner_lap has quit IRC | 11:59 | |
| *** _nadya_ has joined #openstack-infra | 11:59 | |
| *** koolhead17 has quit IRC | 12:06 | |
| *** koolhead17 has joined #openstack-infra | 12:07 | |
| *** kgiusti has joined #openstack-infra | 12:08 | |
| *** weshay has joined #openstack-infra | 12:08 | |
| *** markmcclain has joined #openstack-infra | 12:09 | |
| sdague | lifeless: well now my fail rate for changing launchpad states has returned to the "normal" 20% | 12:10 |
| openstackgerrit | Chmouel Boudjnah proposed a change to stackforge/dox: Add tests for tox_ini https://review.openstack.org/120765 | 12:11 |
| *** koolhead17 has quit IRC | 12:12 | |
| *** mbacchi has quit IRC | 12:13 | |
| *** dprince has joined #openstack-infra | 12:14 | |
| *** mbacchi has joined #openstack-infra | 12:14 | |
| *** mwagner_lap has joined #openstack-infra | 12:15 | |
| *** e0ne has quit IRC | 12:16 | |
| *** KanagarajM has quit IRC | 12:17 | |
| *** ppai has quit IRC | 12:17 | |
| *** s0nea has quit IRC | 12:17 | |
| *** aysyd has joined #openstack-infra | 12:20 | |
| *** zz_jgrimm is now known as jgrimm | 12:20 | |
| *** nosnos has quit IRC | 12:22 | |
| *** zz_dimtruck is now known as dimtruck | 12:22 | |
| *** annegentle has joined #openstack-infra | 12:22 | |
| *** nosnos has joined #openstack-infra | 12:22 | |
| *** masayukig has quit IRC | 12:24 | |
| *** koolhead17 has joined #openstack-infra | 12:26 | |
| *** nosnos has quit IRC | 12:27 | |
| *** annegentle has quit IRC | 12:28 | |
| *** cnesa9 has joined #openstack-infra | 12:28 | |
| *** doug-fish has joined #openstack-infra | 12:28 | |
| daya_k | hello, i updated devstack-gate yesterday, and since then i can no longer start mysql, because it is not able to create mysqld.log in /var/log | 12:28 |
| daya_k | i created the file and set owner to mysql:mysql but still failing | 12:29 |
| daya_k | wanted to check on the forum on what the ownership should be | 12:29 |
| *** masayukig has joined #openstack-infra | 12:30 | |
| *** lttrl has quit IRC | 12:30 | |
| *** mpaolino has quit IRC | 12:31 | |
| *** pblaho|afk is now known as pblaho | 12:32 | |
| *** dkranz has joined #openstack-infra | 12:32 | |
| *** s0nea has joined #openstack-infra | 12:33 | |
| *** bswartz1 has joined #openstack-infra | 12:34 | |
| *** gokrokve has joined #openstack-infra | 12:34 | |
| *** bswartz has quit IRC | 12:36 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 12:37 | |
| fungi | SpamapS: one note about sshfp records. you need dnssec implemented for the zone containing them and for the resolved chain to your clients, otherwise they (at least openssh but probably others) will not trust the recorss | 12:37 |
| fungi | records | 12:37 |
| *** julim has joined #openstack-infra | 12:39 | |
| *** gokrokve has quit IRC | 12:39 | |
| fungi | LuckyVega: are you subscribed to the third-party-announce@lists.openstack.org mailing list? it looks like the huawei ci account was disabled a week ago according to http://lists.openstack.org/pipermail/third-party-announce/2014-September/000007.html | 12:40 |
| *** lttrl has joined #openstack-infra | 12:41 | |
| fungi | omr: you'll have to be more specific. a job running where? was this a devstack-based job running in the openstack (upstream) ci systems or something you're trying to run yourself? | 12:41 |
| omr | fungi: I am talking about a devstack job that build a version from the upstream, I run it with Jenkins and somtimes I got timeout expired during the tempest, I do not want to increase the timeout but I want some idea how to save the logs | 12:44 |
| omr | The macro that create the logs in artifacts is not been called when I got aborted.. | 12:44 |
| fungi | HeOS: i think i may be misunderstanding the commit message in 120747. it claims to depend on 117199 but it looks to me like 117199 depends on 120747 instead... | 12:44 |
| *** michchap_ has quit IRC | 12:45 | |
| fungi | omr: if you're using devstack-gate to set up and run devstack and tempest, then you can pass it a shorter timeout value than your jenkins job timeout. we usually make it about 5 minutes less to give a timed out devstack-gate run an opportunity to collect and upload logs before jenkins kills it | 12:45 |
| *** michchap has joined #openstack-infra | 12:46 | |
| *** tomoe has joined #openstack-infra | 12:46 | |
| daya_k | update: i tried 'sudo chcon system_u:object_r:mysqld_log_t:s0 /var/log/mysqld.log ' and 'restorecon -v /var/log/mysqld.log' but its still not launching | 12:46 |
| fungi | omr: you can see how we apply it in our jenkins-job-builder configuration, for example here http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml | 12:46 |
| *** radez_g0n3 is now known as radez | 12:47 | |
| fungi | daya_k: are you using a platform that installs mariadb when you ask for the mysql packages? i believe on those platforms (fedora? rhel7?) we install mysql-community-server instead of mysql-server to get around that | 12:47 |
| HeOS | fungi, I will correct dependencies. | 12:47 |
| HeOS | One moment | 12:48 |
| daya_k | fungi: i have community-mysql installed, this is on fedora 20 | 12:48 |
| fungi | daya_k: you might want to confer with ianw since i believe he did a lot of the legwork to get devstack running on f20 | 12:48 |
| daya_k | fungi : i had faced this issue before, so checked to make sure community-mysql is installed, and i'm just trying to start it with 'systemctl status mysqld.service' | 12:49 |
| *** koolhead17 has quit IRC | 12:49 | |
| daya_k | fungi : thanks i will check. the prob is this was all working until yesterday! | 12:49 |
| *** dims__ has quit IRC | 12:49 | |
| *** dims_ has joined #openstack-infra | 12:50 | |
| daya_k | fungi : i had other issues becuase devstack-gate was outdated, i wasnt loading some oslo projects, so updated devstack-gate | 12:50 |
| fungi | daya_k: how are you setting up your f20 workers? using our http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_node_devstack.sh script or something else? | 12:50 |
| *** tomoe has quit IRC | 12:50 | |
| *** dims_ is now known as dims | 12:51 | |
| openstackgerrit | Andrey Nikitin proposed a change to openstack-infra/jeepyb: Update mechanism of creating groups in database and managing access rights with jeepyb https://review.openstack.org/117199 | 12:51 |
| daya_k | fungi : i dont have nodepool set up. am re-using the node, but when i did, i used this script, yes | 12:51 |
| daya_k | on a side note, will be good if there are some pointers on how to use nodepools. i see some scripts, but to pull it all together, dont see any doc. | 12:52 |
| openstackgerrit | Andrey Nikitin proposed a change to openstack-infra/gerritlib: Update listGroups function in gerrit.py https://review.openstack.org/120747 | 12:52 |
| fungi | daya_k: well, there's the nodepool documentation http://ci.openstack.org/nodepool and then documentation on how we configure and run our nodepool server http://ci.openstack.org/nodepool.html | 12:53 |
| daya_k | fungi : i had added some cleanup scripts including unstack before every run, and it has worked ok for almost a month | 12:53 |
| *** marun has joined #openstack-infra | 12:53 | |
| fungi | daya_k: also, you can see, for example, the nodepool configuration we use (admittedly somewhat large) http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 12:54 |
| daya_k | fungi : i'll try and set it up, thanks for the links | 12:54 |
| *** HenryG_zzzz is now known as HenryG | 12:55 | |
| *** miqui has joined #openstack-infra | 12:55 | |
| *** sandywalsh has joined #openstack-infra | 12:56 | |
| HeOS | fungi, done. | 12:56 |
| *** dalgaaf_ has joined #openstack-infra | 12:57 | |
| LuckyVega | fungi: thanks, I will fix that issue soon | 12:59 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: StackTach.v3 Quincy REST API https://review.openstack.org/120582 | 13:01 |
| *** jgrimm is now known as zz_jgrimm | 13:03 | |
| *** dane_leblanc has quit IRC | 13:04 | |
| gilliard | How are the new additions to gerrit UI (Toggle CI, etc) developed? Is there an openstack-gerrit project? | 13:07 |
| *** hdd has joined #openstack-infra | 13:08 | |
| fungi | gilliard: it's a javascript overlay | 13:08 |
| *** dane_leblanc has joined #openstack-infra | 13:08 | |
| dims | gilliard: look under git://git.openstack.org/openstack-infra/config | 13:08 |
| fungi | gilliard: no actual gerrits were forked in the making of this feature | 13:08 |
| dims | fungi: that's funny :) | 13:09 |
| sdague | fungi: nice | 13:09 |
| gilliard | good, thanks fungi | 13:09 |
| *** vhoward has joined #openstack-infra | 13:09 | |
| omr | fungi: How do you configure the devstack-gate? | 13:10 |
| omr | timeout | 13:10 |
| *** ramashri has joined #openstack-infra | 13:10 | |
| fungi | omr: did you look at the example i linked? | 13:11 |
| fungi | omr: note that in that first template you see, the build-timeout wrapper is set to 125, that's the job timeout in jenkins | 13:12 |
| omr | fungi: I see DEVSTACK_GATE_TIMEPUT | 13:12 |
| fungi | omr: the DEVSTACK_GATE_TIMEOUT of 120 we're exporting in the shell builder is consumed by the devstack-gate wrap script to time out the test payload with enough of a lead that it can still copy logs before the job timeout is hit | 13:13 |
| *** dustins has joined #openstack-infra | 13:13 | |
| omr | fungi: yes i see, I do not know DEVSTACK_GATE_TIMEOUT in my CI I will check it | 13:14 |
| omr | fungi: Thanks a lot | 13:14 |
| fungi | omr: you're welcome | 13:14 |
| *** tomoe has joined #openstack-infra | 13:15 | |
| *** ramashri has quit IRC | 13:15 | |
| *** ramashri has joined #openstack-infra | 13:17 | |
| HeOS | fungi, is all good in changes? | 13:17 |
| *** ndipanov has joined #openstack-infra | 13:18 | |
| fungi | HeOS: i'm starting to look more closely at them now, not sure yet. doing many things at the same time since it's still the start of my day | 13:18 |
| *** tomoe has quit IRC | 13:19 | |
| *** tomoe has joined #openstack-infra | 13:19 | |
| HeOS | Ok. | 13:20 |
| *** annegentle has joined #openstack-infra | 13:23 | |
| *** tomoe has quit IRC | 13:24 | |
| *** dimtruck is now known as zz_dimtruck | 13:24 | |
| *** mriedem has joined #openstack-infra | 13:24 | |
| ndipanov | hey guys - I've never seen something like this https://review.openstack.org/#/c/114938/25 | 13:25 |
| ndipanov | i.e. not get an X for the merge conflict | 13:25 |
| ndipanov | and also it seems to have failed the merge with the patch _from it\s own branch_ | 13:25 |
| ndipanov | could it be a gerrit git impl bug? | 13:25 |
| fungi | ndipanov: that is a rare disagreement between cgit (what zuul uses to calculate merge commits) and jgit (what gerrit runs internally as its git server) | 13:26 |
| ndipanov | and the question is - can I do anything other than just re-upload to the back of the queue | 13:26 |
| *** dmsimard_away is now known as dmsimard | 13:26 | |
| ndipanov | fungi, that really sucks then because it colided with something on it's own branch... tht's a serious bug imho | 13:27 |
| fungi | ndipanov: cgit and jgit aren't identical in their merge calculation algorithms and occasionally we'll run across a conflict which zuul thinks is non-encroaching but gerrit then disagrees | 13:27 |
| ndipanov | sigh | 13:27 |
| fungi | ndipanov: it will need to be rebased before gerrit will allow it to merge | 13:27 |
| ndipanov | fungi, so that means back of the (28 hour long) line, right? | 13:28 |
| *** annegentle has quit IRC | 13:28 | |
| *** dangers_away is now known as dangers | 13:28 | |
| fungi | ndipanov: was it part of a long patch series? we *have* seen what seems to be a bug in jgit where an n-way merge fails opaquely claiming there's a conflict if n>~4 | 13:28 |
| ndipanov | yes it was | 13:29 |
| fungi | ndipanov: yes, it will need to go back through check and gate | 13:29 |
| ndipanov | 9 patches | 13:29 |
| LuckyVega | fungi: hi fungi, do you know how to config zuul not to post the "start checking ..." comment? | 13:29 |
| fungi | ndipanov: avoiding long patch series or spacing out their approvals also helps avoid that bug | 13:29 |
| ndipanov | fungi, I see | 13:30 |
| ndipanov | IMHO this is a serious bug that basically makes the FFE rush not an option for larger features | 13:30 |
| *** signed8bit has joined #openstack-infra | 13:31 | |
| *** pcrews has joined #openstack-infra | 13:31 | |
| fungi | ndipanov: yep. totally something the jgit developers need to address, unless you're interested in helping convert gerrit to cgit | 13:31 |
| *** ihrachyshka has quit IRC | 13:32 | |
| ndipanov | didn't we use it before the upgrade | 13:32 |
| ndipanov | ? | 13:32 |
| fungi | i personally am not so keen on spending my time hacking on java spaghetti but ymmv | 13:32 |
| fungi | ndipanov: yeah, i don't recall if it was present before the upgrade, but when we jumped from gerrit 2.4 to 2.8 it came with a much newer bundled jgit release | 13:33 |
| *** torandu has joined #openstack-infra | 13:33 | |
| *** koolhead17 has joined #openstack-infra | 13:33 | |
| fungi | so there's a good chance it was a bug introduced in jgit some time in the past couple years | 13:33 |
| fungi | since we effectively jumped forward a couple years in gerrit versions and hence jgit versions as well | 13:33 |
| *** alkari has quit IRC | 13:34 | |
| ndipanov | fungi, I see | 13:34 |
| fungi | LuckyVega: compare how we have our check and gate pipelines configured in http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 13:34 |
| fungi | LuckyVega: you'll notice in our check pipeline, there is no start action while in our gate pipeline there is | 13:34 |
| *** sabeen has joined #openstack-infra | 13:34 | |
| *** gokrokve has joined #openstack-infra | 13:34 | |
| fungi | LuckyVega: omitting the start action in your pipeline config should prevent it from commenting | 13:35 |
| ndipanov | fungi, ok thanks anyway | 13:35 |
| *** pblaho has quit IRC | 13:35 | |
| fungi | ndipanov: there's some chance that it's already fixed upstream in jgit, in which case it might go away when we move to gerrit 2.9 | 13:35 |
| fungi | or possibly 3.0 since it may end up releasing before we get a chance to upgrade | 13:36 |
| ndipanov | fungi, not in couple of years I hope :) | 13:36 |
| *** mfer has joined #openstack-infra | 13:36 | |
| fungi | ndipanov: no, now that we're not running a fork of gerrit, we should be able to move more quickly | 13:36 |
| *** weshay has quit IRC | 13:36 | |
| fungi | mostly just trying not to perform disruptive upgrades except early in the dev cycle | 13:36 |
| ndipanov | fungi, I will quote you when I go beg PTL to let me merge stuff over the weekend :) | 13:37 |
| fungi | ndipanov: hey, no maintenance scheduled this weekend! | 13:37 |
| ndipanov | \o/ | 13:37 |
| fungi | though we might be moving manila next weekend, jury's still out | 13:37 |
| *** weshay has joined #openstack-infra | 13:39 | |
| *** gokrokve has quit IRC | 13:39 | |
| LuckyVega | fungi: OK, I have removed the start action from layout.yaml. | 13:39 |
| LuckyVega | and restart zuul | 13:39 |
| *** mjturek has joined #openstack-infra | 13:40 | |
| *** vhoward has left #openstack-infra | 13:41 | |
| *** Longgeek has quit IRC | 13:41 | |
| *** esker has joined #openstack-infra | 13:41 | |
| *** e0ne has joined #openstack-infra | 13:42 | |
| LuckyVega | :fungi Since our account is disabled, how to check if this issue is addressed? | 13:42 |
| sandywalsh | hey y'all, trying to figure out this acl-config: setting | 13:42 |
| sandywalsh | my acl group is modules/openstack_project/files/gerrit/acls/stackforge/stacktach.config | 13:42 |
| fungi | LuckyVega: was that the only issue you needed to address? if so, i'll go ahead and reenable the account | 13:43 |
| sandywalsh | but the acl-group examples seem to map to /home/gerrit2/... | 13:43 |
| sandywalsh | not sure how that works | 13:43 |
| sandywalsh | any docs on that flag? | 13:43 |
| dolphm | \o/ keystone, swift and openstack/requirements check/gate jobs are *all* failing due to bug 1368048. a fix in swift was just approved https://review.openstack.org/#/c/120737/ -- may it please be escorted to the front of the gate? | 13:44 |
| uvirtbot | Launchpad bug 1368048 in swift "testQuotedWWWAuthenticateHeader date time encoding error" [Undecided,In progress] https://launchpad.net/bugs/1368048 | 13:44 |
| dstufft | dhellmann: so uh, TIL that someone backported the PEP 420 implicit namespace packages | 13:44 |
| fungi | sandywalsh: that's the local path on the gerrit server that the contents of modules/openstack_project/files/gerrit/acls tree gets installed into | 13:44 |
| *** signed8bit has quit IRC | 13:44 | |
| fungi | dolphm: taking a look now | 13:44 |
| *** yamamoto has quit IRC | 13:45 | |
| *** signed8bit has joined #openstack-infra | 13:45 | |
| *** bknudson has joined #openstack-infra | 13:46 | |
| fungi | dolphm: is that fix something that stable branches are going to need to, or is keystonemiddleware only used by juno swift? | 13:46 |
| dolphm | fungi: good question... if keystonemiddleware master is being tested on stable branches, then yes | 13:47 |
| fungi | dolphm: also, for once the fire is out, is this an indication we're lacking keystonemiddleware integration testing that should have caught it? | 13:47 |
| *** mrmartin has quit IRC | 13:47 | |
| *** resker has joined #openstack-infra | 13:47 | |
| LuckyVega | fungi: According to the mail, posting "starting check jobs" is the only issue | 13:47 |
| *** esker has quit IRC | 13:47 | |
| AJaeger | fungi, the translations jobs work fine again - but now we have a different problem: WE get every 24 hours a new update and patches will take longer than 24 hours to go through the gate (and nova was just approved, so it's only 16 hours)... | 13:47 |
| dolphm | fungi: stable branches might be dependent on python-keystoneclient.middleware.auth_token instead (as keystonemiddleware is new for juno), in which case: no | 13:47 |
| LuckyVega | fungi: http://lists.openstack.org/pipermail/third-party-announce/2014-September/000001.html | 13:47 |
| sandywalsh | fungi: based on that, my current acl-config is /home/gerrit2/acls/stackforge/stacktach.config ... seems correct, but fails. Looking in monster console output for some clue | 13:47 |
| dolphm | quick everyone say things to fungi all at once | 13:47 |
| fungi | yes. and this is not the only channel either | 13:48 |
| * fungi needs a brain multiplexer | 13:48 | |
| AJaeger | So, patches for translation will not go in right now. Do you think the queues will get shorter soon and we should wait - or should we disable the translations sometiems? | 13:48 |
| AJaeger | sorry, fungi... | 13:48 |
| AJaeger | fungi, one thing at a time - I come back later ;) | 13:48 |
| dolphm | fungi: this is why you are an openstack demigod | 13:49 |
| AJaeger | dolphm, ;) | 13:49 |
| *** hdd has quit IRC | 13:49 | |
| *** yamamoto has joined #openstack-infra | 13:50 | |
| *** flaviof_zzz is now known as flaviof | 13:50 | |
| sandywalsh | dolphm: just putting stuff in the queue, assuming it'll come out the other end at some point | 13:50 |
| fungi | dolphm: okay, there's a promote event waiting to get processed once the internal (result, event) queues catch up in zuul. should be at the front in a few minutes i hope | 13:50 |
| *** balles has joined #openstack-infra | 13:50 | |
| dolphm | fungi: awesome, thank you! | 13:51 |
| fungi | dolphm: just showed up, so hopefully barring other random failures, you ought to be sorted in an hour or so | 13:51 |
| sandywalsh | hmm, I think this bashate failure is unrelated to the acl thing ... no mention of stacktach in there at all. | 13:52 |
| dolphm | fungi: if it fails, will a recheck suffice to bring it back to the front of the gate, or do you i need to further harass -infra? | 13:52 |
| fungi | dolphm: you need to keep pestering in that event, yes. we'd want to re-enqueue it into the gate (so it doesn't have to wait for fresh check results) and then promote it to the front again | 13:53 |
| fungi | hopefully that won't be necessary | 13:53 |
| dolphm | fungi: ack. hopefully i'll leave you guys in peace now. thanks again! | 13:53 |
| fungi | dolphm: it's why i'm here, don't hesitate to let me know if you need something | 13:54 |
| *** wenlock has joined #openstack-infra | 13:55 | |
| dolphm | fungibot: sleep 10 | 13:55 |
| fungi | heh | 13:56 |
| *** markmcclain has quit IRC | 13:58 | |
| fungi | LuckyVega: okay, i've reenabled that account now. please make sure you're subscribed to the third-party-announce mailing list so you don't miss disablement notifications in the future. also anteaya would probably want me to recommend attending the weekly meeting for third-party ci operators if you're available... https://wiki.openstack.org/wiki/Meetings/ThirdParty | 13:58 |
| fungi | AJaeger: so... you want to talk about translation update jobs? were you going to ask about decreasing the frequency of update proposals? | 14:00 |
| *** arnaud has joined #openstack-infra | 14:00 | |
| AJaeger | fungi, yes, decreasing frequency is one option - or temporary disabling - or just waiting and hope the queue gets shorter... | 14:00 |
| AJaeger | fungi, our demigod, do you have any words of advise? ;) | 14:01 |
| *** nelsnelson has joined #openstack-infra | 14:02 | |
| AJaeger | we could also push the nova patch to the front of the queue... | 14:03 |
| fungi | AJaeger: so... i guess if the problem is only that it's disruptive during string freeze (because strings aren't really frozen) then we could consider disabling the proposal jobs... though ideally they wouldn't actually have a diff anyway when string freeze is imposed? | 14:03 |
| AJaeger | They have a diff from one day to the other since the date is encoded in them. | 14:03 |
| AJaeger | WE do not propose a new update, if *just* the date changes (a new optimization since a few months). | 14:04 |
| fungi | AJaeger: as for frequency, making it weekly instead of daily would probably just mean larger diffs spaced out further, and overall more translations invalidated as a result of the increased lag? | 14:04 |
| AJaeger | But if there is already an existing patch and we run the job again, it might have just a new date in it.. | 14:04 |
| *** homeless has joined #openstack-infra | 14:04 | |
| *** gargola has quit IRC | 14:05 | |
| *** Longgeek has joined #openstack-infra | 14:05 | |
| AJaeger | fungi, if we do it e.g. weekly, the post job would still send new strings to transifex, just the data in the tree becomes outdated. | 14:05 |
| AJaeger | But looking at projects today, they often only import once a month. | 14:05 |
| fungi | oh, wait, i see what the problem is... when the gate pipeline backup reaches roughly a day, no matter how quickly you approve the new update it will always get sniped out of the gate by another | 14:05 |
| AJaeger | fungi, yes, that's the problem. | 14:06 |
| AJaeger | sorry, I wasn't good enough in explaining it. | 14:06 |
| fungi | okay, so maybe there's a way to make it more intelligent, like checking to see if the change it's about to update is already approved? | 14:06 |
| fungi | and just skipping that update if so | 14:06 |
| AJaeger | fungi, that'S a great idea! Let me do this... | 14:06 |
| fungi | AJaeger: so it'll need an additional gerrit api call... maybe we should be leveraging gerritlib for that if so | 14:07 |
| AJaeger | fungi, check https://jenkins.openstack.org/view/All/job/nova-propose-translation-update/148/consoleText | 14:07 |
| AJaeger | we have that information already with our call to gerrit in: | 14:07 |
| AJaeger | approvals: | 14:08 |
| AJaeger | type: Workflow | 14:08 |
| AJaeger | description: Workflow | 14:08 |
| AJaeger | value: 1 | 14:08 |
| fungi | AJaeger: oh, cool then just use that i guess | 14:08 |
| AJaeger | Now just crafting a proper regex ;) | 14:08 |
| *** andreykurilin has quit IRC | 14:08 | |
| *** andreykurilin_ has joined #openstack-infra | 14:08 | |
| fungi | AJaeger: oh, also we should check to see if it has a -2 verify vote and update in that case even if there is a workflow +1 | 14:08 |
| bmwiedemann | anteaya: is that normal, that I get 4 emails about Cisco CI updating https://review.openstack.org/#/c/120730/ | 14:08 |
| *** eharney has joined #openstack-infra | 14:08 | |
| LuckyVega | :fungi Thanks a lot! I have subscribed the mailing list. I am in China so I probably cannot attend the weekly meeting since I am sleeping that time :) | 14:09 |
| fungi | AJaeger: otherwise if someone approves the patch and then it fails to merge and nobody notices, it will sit there indefinitely with workflow +1 but verify -2 and never get new patchsets until someone notices | 14:09 |
| AJaeger | fungi, yep | 14:09 |
| fungi | LuckyVega: in that case, you might like to browse the meeting minutes the next day when you wake up: http://eavesdrop.openstack.org/meetings/third_party/2014/ | 14:10 |
| *** hdd has joined #openstack-infra | 14:12 | |
| fungi | AJaeger: there is also another more rare hole it could fall into, where it drops out of the gate because we had to restart zuul and nobody reenqueues it for some reason, so it has a workflow +1 but no verify vote from the jenkins user, but that one would be hard to differentiate from a change actually working its way through the gate and rare enough we might not need to bother worrying about it | 14:12 |
| fungi | and all it takes is someone abandoning or rechecking the change to put it back on track | 14:13 |
| AJaeger | fungi, exactly | 14:13 |
| *** chuckC has quit IRC | 14:13 | |
| *** mpaolino has joined #openstack-infra | 14:14 | |
| *** zz_dimtruck is now known as dimtruck | 14:14 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Work toward Python 3.4 support and testing https://review.openstack.org/118769 | 14:15 |
| *** zz_jgrimm is now known as jgrimm | 14:15 | |
| fungi | bmwiedemann: looks like it would only have been three comments from cisco ci, but i agree that's unacceptable. i'll try to find the operators for it real quick | 14:15 |
| *** mrmartin has joined #openstack-infra | 14:16 | |
| EmilienM | jhesketh: hey, I had to rebase https://review.openstack.org/#/c/120285/ - maybe you can +2 again. thx | 14:16 |
| jeblair | mika: that's really weird, the fetch element on line 275 is empty, basically meaning there's no way to actually git fetch the change. if you look in the gerrit ui, does it show fetch commands under the "Download" section of the patchset info in the change? | 14:17 |
| AJaeger | fungi, do you know whether we have any code that does this parsing already? | 14:17 |
| LuckyVega | fungi: fine | 14:17 |
| fungi | dane_leblanc: are you aware of your cisco ci system leaving multiple messages on a change for separate tests? for example https://review.openstack.org/120730 | 14:17 |
| *** mpaolino has quit IRC | 14:17 | |
| bmwiedemann | fungi: it were 4 ... and thanks for caring :-) | 14:17 |
| fungi | AJaeger: it's possible you could look at how zuul queries changes in gerrit | 14:18 |
| *** jergerber has joined #openstack-infra | 14:18 | |
| *** Guest1245 has quit IRC | 14:18 | |
| fungi | AJaeger: since i know it cares about checking certain parameters of a change | 14:18 |
| fungi | AJaeger: it's also possible gerritlib has a method for this, and if not it might be a helpful addition there | 14:19 |
| *** ihrachyshka has joined #openstack-infra | 14:20 | |
| *** wenlock has quit IRC | 14:20 | |
| *** annegentle has joined #openstack-infra | 14:21 | |
| *** flaviof is now known as flaviof_zzz | 14:21 | |
| *** annegentle has quit IRC | 14:21 | |
| *** amotoki has quit IRC | 14:21 | |
| *** yamahata has joined #openstack-infra | 14:23 | |
| fungi | bmwiedemann: ahh, yes, there are indeed four comments there. i can't read :/ | 14:24 |
| *** ashaeron has quit IRC | 14:25 | |
| *** annegentle has joined #openstack-infra | 14:25 | |
| *** dmellado_ has joined #openstack-infra | 14:26 | |
| *** ashaeron has joined #openstack-infra | 14:26 | |
| *** zz_gondoi is now known as gondoi | 14:28 | |
| *** stevemar has joined #openstack-infra | 14:28 | |
| dane_leblanc | fungi: We have multiple plugins being tested, each with their own results. We have only one corporate CI account. | 14:29 |
| fungi | dane_leblanc: okay, not sure whether that's a problem or not, however three of the four are posting useless messages that say they didn't run anything | 14:30 |
| mika | jeblair: nope, if you mean the http://michael-prokop.at/screeni/screenshot.2014-09-11T16:29:45.png as it's present on e.g. https://review.openstack.org/#/c/120433/ - i don't have that available, investigating :) thanks for hint! | 14:30 |
| *** tomoe has joined #openstack-infra | 14:30 | |
| fungi | dane_leblanc: so at least that still needs correcting | 14:30 |
| jeblair | mika: yes, that's exactly what i meant. puzzling! | 14:31 |
| *** vigneshvar has quit IRC | 14:31 | |
| mika | jeblair: indeed, probably something with gitweb integration, let's see :) will report back as soon as i know more :) | 14:32 |
| *** dmellado_ has quit IRC | 14:32 | |
| ekarlso- | phschwartz: u tharr sir ? | 14:32 |
| *** david-lyle has joined #openstack-infra | 14:32 | |
| fungi | mika: is it possible that fetches are controllable via acl and you haven't granted that permission to registered users (or have explicitly revoked it)? | 14:33 |
| mika | fungi: good hint - thx, checking | 14:33 |
| *** tomoe has quit IRC | 14:34 | |
| *** gokrokve has joined #openstack-infra | 14:34 | |
| *** tomoe has joined #openstack-infra | 14:34 | |
| *** jp_at_hp has quit IRC | 14:35 | |
| *** wenlock has joined #openstack-infra | 14:36 | |
| *** dane_leblanc has quit IRC | 14:37 | |
| *** david-lyle has quit IRC | 14:38 | |
| *** amotoki has joined #openstack-infra | 14:38 | |
| *** gokrokve has quit IRC | 14:39 | |
| *** arnaud has quit IRC | 14:40 | |
| *** gokrokve has joined #openstack-infra | 14:41 | |
| *** flaviof_zzz is now known as flaviof | 14:41 | |
| jeblair | "The Supreme Court of the United States is split on whether a possessive singular noun that ends with s should always have an additional s after the apostrophe" --wikipedia | 14:43 |
| jeblair | apparently that's a 5-4 split; it doesn't say whether it's on party lines | 14:43 |
| mordred | jeblair: I'm blown away by multiple things about that | 14:44 |
| *** pkoniszewski has joined #openstack-infra | 14:45 | |
| mordred | jeblair: a) that the supreme court has an opinion on this topic | 14:45 |
| mordred | jeblair: b) that there is a dissenting opinion (first I've heard of it) | 14:45 |
| mordred | jeblair: c) that your morning took you to that information :) | 14:45 |
| *** jp_at_hp has joined #openstack-infra | 14:46 | |
| jeblair | mordred: a) it's not so much of an opinion as someone did a survey of the writing in past opinions and counted usage; however the questionable phrasing of the wikipedia author would lead one to think otherwise :) | 14:46 |
| fungi | i'm mostly flabbergasted by a) | 14:46 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 14:47 |
| jeblair | however, the court really does have an opinion on whether tomato(e)s are fruits or vegetables | 14:47 |
| fungi | ahh, yes, okay, so no scotus ruling on it then. i'm less appalled (or elated?) at my own government now | 14:47 |
| fungi | very, very mildly less | 14:47 |
| AJaeger | fungi, see above for a patch that passed my local testing based on your idea! | 14:47 |
| mordred | jeblair: ahhh... | 14:47 |
| *** markmcclain has joined #openstack-infra | 14:47 | |
| mordred | jeblair: also, really? | 14:47 |
| *** Guest1245 has joined #openstack-infra | 14:47 | |
| jeblair | mordred: and b) yeah, apparently usage is quite mixed: http://en.wikipedia.org/wiki/Apostrophe#Singular_nouns_ending_with_an_.E2.80.9Cs.E2.80.9D_or_.E2.80.9Cz.E2.80.9D_sound | 14:48 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 14:48 |
| sandywalsh | jhesketh: how did you determine it was an ACL issue in https://review.openstack.org/#/c/120582/ ... I don't see anything related to that in the console output | 14:48 |
| AJaeger | argh, forgot to change 1 to -1... | 14:48 |
| *** datsun180b has joined #openstack-infra | 14:48 | |
| fungi | i always avoided adding the extra s to possessives of words already ending in s unless they were proper nouns, but meh | 14:49 |
| *** markmcclain1 has joined #openstack-infra | 14:49 | |
| sandywalsh | jhesketh: the only things I can see are related to style issues on ./modules/openstack_project/files/slave_scripts/run-tarball.sh | 14:49 |
| *** markmcclain1 has quit IRC | 14:49 | |
| *** balles has quit IRC | 14:49 | |
| *** markmcclain1 has joined #openstack-infra | 14:50 | |
| sandywalsh | jhesketh: (and similarly unrelated files) | 14:50 |
| fungi | this is why english has a reputation for being an impossibly nuanced and unnecessarily complex language | 14:50 |
| *** gargola has joined #openstack-infra | 14:52 | |
| *** markmcclain has quit IRC | 14:52 | |
| *** tonytan4ever has joined #openstack-infra | 14:52 | |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: StackTach.v3 Quincy REST API https://review.openstack.org/120582 | 14:52 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 14:53 |
| dstufft | fungi: that's because the only real rule in english is that we make shit up as we go along | 14:53 |
| fungi | dstufft: i know i do | 14:53 |
| *** balles has joined #openstack-infra | 14:54 | |
| fungi | a language that shoots from the hip | 14:54 |
| *** ashaeron has quit IRC | 14:54 | |
| bknudson | I've got a question on this review -- https://review.openstack.org/#/c/119646/ | 14:54 |
| dstufft | Americans invented a language of FREEDOM where you're FREE to do wahtever you want! | 14:54 |
| bknudson | it's got -1 from Citrix XenServer CI | 14:54 |
| bknudson | then I did a recheck for it | 14:55 |
| *** cdent has joined #openstack-infra | 14:55 | |
| markvan | fungi: https://review.openstack.org/#/c/120116/ Next patch in my attempt to fix the openstack cookbook gates | 14:55 |
| sandywalsh | if "amazeballs" can get added to the Oxford dictionary, there's little hope. | 14:55 |
| bknudson | and Citrix XenServer CI says it passed... but it's still got -1 | 14:55 |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 14:55 |
| *** tomoe has quit IRC | 14:55 | |
| *** tomoe has joined #openstack-infra | 14:56 | |
| fungi | BobBall_AWOL: if you're not actually awol, any clue what's going on with the citrix xenserver ci votes on bknudson's https://review.openstack.org/119646 change | 14:57 |
| *** omr_ has joined #openstack-infra | 14:57 | |
| *** atiwari has joined #openstack-infra | 14:58 | |
| *** tomoe has quit IRC | 14:58 | |
| *** tomoe has joined #openstack-infra | 14:58 | |
| omr_ | fungi: "We usually make it about 5 minutes less..to collect logs.." Where do you collect the logs? here: cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh | 14:59 |
| omr_ | fungi: Sorry I meant here: publishers: - devstack-logs | 14:59 |
| *** marun has quit IRC | 15:00 | |
| fungi | omr_: yes | 15:00 |
| fungi | dolphm: that swift change just merged | 15:00 |
| *** chuckC has joined #openstack-infra | 15:00 | |
| *** bswartz1 is now known as bswartz | 15:01 | |
| dolphm | fungi: woot, thanks! | 15:01 |
| *** dimtruck is now known as zz_dimtruck | 15:01 | |
| *** garyh has joined #openstack-infra | 15:01 | |
| *** k4n0 has quit IRC | 15:01 | |
| *** mrmartin has quit IRC | 15:02 | |
| *** afazekas_ has quit IRC | 15:03 | |
| *** dane_leblanc has joined #openstack-infra | 15:04 | |
| *** zz_dimtruck is now known as dimtruck | 15:04 | |
| *** balles has quit IRC | 15:04 | |
| *** katyafervent is now known as katyafervent_awa | 15:05 | |
| *** ashaeron has joined #openstack-infra | 15:05 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 15:06 |
| AJaeger | fungi, the above works now for my local tests but I'll test some more later tonight. Any feedback on it is welcome! | 15:06 |
| jroll | jeblair or someone have a moment to review an old devstack-gate change? it's got a +2 already https://review.openstack.org/#/c/119473/ | 15:06 |
| *** balles has joined #openstack-infra | 15:06 | |
| fungi | AJaeger: i'll have a look | 15:06 |
| AJaeger | thanks, fungi! | 15:06 |
| *** Longgeek has quit IRC | 15:07 | |
| *** Sincler has joined #openstack-infra | 15:07 | |
| jeblair | jroll: did you check on that f20 failure? | 15:07 |
| mordred | sandywalsh: I think amazeballs getting added to the OED is the only thing that's keeping us relevant ... ;) | 15:08 |
| *** jcoufal has quit IRC | 15:08 | |
| *** loki184 has joined #openstack-infra | 15:08 | |
| *** andreykurilin_ is now known as andreykurilin | 15:08 | |
| jroll | jeblair: oops, did not, but this patch shouldn't actually affect current jobs | 15:09 |
| *** omr_ has quit IRC | 15:09 | |
| * jroll looks | 15:09 | |
| AJaeger | fungi, I'll be back online later... bye for now | 15:09 |
| *** AJaeger has quit IRC | 15:09 | |
| fungi | tempest.scenario.test_dashboard_basic_ops.TestDashboardBasicOps.test_basic_scenario[dashboard] | 15:09 |
| fungi | HTTPError: HTTP Error 500: INTERNAL SERVER ERROR | 15:09 |
| jroll | tempest.scenario.test_dashboard_basic_ops.TestDashboardBasicOps.test_basic_scenario | 15:09 |
| jroll | yeah | 15:09 |
| fungi | seems... unlikely to be related to the change | 15:09 |
| jroll | that definitely is unrelated :P | 15:10 |
| fungi | DEVSTACK_GATE_VIRT_DRIVER would have to be set to ironic for that variable to even come into play | 15:11 |
| *** omr has quit IRC | 15:12 | |
| *** sdake has joined #openstack-infra | 15:12 | |
| jeblair | cool, just checking; didn't want to be the person who approved "that change". ian would yell at us. :) | 15:13 |
| *** tomoe has quit IRC | 15:13 | |
| jeblair | jroll: why is this a switch? when will ironic build/not-build ramdisks? | 15:13 |
| *** tomoe has joined #openstack-infra | 15:13 | |
| jroll | fair enough :P | 15:14 |
| jroll | jeblair: /me finds an email | 15:14 |
| *** david-lyle has joined #openstack-infra | 15:15 | |
| jroll | jeblair: http://lists.openstack.org/pipermail/openstack-dev/2014-August/042357.html | 15:15 |
| jroll | jeblair: tl;dr ironic agent repo builds from source, ironic repo uses image on tarballs.o.o | 15:15 |
| *** yamahata has quit IRC | 15:15 | |
| *** marun has joined #openstack-infra | 15:17 | |
| *** tomoe has quit IRC | 15:18 | |
| *** rmcall has joined #openstack-infra | 15:19 | |
| *** markmcclain1 has quit IRC | 15:20 | |
| *** markmcclain has joined #openstack-infra | 15:20 | |
| *** balles has quit IRC | 15:24 | |
| *** reed has joined #openstack-infra | 15:25 | |
| *** rainya has joined #openstack-infra | 15:26 | |
| *** balles has joined #openstack-infra | 15:26 | |
| *** koolhead17 has quit IRC | 15:26 | |
| jeblair | jroll: thanks | 15:28 |
| *** armax has joined #openstack-infra | 15:30 | |
| *** rainya has quit IRC | 15:31 | |
| *** gokrokve_ has joined #openstack-infra | 15:32 | |
| *** rainya has joined #openstack-infra | 15:33 | |
| *** rhsu has joined #openstack-infra | 15:34 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Fix exit code in rally-cli job https://review.openstack.org/120815 | 15:35 |
| *** lttrl has quit IRC | 15:35 | |
| *** atiwari has quit IRC | 15:35 | |
| *** ashaeron has quit IRC | 15:35 | |
| jroll | jeblair: thank you :) | 15:35 |
| *** gokrokve has quit IRC | 15:35 | |
| *** todd_dsm has quit IRC | 15:36 | |
| *** lttrl has joined #openstack-infra | 15:36 | |
| *** gyee has joined #openstack-infra | 15:37 | |
| *** jgrimm is now known as zz_jgrimm | 15:37 | |
| *** harlowja_at_home has joined #openstack-infra | 15:39 | |
| openstackgerrit | Jim Rollenhagen proposed a change to openstack-infra/devstack-gate: Don't use ephemeral volumes for Ironic agent driver https://review.openstack.org/120817 | 15:39 |
| zul | can we get rid of the argparse dependency in kilo in general please ;) | 15:40 |
| *** david-lyle has quit IRC | 15:40 | |
| mordred | zul: isnt argparse deprecated or some shit? | 15:41 |
| *** HeOS has quit IRC | 15:41 | |
| *** todd_dsm has joined #openstack-infra | 15:43 | |
| zul | its still in the requirements.txt for everything and its a pain in the ass | 15:43 |
| fungi | zul: switch everything to optparse? awesome | 15:43 |
| j^2 | hey everyone. Is there any way we can move https://review.openstack.org/#/c/120116/ into merging? | 15:43 |
| *** IvanBerezovskiy has left #openstack-infra | 15:43 | |
| zul | fungi: well not its apart of python2.7 and python3+ | 15:44 |
| fungi | zul: yeah, i was joking | 15:45 |
| fungi | zul: with dropping python 2.6, presumably doable except that it isn't | 15:45 |
| zul | nnnngnnn | 15:45 |
| fungi | zul: because "kilo" is a server release, but lots of our clients (on which they're depending) still have to work with stable server releases and presumably still on stable server platforms (since the servers, insanely, depend on client libraries to talk to one another) | 15:46 |
| zul | fungi: yes dont get me started on that one | 15:46 |
| *** david-lyle has joined #openstack-infra | 15:46 | |
| fungi | zul: so once juno reaches eol, then we can probably stop supporting 2.6 in clients | 15:47 |
| fungi | and with it, the separate argparse | 15:47 |
| *** mmaglana has joined #openstack-infra | 15:47 | |
| *** jistr has quit IRC | 15:47 | |
| fungi | until then, we can curse and swear a loty | 15:48 |
| fungi | lot | 15:48 |
| *** zz_jgrimm is now known as jgrimm | 15:48 | |
| *** ramashri has quit IRC | 15:48 | |
| *** jebertol has joined #openstack-infra | 15:48 | |
| fungi | also booze | 15:48 |
| *** balles has quit IRC | 15:50 | |
| *** andreykurilin has quit IRC | 15:51 | |
| *** ZZelle has quit IRC | 15:52 | |
| *** ZZelle has joined #openstack-infra | 15:53 | |
| *** atiwari has joined #openstack-infra | 15:53 | |
| *** balles has joined #openstack-infra | 15:54 | |
| *** dtantsur is now known as dtantsur|afk | 15:55 | |
| *** che-arne has quit IRC | 15:58 | |
| *** arxcruz has quit IRC | 16:00 | |
| *** pballand has joined #openstack-infra | 16:02 | |
| *** jebertol has quit IRC | 16:02 | |
| *** harlowja_at_home has quit IRC | 16:03 | |
| *** Ajaeger has joined #openstack-infra | 16:04 | |
| *** dannywilson has joined #openstack-infra | 16:07 | |
| *** markmc has quit IRC | 16:07 | |
| *** koolhead17 has joined #openstack-infra | 16:07 | |
| *** bhuvan has joined #openstack-infra | 16:09 | |
| *** e0ne has quit IRC | 16:09 | |
| ihrachyshka | mordred: hey! I have a question on stackforge | 16:09 |
| ihrachyshka | mordred: who's in charge of releasing new pypi packages for those projects? (specifically, sqlalchemy-migrate) | 16:09 |
| ihrachyshka | I need a new release to move forward with my oslo spec | 16:10 |
| fungi | ihrachyshka: whichever group is granted push signed tag permission for that project in gerrit, generally | 16:11 |
| fungi | assuming they're using tag-based releasing a la pbr | 16:11 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Subscription events now appear in UI https://review.openstack.org/114404 | 16:12 |
| fungi | ihrachyshka: but in general, there's no guarantee that we have any actual details on the release process of an arbitrary stackforge project | 16:12 |
| fungi | ihrachyshka: one way is to look on pypi at the index and package owners | 16:13 |
| *** Guest1245 has quit IRC | 16:13 | |
| ihrachyshka | fungi: I see. though I think mordred is core for the project, plus he is probably the one who started the whole stackforge idea (?), so I hope he knows where to check. | 16:13 |
| *** koolhead17 has quit IRC | 16:14 | |
| ihrachyshka | fungi: I've checked pypi, iElectric is Maintainer, ches, evyaro, jandd, openstackci are owners | 16:14 |
| fungi | ihrachyshka: for that one in particular, sure. looks from https://pypi.python.org/pypi/sqlalchemy-migrate like openstackci is a a package index owner, yeah | 16:14 |
| ihrachyshka | not that I know any of them :) | 16:14 |
| fungi | ihrachyshka: well, openstackci is what we use to push packages via release automation | 16:15 |
| ihrachyshka | I would guess so. ;) so there is some permission tag for that? | 16:15 |
| fungi | ihrachyshka: looking at gerrit's acl for that project, the sqlalchemy-migrate-core group has push signed tag permission | 16:15 |
| ihrachyshka | aha! so any core can release new package?.. | 16:16 |
| fungi | ihrachyshka: so in theory anyone listed at https://review.openstack.org/#/admin/groups/186,members could trigger a release, assuming the release automation is set up correctly and the project is configured with the bits we need to support that process | 16:16 |
| *** praneshp has joined #openstack-infra | 16:16 | |
| Ajaeger | fungi, I'm back - and tested my patch further. Looks fine... | 16:16 |
| ihrachyshka | fungi: great. do they have some special button in UI? | 16:17 |
| Ajaeger | would be great to get this in today, there're quite a few approved translated patches that will otherwise updated and need reapproval. | 16:17 |
| *** koolhead17 has joined #openstack-infra | 16:17 | |
| *** koolhead17 has joined #openstack-infra | 16:17 | |
| mordred | ihrachyshka: nope. it's done by pushing git tags | 16:17 |
| mordred | ihrachyshka: are we in need of a new release? | 16:18 |
| fungi | ihrachyshka: they follow https://wiki.openstack.org/wiki/GerritJenkinsGit#Tagging_a_Release | 16:18 |
| *** MaxV has quit IRC | 16:18 | |
| ihrachyshka | mordred: yeah, I need it for glance to be able to run its migration rules against mysqlconnector | 16:18 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Remove query for (fixed) bug 1338844 https://review.openstack.org/120831 | 16:18 |
| uvirtbot | Launchpad bug 1338844 in tempest ""FixedIpLimitExceeded: Maximum number of fixed ips exceeded" in tempest nova-network runs since 7/4" [High,Fix released] https://launchpad.net/bugs/1338844 | 16:18 |
| mordred | dhellmann: any thoughts from you? | 16:18 |
| *** markmcclain has quit IRC | 16:19 | |
| *** marcoemorais has joined #openstack-infra | 16:20 | |
| *** rushiagr is now known as rushiagr_away | 16:21 | |
| vishy | anyone seen this bug before? http://logs.openstack.org/99/118699/3/check/check-grenade-dsvm/dbe8ade/logs/devstack-gate-cleanup-host.txt | 16:22 |
| *** annegentle has quit IRC | 16:22 | |
| ihrachyshka | mordred: I think he is not avail atm, but if you want to hear whether he thinks it's needed to release the package, you can see his comment at: https://blueprints.launchpad.net/oslo.db/+spec/enable-mysql-connector | 16:23 |
| vishy | ah nm i’m looking in the wrong file | 16:23 |
| ihrachyshka | 'Need a review in sqlalchemy-migrate: https://review.openstack.org/#/c/110611/ - landed but not released 9/9 - dhellmann' | 16:23 |
| mika | fungi: jeblair: according to https://gerrit-documentation.storage.googleapis.com/Documentation/2.9/user-review-ui.html the 2.9 version of gerrit seems to require the download-commands plugin, i'm giving this a shot now | 16:23 |
| *** annegentle has joined #openstack-infra | 16:23 | |
| fungi | mika: that's excellent information--thanks! | 16:24 |
| *** rwsu has quit IRC | 16:24 | |
| fungi | zaro: ^ something to keep in mind for when we're ready to move forward to 2.9 i suppose | 16:24 |
| *** ndipanov is now known as ndipanov_gone | 16:24 | |
| *** bhuvan has quit IRC | 16:25 | |
| *** loki184 has quit IRC | 16:26 | |
| j^2 | Is there any way we can move https://review.openstack.org/#/c/120116/ into merging? | 16:28 |
| *** emagana has joined #openstack-infra | 16:28 | |
| *** rwsu has joined #openstack-infra | 16:29 | |
| *** _nadya_ has quit IRC | 16:30 | |
| fungi | j^2: i appreciate your interest. the patch is a day old, has not (yet) been looked at by any core reviewers, and is for an unofficial (stackforge) project while our priorities have to place official (openstack) project issues first. asking every 45 minutes isn't likely to change that, but we do try to review all changes as time allows | 16:31 |
| j^2 | fungi: awesome thanks :) i just wanted to make sure it was on the radar :) | 16:31 |
| *** _nadya_ has joined #openstack-infra | 16:31 | |
| *** che-arne|2 has joined #openstack-infra | 16:31 | |
| *** pkoniszewski has quit IRC | 16:32 | |
| *** rwsu has quit IRC | 16:33 | |
| *** yfried has quit IRC | 16:33 | |
| *** maxbit has joined #openstack-infra | 16:35 | |
| *** balles has quit IRC | 16:35 | |
| jogo | random question oslo.db-python33 is non-voting but runs in the gate | 16:35 |
| jogo | is that a concern in anyway? | 16:35 |
| *** rainya has quit IRC | 16:35 | |
| *** rwsu has joined #openstack-infra | 16:35 | |
| jogo | in elastic-recheck we have no way of detecting if something is voting or not | 16:35 |
| fungi | jogo: it was probably voting and then emergency-switched to non-voting without being removed from the gate. there's likely no benefit to running it there, but also not particularly a rush to remove it either | 16:36 |
| jogo | fungi: we can hide it from the e-r page | 16:37 |
| mtreinish | jogo: we can always revive that patch which pulls in the layout.yaml and makes decisions from that... | 16:37 |
| mordred | ihrachyshka: looking | 16:37 |
| *** balles has joined #openstack-infra | 16:38 | |
| fungi | mtreinish: jogo: yeah, i'm doing basically that for evaluating the impact of the pypy and py34 trusty moves (using zuul's parser, because it's not trivial since it can need to match on regular expressions too) | 16:38 |
| *** ramashri has joined #openstack-infra | 16:39 | |
| *** bhuvan has joined #openstack-infra | 16:40 | |
| mtreinish | fungi: oh yeah forgot about that, the change earlier was a misguided attempt to figure out if a job was in a gate pipeline or not (or something like that) | 16:40 |
| mtreinish | but we already had a more reliable method for doing it | 16:40 |
| *** derekh has quit IRC | 16:40 | |
| fungi | another option though might be to get zuul to pass along voting metadata as a job parameter and then reflect that in the zmq event plugin so it can be indexed in logstash and filtered with kibana | 16:40 |
| *** ihrachyshka has quit IRC | 16:41 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Write cache to file on process exit https://review.openstack.org/117838 | 16:41 |
| jogo | fungi: that would be awesome, if we stored voting or not in elasticsearch | 16:41 |
| fungi | but that engineering effort may not be worth the benefits | 16:41 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Stop running non-voting oslo.db-python33 in gate https://review.openstack.org/120835 | 16:41 |
| jogo | fungi: well until then ^ | 16:41 |
| mtreinish | jogo, fungi: https://review.openstack.org/#/c/98264/ | 16:42 |
| *** jgrimm is now known as zz_jgrimm | 16:42 | |
| *** ihrachyshka has joined #openstack-infra | 16:42 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove query for (fixed) bug 1338844 https://review.openstack.org/120831 | 16:42 |
| uvirtbot | Launchpad bug 1338844 in tempest ""FixedIpLimitExceeded: Maximum number of fixed ips exceeded" in tempest nova-network runs since 7/4" [High,Fix released] https://launchpad.net/bugs/1338844 | 16:42 |
| fungi | mtreinish: yeah, that would probably need to import part of zuul to reuse its shared queue calculator | 16:43 |
| *** loki184 has joined #openstack-infra | 16:43 | |
| jogo | mtreinish: AFAIK jeblair is working on a better way of doing that | 16:43 |
| jogo | mtreinish: adding the shared queue stuff into logstash | 16:43 |
| jeblair | whamidoin? | 16:43 |
| jeblair | oh yeah | 16:43 |
| jogo | jeblair: \o/ | 16:44 |
| mordred | jamielennox: the session work in novaclient, cinderclient and neutronclient - are those released yet? or just in master so far? | 16:44 |
| jeblair | mtreinish: zuul knows that the shared queue for the integrated projects is called 'integrated'. i just haven't made it pass that info to graphite yet. | 16:44 |
| mordred | jamielennox: also, do you know where glanceclient stands? | 16:45 |
| *** r-daneel has joined #openstack-infra | 16:45 | |
| *** ildikov has quit IRC | 16:45 | |
| jogo | jeblair: can that go into logstash as well? | 16:45 |
| *** salv-orlando has joined #openstack-infra | 16:45 | |
| jeblair | fungi, mtreinish: do we also want that to go into logstash? | 16:45 |
| mordred | jogo: no. no logstash for you | 16:45 |
| jeblair | jogo: heh :) | 16:45 |
| jeblair | yeah, i think we can do as fungi suggests and also pass that as a ZUUL param, then logstash can pick it up | 16:45 |
| jogo | jeblair: sweet | 16:45 |
| fungi | jeblair: it sounds like elastic-recheck would be interested in subclassifying failures based on whether they're in projects in the integrated gate queue | 16:46 |
| jeblair | so both voting status, and 'name of shared change queue, if it is named' | 16:46 |
| jeblair | (not all change queues are named) | 16:46 |
| *** rushiagr_away is now known as rushiagr | 16:46 | |
| fungi | jeblair: well, my suggestion around that was actually about passing voting/non-voting metadata through from zuul to logstash, but yeah applies equally to queue name | 16:47 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Klugman - cmdline tool and client library for StackTach.v3 https://review.openstack.org/120838 | 16:49 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Quince - Quincy-compliant API implementation for StackTach.v3. Supports Winchester. https://review.openstack.org/120839 | 16:49 |
| *** patrickeast has joined #openstack-infra | 16:53 | |
| *** zz_jgrimm is now known as jgrimm | 16:55 | |
| jeblair | fungi, mordred, clarkb, SergeyLukjanov: please don't create new servers on ci-puppetmaster from this point forward (we're moving to a new puppetmaster for puppet3) | 16:55 |
| fungi | jeblair: yep--thanks for the reminder! | 16:56 |
| openstackgerrit | A change was merged to openstack/requirements: Rally needs bumping python-novaclient https://review.openstack.org/119776 | 16:57 |
| *** loki184 has quit IRC | 16:57 | |
| *** jlibosva has quit IRC | 16:58 | |
| *** rainya has joined #openstack-infra | 17:00 | |
| arosen | mordred: any idea about this http://paste.openstack.org/show/110227/ ? | 17:01 |
| arosen | Still stuck on it :( | 17:01 |
| *** dimtruck is now known as zz_dimtruck | 17:02 | |
| *** markmcclain has joined #openstack-infra | 17:02 | |
| *** Sukhdev has joined #openstack-infra | 17:04 | |
| *** dizquierdo has quit IRC | 17:04 | |
| *** sdake_ has joined #openstack-infra | 17:05 | |
| mordred | arosen: looking | 17:05 |
| jeblair | mordred: https://etherpad.openstack.org/p/infra-puppet3 | 17:06 |
| jeblair | mordred: the ansible is not right there -- the puppetmaster arg isn't going all the way through; any ideas | 17:06 |
| mordred | jeblair: looking | 17:06 |
| mordred | arosen: old version of setuptools? where/how did you install? | 17:07 |
| dhellmann | jogo: +1 on your patch to remove that py33 job from oslo.db for now | 17:08 |
| dhellmann | mordred, ihrachyshka : if one of you gives me the permissions I need, I will release sqlalchemy-migrate | 17:08 |
| arosen | mordred: I used this script from asselin that sets up jenkins. Should i pip --upgrade setuptools ? | 17:08 |
| mordred | dhellmann: I released it | 17:08 |
| jogo | dhellmann: thanks | 17:09 |
| mordred | arosen: well ... how did you install pip? | 17:09 |
| *** harlowja_away is now known as harlowja | 17:09 | |
| *** shashankhegde has joined #openstack-infra | 17:09 | |
| mordred | because a good pip install should lead you to have a solid setuptools install | 17:09 |
| dhellmann | mordred: well, good, then you don't need to give me permissions :-) | 17:09 |
| mordred | arosen: so I'm going down the rabbithole to see where thigns went off | 17:09 |
| *** Sukhdev has quit IRC | 17:09 | |
| *** doude has quit IRC | 17:10 | |
| arosen | mordred: via get_pip_url=https://bootstrap.pypa.io/get-pip.py | 17:10 |
| mordred | jeblair: are you running it on ci-puppetmaster and observing the param not plumbing through/ | 17:10 |
| mordred | ? | 17:10 |
| jeblair | mordred: yes | 17:10 |
| mordred | arosen: ok. cool. thats good | 17:10 |
| mordred | jeblair: k. one sec | 17:10 |
| *** Guest55717 has joined #openstack-infra | 17:10 | |
| arosen | mordred: as soon as i do: cd /opt/nodepool; sudo python setup.py develop; everything broke. | 17:10 |
| fungi | arosen: mordred: perhaps an incarnation of https://bitbucket.org/pypa/setuptools/issue/252/import-error-on-contextualzipfile-inside | 17:10 |
| mordred | arosen: yah. so - don't do python setup.py develop | 17:10 |
| mordred | arosen: always do pip install -e . instead | 17:11 |
| *** _nadya_ has quit IRC | 17:11 | |
| arosen | mordred: noted! | 17:11 |
| jeblair | arosen: (and "pip install ." instead of "python setup.py install", for the non-development case) | 17:12 |
| mordred | arosen: but yeah, try pip install -U setuptools==3.6 adn then pip install -U setuptools to see if you can reset the state | 17:13 |
| mordred | arosen: because that bug fungi pasted certainly does look relevant | 17:13 |
| fungi | possibly didn't remove *all* of the relevant distro packages for pip/setuptools/pkg-resources before installing pip from the script | 17:14 |
| arosen | I'm getting this now: http://paste.openstack.org/show/110233/ | 17:14 |
| harlowja | sdague u in? | 17:14 |
| clarkb | o/ | 17:16 |
| clarkb | jeblair: noted about not spinning up new servers on old puppetmaster | 17:16 |
| *** ZZelle_ has joined #openstack-infra | 17:17 | |
| SergeyLukjanov | jeblair ack | 17:17 |
| clarkb | jeblair: mordred: need more eyes on ansible parameter plumbing? | 17:17 |
| jeblair | clarkb: yes. i have not figured it out yet :( | 17:17 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: DevStack-like dev environment for StackTach.v3 https://review.openstack.org/120844 | 17:17 |
| clarkb | kk /me pulls up the code | 17:17 |
| Ajaeger | clarkb, SergeyLukjanov : could you review https://review.openstack.org/#/c/120800/, please? | 17:18 |
| jeblair | clarkb: seems to match the documentation here: http://docs.ansible.com/playbooks_roles.html#roles | 17:18 |
| clarkb | ya fungi linked to a couple documents including that one iirc | 17:19 |
| clarkb | and that is what I based my change on | 17:19 |
| clarkb | jeblair: is it failing because the required argument is missing? /me looks for puppet run all logs | 17:19 |
| *** Ryan_Lane has joined #openstack-infra | 17:19 | |
| asselin | arosen, mordred hi...catching up. had a firedrill | 17:20 |
| *** pballand has quit IRC | 17:21 | |
| clarkb | yup 'msg: missing required arguments: puppetmaster' so the code updated | 17:21 |
| asselin | arosen, why did you do this? 'cd /opt/nodepool; sudo python setup.py develop" | 17:21 |
| clarkb | jeblair: maybe we need to quote the string? it is yaml so we shouldn't need to | 17:21 |
| jeblair | clarkb: do we need to add it to the docstring? | 17:22 |
| * jeblair tries that | 17:22 | |
| asselin | arosen, the scripts install nodepool for you using infra's own nodepool puppet scripts | 17:22 |
| clarkb | jeblair: the example in the docstring uses a different arg passing format. maybe we should try that? | 17:24 |
| jeblair | clarkb: that doesn't look like valid yaml :/ | 17:25 |
| clarkb | jeblair: ya I didn't think so either = isn't a thing right? | 17:26 |
| *** tonytan4ever has quit IRC | 17:26 | |
| jeblair | clarkb: yeah. for completeness, i tried: - { role: puppet, puppetmaster=ci-puppetmaster.openstack.org } | 17:27 |
| jeblair | it did not work | 17:27 |
| jeblair | (no other variants i tried are remotely parseable | 17:27 |
| jeblair | i also added tho param to the DOCUMENTATION docstr, but that didn't help either | 17:27 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Cleanup javascript build https://review.openstack.org/111407 | 17:27 |
| clarkb | jeblair: maybe it needs - role: puppet\n puppetmaster: ci-puppetmaster.openstack.org? | 17:28 |
| jeblair | clarkb: should be equivalent | 17:28 |
| clarkb | though thats equivalent yaml.. | 17:28 |
| clarkb | ya | 17:28 |
| *** masayukig has quit IRC | 17:28 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Added documentation for StoryBoard https://review.openstack.org/108460 | 17:28 |
| *** lttrl has quit IRC | 17:29 | |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Notigen - OpenStack-like notification generator/simulator. https://review.openstack.org/120849 | 17:29 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Project may now be assigned to multiple project groups. https://review.openstack.org/111815 | 17:29 |
| clarkb | and grep tells me I spelled puppetmaster correctly in both locations | 17:29 |
| fungi | Ajaeger: i think 120800 has an oversight | 17:29 |
| fungi | Ajaeger: commented inline | 17:30 |
| Ajaeger | fungi, ok will check and update. | 17:30 |
| jeblair | so if ansible threw an actual exception and traceback, i would be able to figure out why this is happening. but instead, it throws a nice 'helpful' terse error message :( | 17:30 |
| arosen | asselin: the reason why i did that is i needed to pull in a few bug fixes to node pool | 17:30 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: DevStack-like dev environment for StackTach.v3 https://review.openstack.org/120844 | 17:30 |
| clarkb | http://docs.ansible.com/developing_modules.html#common-module-boilerplate I think the role module code itself is fine. The error saying requiring thing is missing leads to that conclusion as well | 17:30 |
| arosen | asselin: how do i go about pulling in and installing the lastest code using this? | 17:31 |
| clarkb | greghaynes: you ansible https://review.openstack.org/#/c/120577/ any idea why puppetmaster isn't being passed as a parameter to the puppet role in that change? | 17:31 |
| asselin | arosen, I see. I just installed and everything works fine now. I guess they merged to master? if so, just go and checkout master, clean that repo, and rerun the script. It should update it to master. | 17:32 |
| arosen | asselin: Though in mysetup if i rerun the script it fails with those cert errors :/ | 17:32 |
| arosen | I haven't started digging into that. | 17:33 |
| arosen | asselin: i'm setting up another clean env hopefully I can it working this time. Eventually though I want to make sure i have a way to update the code though :/ | 17:33 |
| asselin | arosen, :( I've no idea about those cert errors, but as i told you before, you really really need to be able to rerun the script a thousand times.... | 17:33 |
| jeblair | clarkb: do we need to pass it through the puppet _task_ ? | 17:33 |
| arosen | asselin: in your setup you can rerun the script without failure? | 17:34 |
| *** masayukig has joined #openstack-infra | 17:34 | |
| clarkb | jeblair: oh maybe? is that an extra level of indirection? | 17:34 |
| jeblair | clarkb: (it doesn't mention any other args, but maybe they don't work either) | 17:34 |
| *** dane_leblanc has quit IRC | 17:34 | |
| asselin | arosen, yes. everything I make a change the jenkins job, nodepool config, zuul layout. I run the script and it updates everthing to the latest. | 17:34 |
| arosen | asselin: and you're using ubuntu12.04 as the base image? | 17:34 |
| clarkb | jeblair: ya I think that must be it | 17:35 |
| asselin | arosen, yes, downloaded from ubuntu website. | 17:35 |
| clarkb | jeblair: because it lives under the roles file | 17:35 |
| jeblair | clarkb: yep | 17:35 |
| *** Sukhdev has joined #openstack-infra | 17:35 | |
| *** dane_leblanc has joined #openstack-infra | 17:35 | |
| jeblair | err: /File[/var/lib/puppet/lib]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet://notahost-puppetmaster.openstack.org/plugins: getaddrinfo: Name or service not known | 17:35 |
| *** balles has quit IRC | 17:35 | |
| jeblair | clarkb: ^ i inserted a bogus hostname in the puppet task for a quick test | 17:35 |
| asselin | arosen, only thing I do is setup the image to work with my lovely proxies. Once I have external network connection, the script does the rest. | 17:36 |
| jeblair | clarkb: that tells us 2 things: that's where it's complaining about, and the puppetmaster setting in the playbook is not overriding the value in the task | 17:36 |
| *** marun has quit IRC | 17:36 | |
| jeblair | clarkb: so we could either move the parameter to the task and put the playbook the way it was | 17:37 |
| jeblair | clarkb: or see if we can figure out how to make it a pass-through | 17:37 |
| clarkb | jeblair: or I think we need to do jinja2 type substition in the role | 17:37 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Update query for bug 1334142 https://review.openstack.org/120851 | 17:37 |
| uvirtbot | Launchpad bug 1334142 in nova "A server creation fails due to adding interface failure" [High,Fix released] https://launchpad.net/bugs/1334142 | 17:37 |
| jeblair | clarkb: does it use jinja? | 17:37 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: A time expressions library implementing a mini-language for manipulating datetimes. https://review.openstack.org/120852 | 17:37 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 17:38 |
| clarkb | jeblair: http://docs.ansible.com/playbooks_variables.html#using-variables-about-jinja2 I think so | 17:38 |
| jeblair | clarkb: so maybe "puppetmaster: {{puppetmaster}}" ? | 17:38 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: A time expressions library https://review.openstack.org/120852 | 17:38 |
| Ajaeger | fungi: Handling -2 now as well thanks for catching it! | 17:38 |
| clarkb | jeblair: ya | 17:38 |
| *** balles has joined #openstack-infra | 17:38 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:39 | |
| clarkb | jeblair: https://github.com/ansible/ansible-examples/blob/master/mongodb/roles/mongod/tasks/main.yml random example that seems to do similar | 17:39 |
| *** emagana has quit IRC | 17:39 | |
| *** ramashri has quit IRC | 17:39 | |
| jeblair | clarkb: yep, that's it, though its actually 'puppetmaster: "{{puppetmaster}}"' becase of yaml quoting | 17:40 |
| jeblair | clarkb: i'll put together a change | 17:40 |
| *** emagana has joined #openstack-infra | 17:40 | |
| *** salv-orlando has quit IRC | 17:42 | |
| arosen | asselin: on a clean install of your script i always run into this: http://paste.openstack.org/show/110250/ | 17:43 |
| *** pballand has joined #openstack-infra | 17:44 | |
| asselin | arosen, I've run into that before. It's likely a nodepool bug. Here's what I do to resolve it: | 17:44 |
| asselin | 1. stop the nodepoold process (kill -9) | 17:44 |
| *** rainya has quit IRC | 17:44 | |
| asselin | 2. sudo mysql -p | 17:44 |
| asselin | 3. use nodepool; | 17:44 |
| *** amuller has joined #openstack-infra | 17:45 | |
| asselin | 4. delete all the snapshots & images from the nodepool database. | 17:45 |
| *** emagana has quit IRC | 17:45 | |
| mordred | jeblair: gah. sorry, I was starting to look at that and then got sucked onto a quick call - looks like you found it? | 17:45 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Plumb puppetmaster through ansible task https://review.openstack.org/120855 | 17:45 |
| arosen | asselin: actually sorry I just manually applied this patch from clarkb and it fixed the issue for me: https://review.openstack.org/#/c/120639/1/nodepool/cmd/nodepoolcmd.py | 17:45 |
| jeblair | mordred, clarkb, fungi: yep, clarkb and i found it: https://review.openstack.org/120855 | 17:45 |
| asselin | 5. restart nodepoold | 17:45 |
| *** jp_at_hp has quit IRC | 17:46 | |
| jeblair | it's about to run with those changes in place | 17:46 |
| asselin | arosen, ok I will tag that and test it out | 17:46 |
| jeblair | and it seems to be succeeding | 17:46 |
| *** openstackgerrit has quit IRC | 17:46 | |
| *** rainya has joined #openstack-infra | 17:47 | |
| mordred | jeblair: +2 - shall I +A? | 17:47 |
| fungi | jeblair: okay, so what we had wasn't wrong so much as merely incomplete ;) | 17:47 |
| *** zz_dimtruck is now known as dimtruck | 17:47 | |
| jeblair | yep, should be good to +a | 17:47 |
| *** openstackgerrit has joined #openstack-infra | 17:47 | |
| *** pelix has quit IRC | 17:47 | |
| jeblair | so i think puppet2master should be back in business (may need to manually apply that one more time after it merges) | 17:48 |
| *** ramashri has joined #openstack-infra | 17:48 | |
| jeblair | i'm moving on to getting the puppet3master in shape | 17:48 |
| clarkb | I approved the change | 17:48 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: A data extraction and transformation library for OpenStack notifications. https://review.openstack.org/120858 | 17:48 |
| jeblair | clarkb: am i missing something, or does the apache server not actually appear in our puppetmaster manifest? | 17:51 |
| mordred | jeblair: I believe the apache server does not actually appear in our puppetmaster manifest | 17:51 |
| *** portante has quit IRC | 17:51 | |
| clarkb | yup it appears to be missing | 17:52 |
| jeblair | mordred: is there any configuration for it? | 17:52 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add non-isolated jobs to experimental pipeline https://review.openstack.org/120861 | 17:52 |
| jeblair | maybe it used to be pulled in by the puppet package or something? | 17:52 |
| fungi | Ajaeger: spotted a new problem introduced with your latest patchset there | 17:52 |
| mordred | jeblair: I think our puppetmaster manifest is likely incomplete | 17:52 |
| Ajaeger | fungi: Thanks for spotting | 17:53 |
| *** mrmartin has joined #openstack-infra | 17:53 | |
| clarkb | jeblair: there is a /etc/apache2/sites-enabled/puppetmaster file. It looks straightforward. I can work on puppettizing it if you like | 17:53 |
| jeblair | mordred: where did /etc/apache2/sites-available/puppetmaster come from? | 17:53 |
| jeblair | clarkb: cool, thx | 17:53 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Clean up bashate failures https://review.openstack.org/118944 | 17:53 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 17:53 |
| nibalizer | the puppetnaster-passenger package | 17:53 |
| Ajaeger | fungi: removed the "," - thanks | 17:53 |
| jeblair | nibalizer: that is missing from our new puppetmaster; should we start by installing that? | 17:54 |
| Ajaeger | thanks fungi! | 17:54 |
| nibalizer | jeblair: yea install that package | 17:55 |
| *** portante has joined #openstack-infra | 17:55 | |
| mordred | and apache | 17:55 |
| jeblair | mordred: it's pulling in apache | 17:55 |
| mordred | oh, good | 17:55 |
| fungi | so presumably the manifest for our master should ensure puppetmaster-passenger | 17:55 |
| mordred | yeah. I think so | 17:55 |
| clarkb | oh I don't even need to manage the vhost? | 17:55 |
| mordred | nope | 17:56 |
| clarkb | nibalizer: does puppetmaster-passenger install that too? /m dpkg -L's it | 17:56 |
| jeblair | clarkb: correct, you can drop what you were about to do and just require puppetmaster-passenger | 17:56 |
| mordred | puppetmaster-passenger does the right thing | 17:56 |
| clarkb | jeblair: cool | 17:56 |
| jeblair | i think it's a postinstall hook actually, so dpkg -L doesn't show it, but it's there | 17:56 |
| mordred | how "clever | 17:56 |
| mordred | " | 17:56 |
| jeblair | comes from /usr/share/puppetmaster-passenger/apache2.site.conf.tmpl | 17:56 |
| jeblair | which is in dpkg -L | 17:56 |
| *** rmcall has quit IRC | 17:57 | |
| *** tonytan4ever has joined #openstack-infra | 17:57 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Install puppetmaster-passenger on puppetmasters https://review.openstack.org/120863 | 17:58 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Binary data archiving library - supports uploading to object storage. https://review.openstack.org/120864 | 17:59 |
| jeblair | clarkb, nibalizer: current error in bottom of etherpad: https://etherpad.openstack.org/p/infra-puppet3 | 18:00 |
| *** rmcall has joined #openstack-infra | 18:01 | |
| jeblair | 'puppet module list' is empty | 18:01 |
| jeblair | install_modules.sh reports: | 18:02 |
| jeblair | Error: can't convert nil into String | 18:02 |
| jeblair | Error: Try 'puppet help module install' for usage | 18:02 |
| jeblair | picked a random module to install and pasted the output there | 18:02 |
| clarkb | hrm | 18:03 |
| mordred | jeblair: nice | 18:03 |
| mordred | jeblair: I think you forgot to ruby ruby the ruby before ruby ruby | 18:04 |
| jeblair | huh. /etc/puppet/modules/ exists and is full of modules. | 18:04 |
| clarkb | according to their docs that is still a valid command | 18:04 |
| *** tonytan4ever has quit IRC | 18:04 | |
| mordred | jeblair: is the moduledir set right in /etc/puppet/puppet.conf ? | 18:04 |
| jeblair | basemodulepath = /etc/puppet/modules | 18:04 |
| mordred | awesome | 18:04 |
| mordred | that's spectacular | 18:04 |
| jeblair | is that the right name? | 18:04 |
| mordred | jeblair: try: puppet module --modulepath-/etc/puppet/modules install blah | 18:06 |
| jeblair | https://docs.puppetlabs.com/references/stable/configuration.html#modulepath | 18:06 |
| *** SumitNaiksatam has joined #openstack-infra | 18:06 | |
| *** tonytan4ever has joined #openstack-infra | 18:06 | |
| *** marcoemorais has quit IRC | 18:06 | |
| clarkb | https://docs.puppetlabs.com/puppet/3.6/reference/dirs_modulepath.html | 18:06 |
| *** marcoemorais has joined #openstack-infra | 18:06 | |
| clarkb | so it defaults to basemodulepath. maybe try an explicit modulepath anyways? | 18:06 |
| *** marcoemorais has quit IRC | 18:06 | |
| mordred | mordred@puppetmaster:~$ sudo puppet config print modulepath --section master | 18:07 |
| mordred | nothing | 18:07 |
| mordred | that reports empty | 18:07 |
| *** marcoemorais has joined #openstack-infra | 18:07 | |
| jeblair | mordred: --modulepath=/etc/puppet/modules dtrt | 18:07 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add non-voting gate-git-review-python34 job https://review.openstack.org/119964 | 18:08 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add Python 3.4 and PyPy to Ubuntu Trusty slaves https://review.openstack.org/118859 | 18:08 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch sphinxcontrib-docbookrestapi to python-jobs https://review.openstack.org/118860 | 18:08 |
| *** marcoemorais has quit IRC | 18:08 | |
| *** lttrl has joined #openstack-infra | 18:08 | |
| *** marcoemorais has joined #openstack-infra | 18:08 | |
| mordred | jeblair: lovely | 18:08 |
| jeblair | i don't understand what the documentation wants me to do | 18:08 |
| *** rushiagr is now known as rushiagr_away | 18:09 | |
| *** dimtruck is now known as zz_dimtruck | 18:09 | |
| mordred | jeblair: so, I'm goign to suggest we make that puppet.conf use modulepath and not basemodulepath | 18:09 |
| nibalizer | jeblair: puppet --version ? | 18:09 |
| jeblair | root@puppetmaster:~# puppet --version | 18:09 |
| jeblair | 3.6.2 | 18:09 |
| mordred | on ci-puppetmaster, we set modulepath in the [master] section | 18:09 |
| clarkb | mordred: well we still want basemodulepath for other things potentially | 18:09 |
| mordred | clarkb: why? | 18:09 |
| mordred | we don't have that on the old one | 18:09 |
| jeblair | Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1095:in `block in issue_deprecations') | 18:10 |
| jeblair | and then it proceeds to fail | 18:10 |
| nibalizer | yea they changed how you do this, something called directory environments | 18:10 |
| mordred | jeez | 18:10 |
| clarkb | mordred: oh nevermind I thought other things used it too | 18:10 |
| *** emagana has joined #openstack-infra | 18:10 | |
| jeblair | so by deprecated, they mean, broken. | 18:10 |
| mordred | nibalizer: I'm not going to like your follow up answer here am I? | 18:10 |
| nibalizer | so if you want to remove environmentpath and basemoudlepath from puppet.conf i bet modulepath goes back to working | 18:10 |
| clarkb | the best kind of deprecated | 18:10 |
| *** ildikov has joined #openstack-infra | 18:10 | |
| clarkb | nibalizer: any idea why basemodulepath does not work? | 18:11 |
| mordred | well, we need to set modulepath, because it SHOULD be set to : modulepath=/opt/config/$environment/modules:/etc/puppet/modules | 18:11 |
| nibalizer | i was planning to roll us forward to using directory environments, because why not | 18:11 |
| mordred | or, _something_ needs to be set to that | 18:11 |
| jeblair | nibalizer: correct; when i do that, i get the deprecation warning and it works | 18:11 |
| nibalizer | clarkb: jeblair can i have puppet module --configprint modulepath | 18:11 |
| jeblair | Error: 'module' has no default action. See `puppet help module`. | 18:11 |
| * nibalizer table flips | 18:11 | |
| nibalizer | uh idunno puppet master --configprint modulepath ? | 18:11 |
| *** e0ne has joined #openstack-infra | 18:12 | |
| clarkb | fungi: is 118859 ready to go now? no additional pypy packages? | 18:12 |
| nibalizer | mordred: yea, what they've done with directory environments is allow us to put /etc/puppet/modules in basemodulepath then each environment has its own config file which adds on other directories to the path | 18:12 |
| clarkb | fungi: looks like you added the -dev package afterall | 18:12 |
| nibalizer | but the way they decided to cram it down our throats was pretty poorly done imho | 18:13 |
| jeblair | nibalizer: except that it doesn't seem to honor basemodulepath | 18:13 |
| arosen | asselin: The next thing i want to bit off is collecting the logs from the test runs. Are you about to mvn package the scp-plugin? It fails to build for me. | 18:13 |
| fungi | clarkb: yep, turned out they were indeed needed. with that addition, everything passes except for zaqar, which is exposing that pypy bug i need to pester ubuntu to backport | 18:13 |
| jeblair | puppet master --configprint modulepath | 18:14 |
| jeblair | /etc/puppet/modules | 18:14 |
| *** rushiagr_away is now known as rushiagr | 18:14 | |
| asselin | arosen, I can e-mail it? | 18:14 |
| jeblair | root@puppetmaster:~# puppet master --configprint modulepath | 18:14 |
| jeblair | is empty if i switch back to basemodulepath and environments | 18:14 |
| fungi | clarkb: so that stack of 3 should be safe to merge whenever. obviously we won't press forward with switching projects over until the pypy and python 3.4 fixes get backported in trusty | 18:14 |
| asselin | arosen, not sure where it should be hosted for now... | 18:14 |
| jeblair | nibalizer: okay, so removing environmentpath lets basemodulepath work | 18:14 |
| clarkb | fungi: yup +2 on the first one looking at the others now | 18:15 |
| clarkb | fungi: oh I am already +2 on them because non trivial change vote preservation | 18:15 |
| nibalizer | jeblair: wow that doesn't even make sorta sense, those variables are supposed to work together | 18:15 |
| clarkb | fungi: we should try to get in the package change today so that we hae images ready for the bits that do work | 18:15 |
| *** zz_dimtruck is now known as dimtruck | 18:16 | |
| jeblair | nibalizer: do we need to set up some actual environments in environmentdir | 18:16 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add non-isolated jobs to experimental pipeline https://review.openstack.org/120861 | 18:16 |
| jeblair | er, environmentpath | 18:16 |
| nibalizer | jeblair: hrm possibly | 18:17 |
| fungi | clarkb: yeah, i'm tempted to switch over all the projects which are fine and merely hold back the handful which aren't, but that can be revisited tomorrow when there are updated images | 18:17 |
| jeblair | nibalizer: 'production' still the default? | 18:17 |
| clarkb | fungi: can you update the etherpad with zaqar info? I don't see it | 18:17 |
| nibalizer | jeblair: https://docs.puppetlabs.com/puppet/latest/reference/environments_configuring.html | 18:17 |
| fungi | clarkb: oh, sure. since it was pypy rather than py33 i didn't bother. adding at the bottom | 18:18 |
| nibalizer | jeblair: yes production is still the default | 18:18 |
| fungi | er, rather than py34 | 18:18 |
| jeblair | nibalizer: 'mkdir /etc/puppet/environments/production' solved it | 18:18 |
| jeblair | didn't even need to put anything in there | 18:18 |
| clarkb | fungi: I don't see it . one of us must not be connected to etherpad properly. probably me | 18:18 |
| jeblair | nibalizer: so should /etc/puppet/environments/production == the git checkout of the config repo? | 18:19 |
| jeblair | nibalizer: we put it in /opt/config/production on the old master, but i'm thinking the new master should be ^^ ? | 18:19 |
| clarkb | fungi: oh envermind yo uare in the procss of adding that info | 18:19 |
| *** mmaglana has quit IRC | 18:20 | |
| nibalizer | jeblair: http://paste.openstack.org/show/110276/ | 18:20 |
| nibalizer | jeblair: my reccomendation comes in two parts | 18:21 |
| nibalizer | one is final state, the other is what i think we should do durring transition to final state | 18:21 |
| nibalizer | i think we leave the git checkouts in /opt for now | 18:21 |
| nibalizer | then refer to them using a modulepath = <path> directive in environment.conf | 18:21 |
| nibalizer | for final state, | 18:21 |
| jeblair | nibalizer: can you add that file to puppet then? | 18:22 |
| nibalizer | i think we put the git checkout in /etc/puppet/environments/<envname>/config | 18:22 |
| jeblair | /etc/puppet/environments/production/environment.conf | 18:22 |
| nibalizer | im not sure if we want to put environment.conf into puppet, i mean we can make that directory a git checkout but we could also just go one directory deeper and not keep that file in git | 18:22 |
| jeblair | nibalizer: /etc/puppet/environments/production/environment.conf is not going to exist on the puppetmaster if it is not in puppet | 18:23 |
| nibalizer | okay, so you're thinking we put environment.conf in the root of the config repo | 18:23 |
| jeblair | nibalizer: no | 18:23 |
| nibalizer | or you're saying manage that file in puppet? | 18:23 |
| clarkb | fungi: wow that bug report isn't really confidence inspiring :) | 18:24 |
| jeblair | nibalizer: your recommendation is to have /etc/puppet/environments/production/ be an empty directory except for environment.conf, right? | 18:24 |
| nibalizer | yes | 18:24 |
| clarkb | fungi: can we test zaqar with the --jit off flag? | 18:24 |
| jeblair | nibalizer: and that environment.conf points to the manifest and modulepath | 18:24 |
| jeblair | nibalizer: right, so if we want to do that, we should create/manage that file in puppet | 18:24 |
| clarkb | fungi: and if that fixes it maybe use that as a workaround for now? | 18:24 |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/requirements: Bump sqlalchemy-migrate to 0.9.2 https://review.openstack.org/120871 | 18:24 |
| fungi | clarkb: probably. i'll give it a shot (that's an interpreter flag so i can just stuff it into tox.ini? | 18:24 |
| clarkb | fungi: ya I think so | 18:25 |
| *** dimtruck is now known as zz_dimtruck | 18:25 | |
| nibalizer | ok | 18:25 |
| jeblair | nibalizer: ok, while you're puppeting that, i'll install it manually to keep moving | 18:26 |
| *** mikedillion has joined #openstack-infra | 18:27 | |
| *** vigneshvar has joined #openstack-infra | 18:28 | |
| *** zz_dimtruck is now known as dimtruck | 18:29 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add production environment to puppet https://review.openstack.org/120872 | 18:29 |
| nibalizer | jeblair: pushed ^ | 18:30 |
| jeblair | nibalizer: cool, thx. is /etc/puppet/hieradata/production/common.yaml still the place i should put our monolithing hiera file? | 18:30 |
| *** mikedillion has quit IRC | 18:30 | |
| * nibalizer thinks for a second | 18:30 | |
| nibalizer | I think so yes | 18:31 |
| *** che-arne|2 has quit IRC | 18:31 | |
| daya_k | ianw : ping | 18:31 |
| nibalizer | do you have an /etc/puppet/hiera.yaml on disk right now? what datadir: does it use? | 18:31 |
| Ajaeger | clarkb, jeblair: Due to the length of the gate right now , translation jobs have no chance to merge in since the periodic job will update them every day. Could you review later today the optimization that I wrote based on some great ideas by fungi, please? https://review.openstack.org/#/c/120800/ | 18:31 |
| *** amotoki has quit IRC | 18:31 | |
| clarkb | Ajaeger: yup | 18:31 |
| *** amotoki has joined #openstack-infra | 18:32 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Remove havana compat jobs from tempest experimental https://review.openstack.org/120876 | 18:32 |
| *** amcrn has joined #openstack-infra | 18:32 | |
| *** marcoemorais has quit IRC | 18:32 | |
| *** marcoemorais has joined #openstack-infra | 18:33 | |
| clarkb | fungi: yup --jit off fixes it. but I have to set it in .testr.conf because of the subprocessing | 18:33 |
| clarkb | fungi: probably need to put a conditional evaluation in there or something | 18:34 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Openstackid.org openid instance https://review.openstack.org/119364 | 18:34 |
| fungi | clarkb: ahh, i was trying to get it to work in tox.ini and running into interpreter errors | 18:34 |
| clarkb | but checking if py27 barfs on the flag first :) | 18:34 |
| clarkb | oh I expect that I will run ito that shortly | 18:34 |
| *** yfried has joined #openstack-infra | 18:34 | |
| clarkb | 'unknown option' :( | 18:36 |
| *** amotoki has quit IRC | 18:36 | |
| *** marcoemorais has quit IRC | 18:37 | |
| anteaya | JayF: thanks for your feedback, from what I understand this system for you to click a button to retain your membership is new | 18:38 |
| *** marcoemorais has joined #openstack-infra | 18:38 | |
| fungi | clarkb: yeah "Unknown option: --" | 18:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Plumb puppetmaster through ansible task https://review.openstack.org/120855 | 18:39 |
| *** unicell has quit IRC | 18:39 | |
| clarkb | fungi: I may have a hack that works | 18:41 |
| clarkb | its a super hack | 18:42 |
| *** ihrachyshka has quit IRC | 18:42 | |
| clarkb | also zaqar is looking for mongodb... | 18:42 |
| nibalizer | jeblair: im going afk for lunch, back in probably hour and a half | 18:42 |
| *** emagana has quit IRC | 18:42 | |
| fungi | hrm... i wonder did we install mongodb on precise-py3k nodes | 18:42 |
| *** emagana has joined #openstack-infra | 18:43 | |
| fungi | clarkb: yeah, we have mongodb on our precise-py3k nodes | 18:43 |
| fungi | clarkb: oh, it's installed on our bare-trusty nodes too | 18:43 |
| jogo | jeblair: if you want a good example of a gate wrecking ball: https://review.openstack.org/117839 | 18:44 |
| clarkb | fungi: ya it should be fine on the test slaves | 18:44 |
| fungi | clarkb: if you want, i have 198.61.165.181 (bare-trusty) and 104.130.27.139 (py3k-precise) held at the moment for experimentation | 18:44 |
| jogo | jeblair: the patch is from the 9th, and somewhere between now and then the unit test for that started failing | 18:44 |
| clarkb | fungi: nah its ok | 18:44 |
| clarkb | fungi: http://paste.openstack.org/show/110283/ how ugly is that | 18:45 |
| jeblair | jogo: ah, so it's also a case that may have been caught by a freshness check; though <48h would have been required presumably | 18:45 |
| *** MaxV has joined #openstack-infra | 18:46 | |
| fungi | clarkb: yeah, i can see maybe doing that in the short-term so we can just flip the node label for pypy jobs and not look back | 18:47 |
| jogo | jeblair: well maybe its not clear to me when the unit tests started to fail | 18:47 |
| *** emagana has quit IRC | 18:47 | |
| jogo | jeblair: they may have started failing just 12 hours ago | 18:47 |
| jeblair | nod | 18:47 |
| fungi | clarkb: wanna propose that to zaqar when you get a moment (or i can and put you as co-author) and i'll be happy to pester flaper87 to approve it | 18:48 |
| *** MarkAtwood has joined #openstack-infra | 18:48 | |
| jogo | jeblair: although the patch was approved in about 24 hours, and then spent 28 in the gate queue | 18:48 |
| *** lcheng_ has joined #openstack-infra | 18:50 | |
| *** jab has joined #openstack-infra | 18:50 | |
| *** jab has joined #openstack-infra | 18:50 | |
| clarkb | fungi: I can do it. wanted to make sure I wasn't completely insane before proposing it | 18:50 |
| fungi | clarkb: since this mostly confirms it is the bug i thought, let me go ahead and get the one for ubuntu opened and you can link that in your commit message | 18:51 |
| clarkb | fungi: thanks | 18:52 |
| *** vigneshvar has quit IRC | 18:53 | |
| *** __afazekas is now known as afazekas | 18:53 | |
| *** _buzz_ has joined #openstack-infra | 18:54 | |
| fungi | clarkb: any idea how we can narrow down an easier reproduction case for this than "run the unit tests for zaqar under pypy"? | 18:55 |
| clarkb | fungi: not really :( | 18:55 |
| clarkb | fungi: that particular upstream bug is light on details and after gdbing cpython I am not reall interested in debugging pypy :) | 18:56 |
| fungi | i'll see if i can whack it down smaller real fast | 18:56 |
| *** mrmartin has quit IRC | 18:56 | |
| clarkb | fwiw I basically said upstream bug has simple reproduction case, but this is what I am doing | 18:56 |
| clarkb | for the py34 bug | 18:56 |
| *** _buzz has quit IRC | 18:57 | |
| *** dane_leblanc has quit IRC | 18:57 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/config: Add version-properties builder to macros.yaml https://review.openstack.org/120381 | 18:58 |
| fungi | right, in this case the bug is "A: this seems broken. B: have you tried latest development version? A: yep, all fine there--thanks!" | 18:58 |
| fungi | and bisecting pypy commits is not something i really have time to spend on | 18:58 |
| clarkb | it does look like he sporadic pip timeouts bug is hitting us less often over the last couple days | 18:59 |
| fungi | but hopefully the package maintainer for it in ubuntu does | 18:59 |
| *** rushiagr is now known as rushiagr_away | 19:00 | |
| *** _nadya_ has joined #openstack-infra | 19:00 | |
| *** mrmartin has joined #openstack-infra | 19:01 | |
| *** melwitt has joined #openstack-infra | 19:01 | |
| fungi | trying 'testr run' with that pypy tox venv activated | 19:01 |
| *** gokrokve has joined #openstack-infra | 19:01 | |
| fungi | ahh, it's arising from 'discover -t . ./tests' | 19:02 |
| clarkb | oh so it is an import error. which kinda makes sense that --jit off would affect that | 19:02 |
| clarkb | --jit on would delay some of that processing and have different code paths in theory | 19:03 |
| fungi | 'discover -t .' also seems to do it | 19:03 |
| clarkb | fungi: so now you need to figure out which file does it | 19:03 |
| clarkb | fungi: you should be able to do a find . -name'*.py' -exec discover -t . {} | 19:04 |
| clarkb | or similar | 19:04 |
| fungi | seems it's coming from tests.unit.queues.transport.wsgi.test_v1_0.TestMessagesSqlalchemy.test_when_claim_deleted_then_messages_unclaimed | 19:04 |
| clarkb | oh add a -print too | 19:04 |
| *** dprince has quit IRC | 19:05 | |
| *** russellb has quit IRC | 19:05 | |
| *** gokrokve_ has quit IRC | 19:05 | |
| *** vigneshvar has joined #openstack-infra | 19:06 | |
| *** gokrokve has quit IRC | 19:06 | |
| *** JayF has quit IRC | 19:06 | |
| *** dtroyer has quit IRC | 19:06 | |
| *** adam_g has quit IRC | 19:06 | |
| *** StevenK has quit IRC | 19:06 | |
| *** russellb has joined #openstack-infra | 19:07 | |
| *** lcheng_ has quit IRC | 19:07 | |
| *** lbragstad has quit IRC | 19:07 | |
| *** rmcall has quit IRC | 19:08 | |
| *** dtroyer has joined #openstack-infra | 19:08 | |
| * clarkb wanders off ot find lunch. let me know what the eventual bug number is and I will push my workaround hack | 19:09 | |
| *** baoli has quit IRC | 19:09 | |
| *** fungi has quit IRC | 19:09 | |
| anteaya | Ajaeger: hi | 19:09 |
| *** freyes has quit IRC | 19:09 | |
| anteaya | Ajaeger: can you help me understand why pulling in some boilerplate from cookiecutter is preferable to you rather than beginning with a blank repo? | 19:10 |
| *** gondoi has quit IRC | 19:10 | |
| *** dstufft has quit IRC | 19:10 | |
| *** freyes has joined #openstack-infra | 19:10 | |
| *** lbragstad has joined #openstack-infra | 19:10 | |
| *** dstufft has joined #openstack-infra | 19:10 | |
| *** gondoi has joined #openstack-infra | 19:10 | |
| Ajaeger | anteaya: I don't care either way - my point was that the jobs are voting and thus an empty repo will not work. | 19:10 |
| *** d34dh0r53 has quit IRC | 19:11 | |
| *** d34dh0r53 has joined #openstack-infra | 19:11 | |
| *** shashankhegde has quit IRC | 19:11 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Update query for bug 1334142 https://review.openstack.org/120851 | 19:11 |
| uvirtbot | Launchpad bug 1334142 in nova "A server creation fails due to adding interface failure" [High,Fix released] https://launchpad.net/bugs/1334142 | 19:11 |
| *** jab has quit IRC | 19:12 | |
| Ajaeger | anteaya: Or is that impression wrong? I did not leave a rating since I wasn't sure... | 19:12 |
| *** alaski has quit IRC | 19:12 | |
| *** StevenK_ has joined #openstack-infra | 19:12 | |
| anteaya | Ajaeger: help me understand why having voting jobs with an empty repo won't work | 19:12 |
| *** alaski has joined #openstack-infra | 19:12 | |
| anteaya | Ajaeger: my understanding is that if you have voting jobs with an empty repo, all patches must pass the voting jobs | 19:12 |
| Ajaeger | anteaya: how will you get changes into that repo? You need to make an initial patch that passes all changes, won't you? | 19:13 |
| anteaya | yes | 19:13 |
| anteaya | how is that difficult to do? | 19:13 |
| Ajaeger | Is that really done this way? Sounds like a large change for me... | 19:13 |
| anteaya | why would the size of the change affect whether the patch can pass tests or not? | 19:14 |
| *** baoli has joined #openstack-infra | 19:14 | |
| *** Guest75250 has quit IRC | 19:14 | |
| anteaya | why would it be harder for an initial patch to pass tests compared to subsequent patches? | 19:15 |
| *** russellb has quit IRC | 19:15 | |
| *** dolphm has quit IRC | 19:16 | |
| openstackgerrit | Christian Fetzer proposed a change to openstack-infra/jenkins-job-builder: Add support for the Jenkins Clang Scan-Build publisher plugin. https://review.openstack.org/117958 | 19:16 |
| *** dtroyer has quit IRC | 19:17 | |
| Ajaeger | anteaya: mmh, will take into account next time. | 19:17 |
| *** david-lyle has quit IRC | 19:17 | |
| *** StevenK_ has quit IRC | 19:17 | |
| *** rhsu has quit IRC | 19:17 | |
| *** dolphm has joined #openstack-infra | 19:17 | |
| *** Guest3192 has quit IRC | 19:17 | |
| *** david-lyle has joined #openstack-infra | 19:17 | |
| *** StevenK_ has joined #openstack-infra | 19:17 | |
| *** annegentle has quit IRC | 19:18 | |
| *** freyes has quit IRC | 19:19 | |
| *** lbragstad has quit IRC | 19:19 | |
| *** jroll has quit IRC | 19:19 | |
| Ajaeger | anteaya: thanks! | 19:19 |
| clarkb | Ajaeger: 120800 looks good. It did take me a couple reads to really grok it though. Do you think it might be worthwhile to maybe have a single if with lots of conditions &&'d together? | 19:19 |
| clarkb | Ajaeger: that way the nesting is less ugly? | 19:19 |
| anteaya | Ajaeger: thank you | 19:19 |
| *** gondoi has quit IRC | 19:19 | |
| *** gondoi has joined #openstack-infra | 19:19 | |
| *** zeus has joined #openstack-infra | 19:20 | |
| *** zeus is now known as Guest1482 | 19:20 | |
| *** d34dh0r53 has quit IRC | 19:20 | |
| *** d34dh0r53 has joined #openstack-infra | 19:20 | |
| *** dprince has joined #openstack-infra | 19:20 | |
| *** alaski has quit IRC | 19:21 | |
| sdague | hey, anyone know who pushed a new sqlalchemy migrate ? | 19:21 |
| sdague | during freeze | 19:21 |
| clarkb | sdague: mordred? its a stackforge project though so not subject to such things in theory | 19:21 |
| sdague | because, it has actually ruined nova | 19:21 |
| sdague | 100% unit test failures | 19:22 |
| clarkb | some other project needed it for mysqlconnector compat | 19:22 |
| clarkb | sdague: nice | 19:22 |
| sdague | so... I believe there needs to be face punching on that | 19:22 |
| clarkb | sdague: I think there is a proposed requirements bump change too. maybe you should -2 it then push an exclusion? | 19:22 |
| Ajaeger | clarkb: do you have some ideas to make it easier to understand? I'll change the ifs now... | 19:22 |
| *** StevenK_ has quit IRC | 19:22 | |
| sdague | yeh, and I need you to escalate that | 19:22 |
| *** StevenK_ has joined #openstack-infra | 19:23 | |
| clarkb | Ajaeger: maybe a comment explaning what each grep checks for? | 19:23 |
| *** lbragstad has joined #openstack-infra | 19:23 | |
| clarkb | Ajaeger: eg check that change is approved and not -1 or -2 by jenkins and not -2 by human reviewers and not WIP | 19:24 |
| *** gondoi is now known as zz_gondoi | 19:24 | |
| *** pabelanger has quit IRC | 19:24 | |
| clarkb | Ajaeger: just something to help make sense of the dense json | 19:24 |
| Ajaeger | http://paste.openstack.org/show/110297/ | 19:25 |
| Ajaeger | clarkb: does this look better? | 19:25 |
| clarkb | sdague: wow 500 MB nova unittest console logs too | 19:25 |
| sdague | clarkb: yeh, well when every test fails, you get that | 19:25 |
| *** KanagarajM has joined #openstack-infra | 19:25 | |
| clarkb | Ajaeger: yup I think that is great | 19:26 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 19:26 |
| Ajaeger | clarkb: thanks! | 19:26 |
| *** dolphm has quit IRC | 19:26 | |
| clarkb | Ajaeger: thank you! | 19:26 |
| *** dolphm has joined #openstack-infra | 19:26 | |
| Ajaeger | fungi, clarkb: Please review again! | 19:26 |
| Ajaeger | This passed my local tests but check carefully ;) | 19:26 |
| *** pabelanger has joined #openstack-infra | 19:27 | |
| *** StevenK_ has quit IRC | 19:27 | |
| *** lbragstad has quit IRC | 19:28 | |
| *** vigneshvar has quit IRC | 19:28 | |
| *** zz_gondoi has quit IRC | 19:28 | |
| *** StevenK_ has joined #openstack-infra | 19:28 | |
| sdague | something up with the gerrit ipv6 addr? | 19:28 |
| *** Guest1482 has quit IRC | 19:28 | |
| clarkb | sdague: what are you seeing? | 19:28 |
| *** markmcclain has quit IRC | 19:28 | |
| sdague | Connection to 2001:4800:7815:104:3bc3:d7f6:ff03:bf5d timed out while waiting to read | 19:28 |
| sdague | fatal: Could not read from remote repository. | 19:28 |
| * clarkb tests | 19:28 | |
| *** alexpilotti has joined #openstack-infra | 19:29 | |
| *** d34dh0r53 has quit IRC | 19:29 | |
| clarkb | sdague: I am able to hit it from within a rax VM via ipv6 | 19:29 |
| *** d34dh0r53 has joined #openstack-infra | 19:29 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: block sqlalchemy migrate 0.9.2 as it breaks all of nova https://review.openstack.org/120890 | 19:30 |
| clarkb | hrm we lost fungi | 19:31 |
| clarkb | mordred: jeblair jhesketh SergeyLukjanov ^ | 19:31 |
| clarkb | I am +2 but a second vote would be nice before approving and ninja promoting | 19:31 |
| *** tonytan4ever has quit IRC | 19:32 | |
| clarkb | I wonder how quickly we are going to fill up logs.o.o at 500mb a pice :) | 19:33 |
| *** StevenK_ has quit IRC | 19:33 | |
| *** jbryce has quit IRC | 19:33 | |
| clarkb | we have 1.4TB available so not insane | 19:33 |
| *** fungi has joined #openstack-infra | 19:33 | |
| *** JayF has joined #openstack-infra | 19:33 | |
| *** stevebaker has quit IRC | 19:33 | |
| *** StevenK_ has joined #openstack-infra | 19:33 | |
| fungi | freenode is being teh sux0rz. or at least the server i was on decided to wander off from the herd for a while | 19:34 |
| *** roaet_ has quit IRC | 19:34 | |
| clarkb | fungi: welcome back https://review.openstack.org/120890 is the somewhat urgent thing of the moment | 19:34 |
| *** lbragstad has joined #openstack-infra | 19:34 | |
| clarkb | fungi: but Ajaeger also has a new patch for the translation pushes that makes it a lot more readable | 19:34 |
| fungi | i saw in the channel log | 19:34 |
| Ajaeger | fungi, could you recheck https://review.openstack.org/#/c/120800/ late, please | 19:34 |
| fungi | clarkb: new excitement... http://paste.openstack.org/show/110296/ | 19:35 |
| *** jbryce has joined #openstack-infra | 19:35 | |
| *** roaet_ has joined #openstack-infra | 19:35 | |
| fungi | clarkb: seems to sometimes do that and mostly do the empty "..." rpython traceback | 19:35 |
| *** Sukhdev has quit IRC | 19:35 | |
| *** dolphm has quit IRC | 19:35 | |
| clarkb | huh | 19:35 |
| *** dolphm has joined #openstack-infra | 19:35 | |
| clarkb | fungi: speed review and potential ninja promotion for 120890 is probably a good thing | 19:35 |
| *** pabelanger has quit IRC | 19:35 | |
| clarkb | fungi: nova unittests are spitting out >500MB of data per run now | 19:35 |
| *** gondoi has joined #openstack-infra | 19:36 | |
| *** sc68cal_ has joined #openstack-infra | 19:36 | |
| fungi | clarkb: yeah, i'll enqueue and promote | 19:36 |
| sdague | yeh, honestly, I would vote straight ninja merge on this one | 19:36 |
| clarkb | hrm | 19:36 |
| clarkb | sdague: we could do that | 19:36 |
| *** juice_ has joined #openstack-infra | 19:36 | |
| sdague | because you are going to generate a ton of gigs of crap in check if this has to wait 1.5 hrs | 19:36 |
| fungi | yeah, probably so. danger to the infrastructure, file sizes, blah, blah, blah | 19:36 |
| fungi | i'll merge it | 19:36 |
| *** lbragstad has quit IRC | 19:37 | |
| clarkb | wfm | 19:37 |
| *** gondoi has quit IRC | 19:37 | |
| sdague | clarkb: do we need a nodepool rebuild after this? | 19:37 |
| clarkb | sdague: you need a corresponding change to nova though | 19:37 |
| *** koolhead17 has quit IRC | 19:37 | |
| clarkb | sdague: no | 19:37 |
| sdague | clarkb: right, ... unit tests | 19:37 |
| clarkb | ya they won't get auto synced | 19:37 |
| *** maxbit_ has joined #openstack-infra | 19:37 | |
| clarkb | urg | 19:37 |
| *** berendt1 has joined #openstack-infra | 19:38 | |
| *** berendt1 has quit IRC | 19:38 | |
| *** gondoi has joined #openstack-infra | 19:38 | |
| *** StevenK_ has quit IRC | 19:38 | |
| *** d34dh0r53 has quit IRC | 19:38 | |
| *** colettecello has joined #openstack-infra | 19:38 | |
| fungi | merged. but also, openstackgerrit seems to have gone silent on us too | 19:38 |
| *** d34dh0r53 has joined #openstack-infra | 19:38 | |
| fungi | and yeah, i can see about trying to trigger the reqs proposal | 19:39 |
| *** StevenK_ has joined #openstack-infra | 19:39 | |
| *** yamahata__ has joined #openstack-infra | 19:39 | |
| clarkb | fungi: I think they can just do it by hand | 19:39 |
| *** homeless has quit IRC | 19:39 | |
| clarkb | the proposal will deal with out of band updates | 19:39 |
| fungi | oh, well yeah true | 19:39 |
| jogo | fungi: nova patch: https://review.openstack.org/120893 | 19:40 |
| *** juice has quit IRC | 19:40 | |
| *** sc68cal has quit IRC | 19:40 | |
| *** berendt has quit IRC | 19:40 | |
| *** jergerber has quit IRC | 19:40 | |
| *** yamahata_ has quit IRC | 19:40 | |
| *** jlk has quit IRC | 19:40 | |
| *** maxbit has quit IRC | 19:40 | |
| *** gothicmindfood has quit IRC | 19:40 | |
| openstackgerrit | A change was merged to openstack/requirements: block sqlalchemy migrate 0.9.2 as it breaks all of nova https://review.openstack.org/120890 | 19:40 |
| *** mriedem1 has joined #openstack-infra | 19:40 | |
| *** mriedem has quit IRC | 19:40 | |
| clarkb | that one should probably be promoted to make sure it actually fixes the problem and all that | 19:40 |
| fungi | clarkb: oh, actually, it's in the post pipeline, so it's running already | 19:40 |
| *** homeless has joined #openstack-infra | 19:40 | |
| *** juice_ is now known as juice | 19:41 | |
| sdague | https://review.openstack.org/120893 | 19:42 |
| sdague | right, https://review.openstack.org/120893 should get a promote | 19:42 |
| *** jmeridth has quit IRC | 19:43 | |
| *** LinuxJedi has joined #openstack-infra | 19:43 | |
| *** jbryce has quit IRC | 19:43 | |
| *** roaet_ has quit IRC | 19:43 | |
| fungi | sdague: jogo: the reqs update proposer looks like it's already proposed that | 19:44 |
| *** StevenK_ has quit IRC | 19:44 | |
| *** ryanpetrello has quit IRC | 19:44 | |
| *** ryanpetrello has joined #openstack-infra | 19:44 | |
| *** dolphm has quit IRC | 19:44 | |
| mriedem1 | https://review.openstack.org/#/c/117951/ | 19:44 |
| jogo | fungi: I would rather promote the single blacklist change | 19:44 |
| mriedem1 | it did ^ | 19:44 |
| *** StevenK_ has joined #openstack-infra | 19:44 | |
| sdague | fungi: ok, well can we just take jogo's? | 19:44 |
| *** dolphm has joined #openstack-infra | 19:44 | |
| *** dims has quit IRC | 19:44 | |
| jogo | as the other one is big and there are other risks in that | 19:44 |
| *** ryanpetrello is now known as ryanpetre | 19:44 | |
| fungi | sdague: are you sure my -1/inline comment isn't actually an issue? | 19:45 |
| *** vigneshvar has joined #openstack-infra | 19:45 | |
| dansmith | fungi: alreasdy fixed | 19:45 |
| *** amuller has quit IRC | 19:45 | |
| clarkb | fungi: what -1? | 19:45 |
| fungi | oh, jogo uploaded another | 19:45 |
| clarkb | oh ps1 | 19:45 |
| *** Dafna has quit IRC | 19:45 | |
| fungi | sure, looks fine | 19:45 |
| *** gondoi has quit IRC | 19:46 | |
| *** d34dh0r53 has quit IRC | 19:47 | |
| dansmith | hey | 19:48 |
| dansmith | check it out | 19:48 |
| dansmith | that patch that was at the back is now at the front | 19:48 |
| *** gondoi has joined #openstack-infra | 19:48 | |
| clarkb | fungi is a ninja | 19:48 |
| *** lttrl has quit IRC | 19:48 | |
| dansmith | I would give the left one of one of my body parts for that power | 19:48 |
| *** mrmartin has quit IRC | 19:49 | |
| flaper87 | clarkb: fungi Is there something I can help with for the pypy thing? | 19:49 |
| *** StevenK_ has quit IRC | 19:49 | |
| clarkb | flaper87: http://paste.openstack.org/show/110283/ is a current workaround that I will push as soon as fungi has a bug ready for me | 19:49 |
| clarkb | flaper87: initial thoughts on how viable that is for you guys would be good | 19:49 |
| *** StevenK_ has joined #openstack-infra | 19:50 | |
| *** d34dh0r53 has joined #openstack-infra | 19:51 | |
| flaper87 | clarkb: that looks good, pls add me to the reviewers list. I'll make sure it goes through quickly | 19:51 |
| *** mgagne has joined #openstack-infra | 19:51 | |
| *** jmeridth has joined #openstack-infra | 19:51 | |
| *** jroll has joined #openstack-infra | 19:51 | |
| *** annegentle has joined #openstack-infra | 19:52 | |
| fungi | flaper87: yeah, zaqar is the only project with voting pypy jobs which work on precise with the backport pypy ppa but not the pypy in trusty | 19:52 |
| *** berendt has joined #openstack-infra | 19:54 | |
| *** mestery has quit IRC | 19:54 | |
| *** mestery has joined #openstack-infra | 19:54 | |
| fungi | flaper87: if you have access to an ubuntu trusty machine can narrow down what is triggering http://paste.openstack.org/show/110296/ that would be really helpful as well | 19:54 |
| mika | jeblair: fungi: with download-commands plugin it finally works for me :) http://paste.openstack.org/show/110318/ documents what i did, hth | 19:54 |
| flaper87 | fungi: I was digging into that the other day but I didn't have enough time to debug it properly | 19:55 |
| flaper87 | fungi: I have access to a trusty box | 19:55 |
| flaper87 | I'll dig into it tomorrow morning | 19:55 |
| *** rlandy has quit IRC | 19:55 | |
| flaper87 | fungi: I saw some pypy jobs in the gate failing because of sqlite | 19:56 |
| flaper87 | but AFAICT, it's not deterministic | 19:56 |
| flaper87 | fungi: su, is that trusty with pypy from the ppa repo? | 19:56 |
| flaper87 | s/su/so/ | 19:56 |
| clarkb | flaper87: no, its from the main repos | 19:57 |
| flaper87 | ok | 19:57 |
| fungi | flaper87: trusty with the pypy 2.2.1 which comes as part of trusty | 19:57 |
| flaper87 | roger | 19:57 |
| fungi | flaper87: cool, i'm updating https://bitbucket.org/pypy/pypy/issue/1694/ with what i have so far and opening a bug against https://bugs.launchpad.net/ubuntu/+source/pypy | 19:57 |
| *** datsun180b has quit IRC | 20:00 | |
| fungi | hah! when did bitbucket start letting you openid authenticate from github? that seems new | 20:00 |
| clarkb | :( I created a bitbucket account that isn't fair | 20:00 |
| *** weshay has quit IRC | 20:01 | |
| fungi | i already had a bb account anyway, was just surprised to see the gh logo on the bb login page | 20:01 |
| fungi | mildly disconcerting, not that i have any love for either of them | 20:01 |
| *** dimtruck is now known as zz_dimtruck | 20:02 | |
| *** lbragstad has joined #openstack-infra | 20:02 | |
| *** colettecello is now known as gothicmindfood | 20:03 | |
| *** dims_ has joined #openstack-infra | 20:03 | |
| *** rhsu has joined #openstack-infra | 20:03 | |
| *** ramashri has quit IRC | 20:04 | |
| *** che-arne has joined #openstack-infra | 20:06 | |
| *** eharney has quit IRC | 20:07 | |
| *** eharney_ has joined #openstack-infra | 20:07 | |
| *** Sincler has quit IRC | 20:08 | |
| *** shashankhegde has joined #openstack-infra | 20:09 | |
| *** markmcclain has joined #openstack-infra | 20:10 | |
| openstackgerrit | Ian Cordasco proposed a change to openstack/requirements: Update requests to have a hard lower limit of 2.4.1 https://review.openstack.org/120898 | 20:12 |
| *** mbacchi has quit IRC | 20:12 | |
| anteaya | EmilienM: you had replied in a comment that you would remove line 6 - 16, yet those lines are still there: https://review.openstack.org/#/c/120285/4/modules/openstack_project/files/gerrit/acls/stackforge/tripleo-puppet-elements.config | 20:12 |
| *** yfried has quit IRC | 20:13 | |
| Ajaeger | fungi: I'll sign off now and would appreciate another review approval of https://review.openstack.org/#/c/120800/, please. We have 5 translation jobs in the gate now that won't make it otherwise... | 20:13 |
| *** Ajaeger has quit IRC | 20:14 | |
| *** weshay has joined #openstack-infra | 20:15 | |
| EmilienM | anteaya: looking | 20:15 |
| *** kgiusti has quit IRC | 20:16 | |
| anteaya | EmilienM: thanks | 20:16 |
| *** masayukig has quit IRC | 20:17 | |
| *** dims__ has joined #openstack-infra | 20:17 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Introduce tripleo-puppet-elements into Stackforge https://review.openstack.org/120285 | 20:17 |
| EmilienM | anteaya: I don't know how I can't miss it ^ | 20:17 |
| *** pballand has quit IRC | 20:17 | |
| anteaya | EmilienM: thanks | 20:17 |
| *** alaski has joined #openstack-infra | 20:19 | |
| *** e0ne has quit IRC | 20:19 | |
| anteaya | EmilienM: +1 | 20:19 |
| *** pballand has joined #openstack-infra | 20:19 | |
| *** dims_ has quit IRC | 20:20 | |
| *** berendt has quit IRC | 20:21 | |
| *** masayukig has joined #openstack-infra | 20:21 | |
| *** baoli has quit IRC | 20:22 | |
| *** dims__ has quit IRC | 20:22 | |
| *** baoli has joined #openstack-infra | 20:22 | |
| *** HeOS has joined #openstack-infra | 20:22 | |
| *** baoli has quit IRC | 20:23 | |
| *** flaper87 is now known as flaper87|afk | 20:24 | |
| *** baoli has joined #openstack-infra | 20:24 | |
| *** zz_dimtruck is now known as dimtruck | 20:25 | |
| *** Sincler has joined #openstack-infra | 20:26 | |
| clarkb | fungi: https://bitbucket.org/pypy/pypy/issue/1669 may be related? | 20:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Do not generate new translation proposal if one is in the gate https://review.openstack.org/120800 | 20:28 |
| *** lttrl has joined #openstack-infra | 20:28 | |
| *** sdake_ has quit IRC | 20:28 | |
| fungi | clarkb: aha | 20:28 |
| *** sdake_ has joined #openstack-infra | 20:28 | |
| *** emagana has joined #openstack-infra | 20:30 | |
| *** mfer has quit IRC | 20:30 | |
| fungi | ll_dict_getitem__dicttablePtr_Signed_11 instead of 10, but otherwise looks likely | 20:30 |
| *** dprince has quit IRC | 20:30 | |
| *** maxbit_ has quit IRC | 20:30 | |
| fungi | clarkb: ahh, yes, that's the bug which led to the suspected fix for the other more vague one | 20:31 |
| *** mmaglana has joined #openstack-infra | 20:31 | |
| jeblair | mika: wow, glad to hear it, and thanks for that peek into our future :) | 20:32 |
| mika | jeblair: :) | 20:32 |
| *** oanufriev has quit IRC | 20:34 | |
| *** emagana has quit IRC | 20:34 | |
| *** marcoemorais has quit IRC | 20:36 | |
| *** andreykurilin has joined #openstack-infra | 20:37 | |
| *** marcoemorais has joined #openstack-infra | 20:37 | |
| * nibalizer returns | 20:37 | |
| *** mfer has joined #openstack-infra | 20:37 | |
| *** oanufriev has joined #openstack-infra | 20:37 | |
| *** marcoemorais has quit IRC | 20:39 | |
| jeblair | mordred, fungi, clarkb, SergeyLukjanov: i've copied the hieradata git repo over to the new puppetmaster, so either don't change secrets (preferred!) or change them in both places. | 20:39 |
| clarkb | jeblair: rgr | 20:39 |
| fungi | jeblair: thanks for the update | 20:39 |
| mordred | jeblair: ++ | 20:39 |
| fungi | clarkb: bug 1368418 | 20:39 |
| uvirtbot | Launchpad bug 1368418 in pypy "Fatal RPython error: KeyError/Crash in JIT!" [Undecided,New] https://launchpad.net/bugs/1368418 | 20:39 |
| *** radez is now known as radez_g0n3 | 20:40 | |
| *** emagana has joined #openstack-infra | 20:40 | |
| *** marcoemorais has joined #openstack-infra | 20:40 | |
| fungi | clarkb: also bug 1367907 is marked as released | 20:40 |
| clarkb | fungi: flaper87|afk https://review.openstack.org/120906 | 20:40 |
| uvirtbot | Launchpad bug 1367907 in python "Segfault in gc with cyclic trash" [Unknown,Fix released] https://launchpad.net/bugs/1367907 | 20:40 |
| *** marcoemorais has quit IRC | 20:40 | |
| clarkb | fungi: only on the upstream python task | 20:41 |
| fungi | ahh | 20:41 |
| clarkb | fungi: it talks to bugs.python.org and synced state | 20:41 |
| fungi | got it | 20:41 |
| reed | forget python, it's all java from now on | 20:41 |
| * reed trolling | 20:41 | |
| * reed asks for food | 20:42 | |
| fungi | i need to go find food as well | 20:42 |
| fungi | fuel for more hacking | 20:42 |
| clarkb | fungi: with that we can switch to trusty for pypy on all projects yes? | 20:42 |
| clarkb | so probably worth getting in puppet change to rebuild images asap | 20:42 |
| fungi | clarkb: yeah, i'll whip up the change to switch the node label some time tonight so it's ready to approve after the bare-trusty nodes rebuild and your zaqar patch lands | 20:43 |
| fungi | also, i cribbed off your (excellent? terrible?) sru-oriented bug report. thanks for the assist | 20:44 |
| clarkb | I am sure it is terrible and no problem :) | 20:45 |
| *** e0ne has joined #openstack-infra | 20:45 | |
| *** marcoemorais1 has joined #openstack-infra | 20:45 | |
| *** e0ne has quit IRC | 20:45 | |
| jeblair | on puppetmaster.o.o, puppet agent --server puppetmaster.openstack.org --test works | 20:47 |
| jeblair | runs clean | 20:47 |
| clarkb | jeblair: nice | 20:47 |
| daya_k | fungi : i got the mysql issue sorted out, the upgrade to 5.5 is very tricky, lots of notes on the web about it. i tried it, also tried re-install but finally what worked was complete remove, also rm /var/lib/mysql/* because the old passwd was getting stored somewhere in there | 20:47 |
| jeblair | not entirely sure it's doing anything | 20:47 |
| nibalizer | jeblair: wooooo | 20:48 |
| clarkb | jeblair: so we can edit /etc/ansible/hostfile with puppetmaster.o.o as being the first node on puppet3 | 20:48 |
| *** rainya_ has joined #openstack-infra | 20:48 | |
| clarkb | jeblair: don't forget to remove it from the hostfile on old puppetmaster if necessary | 20:48 |
| nibalizer | thats excellent news! | 20:48 |
| *** rainya has quit IRC | 20:48 | |
| *** dimtruck is now known as zz_dimtruck | 20:48 | |
| jeblair | yeah, i think it's not actually doing enough work | 20:49 |
| *** eharney_ is now known as eharney | 20:49 | |
| jeblair | i ran with --debug and am not seeing anything that actually looks like anything more than boilerplate puppet | 20:49 |
| nibalizer | you could change clark's shell and see if puppet puts it back | 20:51 |
| clarkb | nibalizer: way to pick on me | 20:52 |
| clarkb | :P | 20:52 |
| fungi | daya_k: excellent news. i'll try to remember in case i run into the same somewhere | 20:52 |
| nibalizer | or inspect /var/lib/puppet/state , i think there is a catalog object in yaml in there somewhere | 20:52 |
| nibalizer | you could also run puppet agent --configprint <certname> to see how this master is identfying itself | 20:52 |
| clarkb | but ya changing something arbitrary like that and see if puppet fixes seems like a reasonable thing to do | 20:52 |
| daya_k | yep. when i updated devstack-gate. the passwd for mysql changed but some where in /var/lib/mysql, some module had the old passwd | 20:53 |
| * nibalizer is thinking maybe we don't have our /etc/puppet/environments/production/environment.conf file correct | 20:53 | |
| fungi | okay, heading out to grab something to eat... back soon-ish | 20:53 |
| *** ramashri has joined #openstack-infra | 20:53 | |
| jeblair | yeah, i moved clarkb's authorized keys and it did not put them back | 20:53 |
| *** rainya_ has quit IRC | 20:54 | |
| clarkb | :( | 20:54 |
| jeblair | clarkb: i'm nicer, i put it back | 20:54 |
| clarkb | jeblair: let me know if you put them back and want more eyeballs | 20:54 |
| clarkb | oh good :) | 20:54 |
| *** ramashri has quit IRC | 20:54 | |
| jeblair | nibalizer: /opt/config/production/manifest/site.pp: No such file or directory | 20:55 |
| jeblair | yeah, i think the 's' is missing from manifests | 20:55 |
| nibalizer | jeblair: so yea my spider sense says its environment.conf | 20:55 |
| *** ramashri has joined #openstack-infra | 20:55 | |
| nibalizer | jeblair: hahahahah my bad, let me git you a new review | 20:55 |
| *** stevebaker has joined #openstack-infra | 20:55 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add production environment to puppet https://review.openstack.org/120872 | 20:56 |
| *** dannywilson has quit IRC | 20:56 | |
| jeblair | huh, seems to be working now, even tho the previous run did not | 20:57 |
| *** cdent has quit IRC | 20:57 | |
| clarkb | jeblair: even weirde | 20:58 |
| clarkb | *weirder | 20:58 |
| nibalizer | maybe some weird caching stuff | 20:58 |
| nibalizer | so there is this: https://docs.puppetlabs.com/puppet/latest/reference/environments_configuring.html#environmenttimeout | 20:58 |
| jeblair | nibalizer: could be it | 20:58 |
| nibalizer | it says we cant set it to 0 to turn off any caching, which we might want to do | 20:58 |
| nibalizer | er can set it to 0 | 20:59 |
| *** zz_dimtruck is now known as dimtruck | 20:59 | |
| jeblair | it seems to be reliably creating fungi's authorized_key file each time i rm and run it | 20:59 |
| clarkb | maybe it just likes fungi more | 20:59 |
| *** markmcclain has quit IRC | 20:59 | |
| *** mestery has quit IRC | 21:00 | |
| jeblair | nibalizer: i think we can probably leave the timeout, but should keep in in mind in case we see other probs | 21:00 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add production environment to puppet https://review.openstack.org/120872 | 21:00 |
| *** enikanorov_ has quit IRC | 21:00 | |
| *** enikanorov_ has joined #openstack-infra | 21:00 | |
| nibalizer | jeblair: okay, i'll un-add it | 21:00 |
| jeblair | haha | 21:00 |
| jeblair | i don't have a strong pref :) | 21:00 |
| *** mestery has joined #openstack-infra | 21:00 | |
| *** dannywilson has joined #openstack-infra | 21:00 | |
| clarkb | caching for the number of nodes we have seems silly | 21:01 |
| nibalizer | jeblair: me either | 21:01 |
| jeblair | i just think with our 15 minute cycles, we'll probably be okay. but yeah, also not like we'll get a lot from it. | 21:01 |
| jeblair | okay, so i think we're in business | 21:01 |
| clarkb | woot | 21:01 |
| jeblair | clarkb: yeah, i'll move puppetmaster over | 21:01 |
| nibalizer | its more about caching the metadata anyways i think, i.e. not the code but the directory containing the modulepath and stuff | 21:02 |
| nibalizer | i think the point is to prevent the master from stating all kinds of files constantly | 21:02 |
| *** dannywilson has quit IRC | 21:02 | |
| nibalizer | but at our scale, no problem leaving it default | 21:02 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add production environment to puppet https://review.openstack.org/120872 | 21:02 |
| *** dannywilson has joined #openstack-infra | 21:03 | |
| markvan | clarkb: our cookbook gates are still not working, when ya get a chance: https://review.openstack.org/#/c/120116/ | 21:03 |
| jeblair | hrm, the updatepuppetmaster cron is wrong | 21:03 |
| jeblair | */15 * * * * sleep $((RANDOM\%600)) && cd /opt/config/production && git fetch -q && git reset -q --hard @{u} && ./install_modules.sh && touch manifests/site.pp | 21:03 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Clean up bashate failures https://review.openstack.org/118944 | 21:03 |
| jeblair | it's that, instead of | 21:03 |
| jeblair | */15 * * * * bash /opt/config/production/run_all.sh | 21:03 |
| clarkb | jeblair: the path to the repo is wrong? | 21:03 |
| clarkb | oh I see | 21:03 |
| *** mwagner_lap has quit IRC | 21:04 | |
| clarkb | markvan: hrm I am not a fan of curl | bash | 21:04 |
| nibalizer | jeblair: is it the update_slave flag in puppetmaster.pp | 21:04 |
| nibalizer | ? | 21:04 |
| clarkb | markvan: I imagine rvm is too old on precise ? | 21:04 |
| clarkb | markvan: can we install rvm via gem instead? | 21:05 |
| jeblair | nibalizer: yeah, kinda looks like it's true for some reason even tho is set to false in site.pp | 21:05 |
| jeblair | nibalizer: oooh | 21:06 |
| jeblair | nibalizer: i think we want it to be set to true | 21:06 |
| jeblair | but it's set to fals | 21:06 |
| jeblair | what is false supposed to be for? | 21:06 |
| markvan | clarkb: no I don't think it's in gem form. yeah, we thought using rvm would be a better approach, but I'm open to suggestions. If you look at patch set #2 that's what I first proposed, but seems like more unique steps then using rvm. | 21:07 |
| *** dannywilson has quit IRC | 21:07 | |
| nibalizer | jeblair: i dont... know | 21:08 |
| nibalizer | jeblair: im looking at git blame and that whole section looks like its me, and i have no memory of why I've done this | 21:08 |
| *** annegentle has quit IRC | 21:09 | |
| *** emagana has quit IRC | 21:09 | |
| *** _nadya_ has quit IRC | 21:09 | |
| jeblair | nibalizer: best guess is we earlier thought that we might need to update the new puppetmaster some other way | 21:09 |
| jeblair | nibalizer: i'm pulling it out | 21:09 |
| nibalizer | okay | 21:09 |
| *** dannywilson has joined #openstack-infra | 21:09 | |
| *** emagana has joined #openstack-infra | 21:09 | |
| nibalizer | im thinking maybe it was in the days of puppet fired from ssh and i only wanted the ci-puppetmaster to ssh out not the new puppetmaster? | 21:09 |
| jeblair | nibalizer: ah, the old days ;) | 21:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove update_slave from puppetmaster https://review.openstack.org/120911 | 21:10 |
| clarkb | markvan: I just raelly really don't like curl | bash | 21:10 |
| clarkb | markvan: its like the worst thing to do ever :) | 21:10 |
| jeblair | nibalizer, clarkb: https://review.openstack.org/120911 | 21:10 |
| jeblair | clarkb: while you're at it: https://review.openstack.org/#/c/120872/ | 21:11 |
| markvan | clarkb: Being new to some of this...why is that? | 21:11 |
| clarkb | jeblair: +2 | 21:11 |
| *** vigneshvar has quit IRC | 21:11 | |
| *** aysyd has quit IRC | 21:12 | |
| markvan | clarkb: I have no problem going back to the #2 where we are using update-alternatives to get the job done. | 21:12 |
| clarkb | markvan: for a couple reasons. the first is security related. you have no idea if the bash that is coming from the internets is safe. But also because it is error prone. If you don't get the whole file bash will stop running halfway through which may also be dangerous | 21:12 |
| nibalizer | jeblair: lgtm | 21:12 |
| lifeless | sdague: 20% is still extraordinarily high | 21:12 |
| lifeless | sdague: I suggest getting in touch with wgrant in #launchpad about that | 21:12 |
| clarkb | markvan: I would be happy with a wget/curl to file with a sha1 check if possible then run the script | 21:13 |
| clarkb | markvan: that way you have some assurance you got th file you expected and it is complete | 21:13 |
| *** lttrl has quit IRC | 21:13 | |
| *** dustins has quit IRC | 21:13 | |
| nibalizer | hrm, i wonder how often the rvm launcher script changes | 21:13 |
| *** emagana has quit IRC | 21:14 | |
| *** sbalukoff has joined #openstack-infra | 21:14 | |
| markvan | clarkb: yup, your points are valid. Ok, I can look into that. Maybe in the mean time since we are getting back logged on cookbook patches, I'll put the #2 back since it was ready. | 21:14 |
| clarkb | markvan: that works for me too | 21:14 |
| nibalizer | markvan: clarkb rvm launcher script seems to be at least constant for every download, they may change it wihout telling us but otherwise the wget then sha1 is workable | 21:15 |
| jeblair | clarkb: https://review.openstack.org/#/c/120872/ ? | 21:15 |
| markvan | clarkb: thx for the feedback, I update that patch updated to hopefully open our gates again, then look into the wget/sha1 approach (thx nibalizer) | 21:16 |
| clarkb | sorry missed that highlight looking now | 21:16 |
| *** sdake_ has quit IRC | 21:16 | |
| nibalizer | clarkb: re rvm, what about wgeting twice then ensuring they are the same file, then running the second one, that way we dont cary a sha1sum around in our repo that needs to be updated frequently? | 21:16 |
| *** amcrn_ has joined #openstack-infra | 21:16 | |
| alexpilotti | jeblair: hi, I have question on the review process for a new StackForge project: https://review.openstack.org/#/c/119909/ | 21:16 |
| clarkb | nibalizer: jeblair: in 120872 what is $basemodulepath:modules? | 21:16 |
| clarkb | isn't : a path separator? | 21:17 |
| clarkb | nibalizer: jeblair: wondering where modules is rooted or if it is special? | 21:17 |
| jeblair | alexpilotti: ask away, any time :) | 21:17 |
| nibalizer | clarkb: modules is rooted on the dir that environment.conf is in | 21:17 |
| clarkb | nibalizer: thanks | 21:18 |
| nibalizer | so it enables you to put a modules dir with modules in it in the environment(production) directory | 21:18 |
| clarkb | nibalizer: makes sense thanks | 21:18 |
| clarkb | nibalizer: shoulod we set mode, owner and group on that file? | 21:18 |
| clarkb | otherwise I am +2 | 21:19 |
| alexpilotti | jeblair: is there any particular difference in review process compared to, say, a regular Nova, Neutron, etc patch? | 21:19 |
| nibalizer | we're not using it, so maybe we should remove it, i kindof liked the idea that a rooter could use this third place to keep modules to maybe test a module upgrade before we wanted to commit to it | 21:19 |
| clarkb | nibalizer: no I think having it is good. it allows us to move to non deprecated things | 21:19 |
| openstackgerrit | Mark Vanderwiel proposed a change to openstack-infra/config: Fix ruby-191-prep to uninstall 1.8 and install bundler https://review.openstack.org/120116 | 21:19 |
| clarkb | nibalizer: just asking questions about the patch to make it happen | 21:19 |
| nibalizer | clarkb: sure | 21:19 |
| jeblair | alexpilotti: nope | 21:19 |
| alexpilotti | jeblair: e.g. beside the formal puppet config files, layout etc, does the review also involve the project itself? | 21:20 |
| nibalizer | as i said before, i think longer term we'll move the git clone out of /opt and into /etc (since we need to for hiera anyways) | 21:20 |
| sdague | lifeless: honestly, you probably don't want me in the launchpad channel, unless we are trying to break a kick/ban record :) | 21:20 |
| *** amcrn has quit IRC | 21:20 | |
| *** annegentle has joined #openstack-infra | 21:20 | |
| nibalizer | clarkb: im ambivalent on setting owner group mode on that file | 21:20 |
| jeblair | alexpilotti: you may have noted that anteaya checks some technical aspects about the project, like whether it has an excess number of branches or tags that may be undesirable in the import | 21:20 |
| nibalizer | if you want me to i'll update it | 21:20 |
| clarkb | nibalizer: ok. does puppet3 do what puppet2 does and set it to the same mode as is in the puppet master modulkes? | 21:21 |
| clarkb | nibalizer: I know we have been bitten by this before without being explicit | 21:21 |
| alexpilotti | jeblair: yep, I noticed anteaya’s review (tx btw!) | 21:21 |
| jeblair | alexpilotti: but other than that, we don't review anything about stackforge projects | 21:21 |
| jeblair | alexpilotti: it's a mostly self-service system and we assume you know what you're doing :) | 21:21 |
| nibalizer | clarkb: hrm yea better safe than sorry, i'll do root:root 755/644 | 21:22 |
| alexpilotti | jeblair: was just wondering if there’s anything else preventing it from being merged so that we can move the dev process from github to gerrit :-) | 21:22 |
| clarkb | nibalizer: great thank you | 21:22 |
| jeblair | alexpilotti: a shortage of infra reviewers. :) but we do get to everthing eventually; may be a few more days. | 21:22 |
| *** rainya has joined #openstack-infra | 21:22 | |
| alexpilotti | jeblair: as we’re kind of in a limbo now waiting to post major features :-) | 21:22 |
| jeblair | alexpilotti: it's _just_ like nova or neutron | 21:23 |
| alexpilotti | jeblair: ok tx, just wanted to make sure there were no specific blockers or differences! | 21:23 |
| openstackgerrit | Mark Vanderwiel proposed a change to openstack-infra/config: Fix ruby-191-prep to install bundler https://review.openstack.org/120116 | 21:23 |
| *** annegentle has quit IRC | 21:23 | |
| *** ChuckC_ has joined #openstack-infra | 21:24 | |
| *** annegentle has joined #openstack-infra | 21:24 | |
| markvan | clarkb: ^^ | 21:24 |
| *** sdake has quit IRC | 21:24 | |
| anteaya | alexpilotti: if there were blockers to the patch being merged, major ones anyone, I would have let you know in my review | 21:26 |
| *** emagana has joined #openstack-infra | 21:26 | |
| clarkb | markvan: lgtm thanks | 21:26 |
| *** adam_g has joined #openstack-infra | 21:26 | |
| *** rfolco has quit IRC | 21:26 | |
| markvan | clarkb: ty | 21:26 |
| *** adam_g has quit IRC | 21:27 | |
| *** adam_g has joined #openstack-infra | 21:27 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add production environment to puppet https://review.openstack.org/120872 | 21:27 |
| nibalizer | clarkb: jeblair ^ | 21:27 |
| clarkb | nibalizer: perfect +2 | 21:27 |
| openstackgerrit | Stephen Balukoff proposed a change to openstack-infra/config: Enable use of StoryBoard with Octavia https://review.openstack.org/120913 | 21:27 |
| alexpilotti | anteaya: that’s great tx! | 21:27 |
| anteaya | np | 21:27 |
| sbalukoff | Hi folks! I want to play with StoryBoard (in the context of Octavia) | 21:28 |
| sbalukoff | Hence the above review. :D | 21:28 |
| *** Sincler has quit IRC | 21:28 | |
| *** dannywilson has quit IRC | 21:28 | |
| *** dannywilson has joined #openstack-infra | 21:29 | |
| *** dannywilson has quit IRC | 21:30 | |
| jeblair | sbalukoff: thanks for the interest, but we're actually a bit behind on our original goals of using storyboard for openstack-infra and openstack. would you mind waiting until after openstack-infra has managed to use it for a while? | 21:30 |
| *** dkranz has quit IRC | 21:30 | |
| *** emagana has quit IRC | 21:30 | |
| *** dannywilson has joined #openstack-infra | 21:30 | |
| *** emagana has joined #openstack-infra | 21:31 | |
| sbalukoff | jeblair: Would I be stepping on your toes there? (Is using it and providing feedback on bugfixes / feature requests useful at this stage?) | 21:31 |
| *** marcoemorais1 has quit IRC | 21:31 | |
| *** marcoemorais has joined #openstack-infra | 21:32 | |
| *** dannywilson has quit IRC | 21:32 | |
| jeblair | sbalukoff: we're not in a position to make any guarantees about availability, or reliability. i'd rather not be in the position of having other people depend on it before we're even using it ourselves. | 21:32 |
| *** dannywilson has joined #openstack-infra | 21:32 | |
| sbalukoff | Ok, that I knew before making the above request. For the time being, my plan was to use launchpad and StoryBoard in parallel (even though I rather dislike launchpad with a burning passion.) | 21:33 |
| *** marcoemorais1 has joined #openstack-infra | 21:34 | |
| *** marcoemorais1 has quit IRC | 21:34 | |
| sbalukoff | So, the service being unreliable and unavailable at times should be OK, as I will be maintaining launchpad as the "primary" project management system. | 21:35 |
| *** marcoemorais1 has joined #openstack-infra | 21:35 | |
| jeblair | sbalukoff: i hope we'll be in a better position soon, sorry. | 21:35 |
| *** marcoemorais1 has quit IRC | 21:35 | |
| *** emagana has quit IRC | 21:35 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Added documentation for StoryBoard https://review.openstack.org/108460 | 21:35 |
| *** shashankhegde has quit IRC | 21:36 | |
| *** marcoemorais1 has joined #openstack-infra | 21:36 | |
| *** marcoemorais1 has quit IRC | 21:36 | |
| *** marcoemorais1 has joined #openstack-infra | 21:37 | |
| *** dannywilson has quit IRC | 21:37 | |
| *** marcoemorais has quit IRC | 21:37 | |
| *** marcoemorais1 has quit IRC | 21:37 | |
| *** marcoemorais has joined #openstack-infra | 21:37 | |
| *** marcoemorais has quit IRC | 21:37 | |
| *** yamamoto has quit IRC | 21:38 | |
| *** marcoemorais has joined #openstack-infra | 21:38 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Tags db api https://review.openstack.org/113889 | 21:39 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Refactor WSME models https://review.openstack.org/114225 | 21:41 |
| *** hdd has quit IRC | 21:42 | |
| *** dannywilson has joined #openstack-infra | 21:42 | |
| *** resker has quit IRC | 21:42 | |
| asselin | any cores want to +2/+W this simple fix done by clarkb? currently cherry picking it many times already...https://review.openstack.org/#/c/120639/ | 21:42 |
| *** hdd has joined #openstack-infra | 21:42 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove query for bug 1347692 https://review.openstack.org/120916 | 21:42 |
| uvirtbot | Launchpad bug 1347692 in glance "gate failure with tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image (dup-of: 1345955)" [Undecided,Confirmed] https://launchpad.net/bugs/1347692 | 21:42 |
| uvirtbot | Launchpad bug 1345955 in horizon "KeyError: <SassRule <RuleAncestry ()>, 0 props> Failure" [Critical,Confirmed] https://launchpad.net/bugs/1345955 | 21:42 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Subscription events now appear in UI https://review.openstack.org/114404 | 21:42 |
| *** esker has joined #openstack-infra | 21:42 | |
| clarkb | asselin: I take it that means it works? | 21:43 |
| asselin | yes it does | 21:43 |
| clarkb | asselin: I had intended on gettingaround to testing it but kept getting distracted | 21:43 |
| asselin | very well | 21:43 |
| jeblair | lgtm+3 | 21:43 |
| jeblair | asselin, clarkb: thanks | 21:43 |
| anteaya | clarkb: just deleting the thing is easier than polling the db first | 21:43 |
| anteaya | yay | 21:43 |
| asselin | thanks! | 21:44 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove update_slave from puppetmaster https://review.openstack.org/120911 | 21:44 |
| *** mdenny has joined #openstack-infra | 21:45 | |
| *** esker has quit IRC | 21:47 | |
| *** StevenK_ is now known as StevenK | 21:47 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Delete correct item in image delete command. https://review.openstack.org/120639 | 21:49 |
| jeblair | okay, hopefully we can run puppet on puppetmaster now | 21:49 |
| anteaya | yay | 21:50 |
| *** signed8bit has quit IRC | 21:50 | |
| *** atiwari has quit IRC | 21:50 | |
| jeblair | yep that worked | 21:50 |
| clarkb | jeblair: great news | 21:50 |
| jeblair | oh, i have to accept host keys | 21:51 |
| jeblair | maybe i should copy root's known_hosts | 21:51 |
| clarkb | hrm I guess we don't have those puppetted | 21:51 |
| *** rainya has quit IRC | 21:51 | |
| clarkb | jeblair: ++ | 21:51 |
| *** bhuvan_ has joined #openstack-infra | 21:51 | |
| *** jlk has joined #openstack-infra | 21:51 | |
| *** jlk has joined #openstack-infra | 21:51 | |
| *** rainya has joined #openstack-infra | 21:52 | |
| clarkb | jeblair: do you want to split up the hosts that need to be tested? partitioning the work may help us get it done quicker. though may also lead to merge conflict problems | 21:52 |
| jeblair | clarkb: yeah, let's work through this a little bit more single threaded first tho | 21:53 |
| clarkb | sounds good | 21:53 |
| *** mfer has quit IRC | 21:53 | |
| *** bookwar has joined #openstack-infra | 21:53 | |
| *** bhuvan has quit IRC | 21:54 | |
| jeblair | mordred: do we have a way of getting the ssh host key in known hosts when we add a new one? | 21:54 |
| *** annegentle has quit IRC | 21:54 | |
| *** doug-fish has quit IRC | 21:54 | |
| jeblair | is that why we ended up not having puppet run on some new hosts recently? | 21:54 |
| clarkb | jeblair: it is why | 21:54 |
| *** doug-fish has joined #openstack-infra | 21:54 | |
| * nibalizer excited to see the --noop report | 21:54 | |
| clarkb | jeblair: we could completely puppet host keys. Seems like high overhead but it is doable | 21:54 |
| clarkb | jeblair: so that when we spin up a host it installs a host key from hiera and that key can be added to known_hosts on master at the same time | 21:55 |
| nibalizer | you can also have puppet 'export' the host key fingerprint then place them on the other hosts | 21:55 |
| *** mfer has joined #openstack-infra | 21:55 | |
| clarkb | nibalizer: I was just going to say that requires puppetdb... we have one of those ow :) | 21:55 |
| *** mfer has quit IRC | 21:55 | |
| nibalizer | so the sshd daemon generates the key, but the fingerprint gets put in puppetdb, then puppet pulls it out of puppetdb and puts in in known_hosts | 21:55 |
| jeblair | and apparently the private key is not puppeted either? | 21:56 |
| *** bhuvan_ has quit IRC | 21:56 | |
| clarkb | jeblair: that part should be puppeted /me looks | 21:56 |
| *** bhuvan has joined #openstack-infra | 21:56 | |
| *** signed8bit has joined #openstack-infra | 21:56 | |
| jeblair | clarkb: you are right | 21:56 |
| mordred | jeblair: I can add that to the launch script rework | 21:56 |
| *** dmsimard is now known as dmsimard_away | 21:57 | |
| jeblair | oh | 21:57 |
| *** emagana has joined #openstack-infra | 21:57 | |
| jeblair | from="ci-puppetmaster.openstack.org" ssh-rsa | 21:57 |
| jeblair | oops | 21:57 |
| nibalizer | heh | 21:57 |
| clarkb | jeblair: yup it is in base.pp | 21:57 |
| jeblair | clarkb: so dup that? | 21:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Add production environment to puppet https://review.openstack.org/120872 | 21:57 |
| nibalizer | that can be set to from="host1,host2" right? | 21:57 |
| *** emagana has quit IRC | 21:58 | |
| clarkb | jeblair: you shouldn't need to right? it should already apply that key to authorized_keys file on all nodes? | 21:58 |
| clarkb | jeblair: maybe you need the privkey? | 21:58 |
| clarkb | oh | 21:58 |
| *** emagana has joined #openstack-infra | 21:58 | |
| clarkb | derp | 21:58 |
| clarkb | ya I think privkey is not puppeted. the public key in authorized_keys is | 21:58 |
| jeblair | clarkb: it is puppeted. i think i just told you that :) | 21:58 |
| jeblair | 21:56 < clarkb> jeblair: that part should be puppeted /me looks | 21:59 |
| jeblair | 21:56 < jeblair> clarkb: you are right | 21:59 |
| jeblair | clarkb: the actual problem is the host restriction | 21:59 |
| clarkb | oh I need more caffeine | 21:59 |
| *** mriedem1 has quit IRC | 21:59 | |
| clarkb | jeblair: so it only installs on old master and not new? | 21:59 |
| jeblair | clarkb: no, the ssh key restriction in authorized keys says that connections using that key are only accepted when they originate from ci-puppetmaster | 22:00 |
| clarkb | ooohhhhh now I am on the same page with you | 22:00 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Allow ssh from the new puppetmaster https://review.openstack.org/120922 | 22:00 |
| nibalizer | jeblair: clarkb i think thats what you need ^^ | 22:01 |
| jeblair | nibalizer: i think so | 22:02 |
| *** baoli has quit IRC | 22:02 | |
| jeblair | clarkb: mordred: https://review.openstack.org/120922 | 22:02 |
| clarkb | does sshd respect the ','? | 22:03 |
| jeblair | clarkb: yep: A pattern-list is a comma-separated list of patterns. | 22:04 |
| clarkb | found the man page | 22:04 |
| jeblair | man 5 sshd_config | 22:04 |
| clarkb | thanks | 22:04 |
| mordred | jeblair: the options there are going in to the key, right? | 22:04 |
| jeblair | mordred: yep | 22:04 |
| clarkb | mordred: didn't approve so that you can review too | 22:04 |
| mordred | you need to increase the serial and add a "ensure -> absent" on the ond one | 22:05 |
| mordred | because puppet is going to see that key existing and not change it | 22:05 |
| *** Sincler has joined #openstack-infra | 22:05 | |
| clarkb | ugh | 22:05 |
| jeblair | despite the change in content? | 22:05 |
| mordred | I believe that's why I put the serials in there originally | 22:05 |
| mordred | it does NOT behave as you'd expect | 22:05 |
| clarkb | ya I think that is correct | 22:05 |
| clarkb | I remember when we went through that | 22:06 |
| jeblair | nibalizer: want to update? | 22:06 |
| *** weshay has quit IRC | 22:06 | |
| nibalizer | jeblair: sure | 22:07 |
| wenlock | jesusaurus im wondering on whitespaces for this review https://review.openstack.org/#/c/117892/12 are you saying that bashate requires 4 spaces for tab.... this is different than 2 spaces for tabs in puppet, right? shouldn't those be consistent? | 22:08 |
| jesusaurus | wenlock: right, bashate gives an error on 2-space indents | 22:08 |
| wenlock | jesusaurus is there a good reason for that? why not try to be consistent for peoples editors to support 1 standard for spaces for tabs? | 22:09 |
| jesusaurus | its not something i necessarily agree with about bashate, but i dont care too much about how many spaces are in an indent | 22:09 |
| *** mwagner_lap has joined #openstack-infra | 22:09 | |
| jesusaurus | wenlock: it was a devstack decision | 22:09 |
| *** dims_ has joined #openstack-infra | 22:10 | |
| nibalizer | okay updated | 22:10 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Allow ssh from the new puppetmaster https://review.openstack.org/120922 | 22:10 |
| * nibalizer is sad we have to do it this way | 22:10 | |
| clarkb | woot I think the logging.getLevelName thing will be fixed in python \o/ | 22:10 |
| clarkb | lifeless: ^ | 22:10 |
| wenlock | jesusaurus so it was a decision to have different space standards for languages, or was it just that bashate has no way to specify use 2 spaces? or am i missing something? | 22:10 |
| nibalizer | you would think puppet could handle this concept of updating ssh_keys | 22:10 |
| nibalizer | i've verified in puppet 3 that puppet can handle changing the options, not sure about 2.7 | 22:11 |
| jesusaurus | wenlock: it was a devstack decision that shell scripts will use 4-space tabs | 22:11 |
| wenlock | jesusaurus ah, hrm | 22:11 |
| clarkb | mordred: I haven't approved. figure I will let you look and do that | 22:11 |
| clarkb | wenlock: usually you can set formats based on language or file extension in your editor too | 22:12 |
| clarkb | I know my vimrc has a bunch of autocommands to do that | 22:12 |
| wenlock | jesusaurus, ok, sounds good.... must go find a way to tell my editor to use 4 spaces for .sh and 2 spaces for .pp | 22:12 |
| clarkb | though anymore everything is formated like python if possible >_> | 22:12 |
| jesusaurus | wenlock: and i think that the quick static analysis that bashate gives us is worth compromising on the number of spaces | 22:12 |
| mordred | looking | 22:12 |
| wenlock | clarkb do you have a nice solution for that on vim yet? | 22:12 |
| clarkb | wenlock: ya I can paste what I do | 22:12 |
| wenlock | clarkb thanks! | 22:13 |
| mordred | apprvd | 22:13 |
| lifeless | clarkb: :) | 22:13 |
| *** amcrn_ is now known as amcrn | 22:13 | |
| *** KanagarajM has quit IRC | 22:13 | |
| jeblair | i'm pretty sure if we don't like the spaces, we can disable it | 22:13 |
| jeblair | i mean, that specific check | 22:13 |
| wenlock | jesusaurus, oh yeah +1 for bashate, -1 for living with different coding standards on tabs .... must learn to live with it :D | 22:13 |
| arosen | sorry to bug you guys again. I have dsvm-tempest-full enabled right now but i'm getting this failure: http://paste.openstack.org/show/110356/ Any ideas? | 22:14 |
| *** andreykurilin has quit IRC | 22:14 | |
| *** dannywilson has quit IRC | 22:14 | |
| *** dims_ has quit IRC | 22:14 | |
| clarkb | wenlock: http://paste.openstack.org/show/110357/ note I keep meaning to kill the auto shebang additions | 22:14 |
| clarkb | wenlock: I have decided that is a bug | 22:14 |
| *** dannywilson has joined #openstack-infra | 22:14 | |
| wenlock | clarkb awesome, tyvm | 22:14 |
| clarkb | wenlock: and the mode settings | 22:14 |
| clarkb | arosen: gah we depend on pyyaml too | 22:15 |
| clarkb | but I think that one will compile properly without C libs? | 22:15 |
| *** ramashri has quit IRC | 22:15 | |
| clarkb | jeblair: should we add pyyaml to zuul requirements.txt? | 22:15 |
| clarkb | nevermind its in there | 22:16 |
| *** shashankhegde has joined #openstack-infra | 22:16 | |
| *** pcm_ has quit IRC | 22:16 | |
| *** yamamoto has joined #openstack-infra | 22:16 | |
| jeblair | that's devstack-gate | 22:16 |
| clarkb | ohh for test-matrix.py | 22:16 |
| jeblair | yah. is probably in our puppet manifests or something. i think it would be okay to explicitly add an install for it in devstack-gate | 22:17 |
| clarkb | sounds good to me | 22:17 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove query for bug 1357677 https://review.openstack.org/120925 | 22:17 |
| uvirtbot | Launchpad bug 1357677 in nova "Instances failes to boot from volume" [Undecided,Invalid] https://launchpad.net/bugs/1357677 | 22:17 |
| clarkb | jeblair: should that go in the -wrap.sh script? | 22:17 |
| arosen | clarkb: why don't use guys hit these errors? | 22:19 |
| clarkb | arosen: could be that python-yaml is in our base images or something in the image build is pulling it in | 22:19 |
| *** dannywilson has quit IRC | 22:19 | |
| clarkb | arosen: but you are using the same build scripts so more likely the base image? | 22:19 |
| *** weshay has joined #openstack-infra | 22:19 | |
| arosen | clarkb: i guess. My base image is just a plain ubuntu-server12.04 | 22:20 |
| *** cnesa9 has quit IRC | 22:20 | |
| jeblair | maybe in the not-wrap script since that's what calls it? | 22:20 |
| krotscheck | Anyone have an example of a sysvinit script | 22:21 |
| *** yamamoto has quit IRC | 22:21 | |
| clarkb | jeblair: good idea. that way we keep them together | 22:23 |
| clarkb | krotscheck: in the zuul puppet module | 22:23 |
| anteaya | sandywalsh: hi | 22:23 |
| clarkb | krotscheck: modules/zuul/files/zuul.init or some such | 22:23 |
| anteaya | sandywalsh: I see you have a number of patches to create new repos | 22:23 |
| jeblair | krotscheck: also /etc/init.d/skeleton on most debuntu systems | 22:23 |
| *** andreaf has quit IRC | 22:23 | |
| anteaya | sandywalsh: if you were to have the same topic on all of them say "new-project" that would make them more discoverable for reviewers | 22:24 |
| *** LuckyVega has quit IRC | 22:24 | |
| arosen | clarkb: also any idea why my jenkins is only posting the results of the nooop-check-communication test? http://paste.openstack.org/show/110358/ | 22:24 |
| *** weshay has quit IRC | 22:25 | |
| arosen | that's a snip of my zuul/layout.yaml which i think controls that. | 22:25 |
| arosen | or maybe it's related to jenkins_jobs/config/projects.yaml? | 22:25 |
| *** eharney has quit IRC | 22:26 | |
| clarkb | arosen: you may need to reload your config in zuul if the config changed? | 22:26 |
| clarkb | arosen: if you use our init script /etc/init.d/zuul reload should do that | 22:26 |
| arosen | let me try that... I restarted zuul i figured that would do it too. | 22:27 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Update for new home for puppetboard module https://review.openstack.org/119966 | 22:27 |
| wenlock | clarkb, thanks, working like a charm ! https://github.com/wenlock/myhome/blob/master/.vimrc_editor.vim | 22:28 |
| arosen | clarkb: same thing... The dsvm-tempest-full test results aren't getting posted. | 22:28 |
| *** emagana has quit IRC | 22:28 | |
| krotscheck | Thansk | 22:28 |
| clarkb | arosen: not sure on that then. give me a minute to finish this d-g patch first | 22:28 |
| *** emagana has joined #openstack-infra | 22:29 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Install PyYaml for test-matrix.py https://review.openstack.org/120929 | 22:30 |
| clarkb | arosen: ^ that is the first thing | 22:30 |
| clarkb | arosen: is zuul running the test and simply not reporting it? | 22:30 |
| arosen | clarkb: it's only reporting that it ran the noop-communication-test | 22:31 |
| clarkb | arosen: the first thing I would check is connect to your local gearman server `telnet localhost 4730` and run the 'status' command to check the job is registered by jenkins | 22:31 |
| clarkb | arosen: if it isn't zuul should say NOT_REGISTERED but that doesn't seem to happen here so probably not it but a good thing to confirm | 22:32 |
| arosen | http://paste.openstack.org/show/110360/ | 22:32 |
| *** jgrimm is now known as zz_jgrimm | 22:32 | |
| *** emagana has quit IRC | 22:32 | |
| arosen | how do i read this output :/ | 22:32 |
| *** emagana has joined #openstack-infra | 22:32 | |
| clarkb | arosen: ok so dsmv-tempest-full is listed. that means it is registered | 22:32 |
| arosen | yup i see that. | 22:33 |
| arosen | I see it in the jenkins ui and the job running. | 22:33 |
| jhesketh | Morning | 22:33 |
| clarkb | arosen: the numbers are in order, number queued, number running, numbre of nodes able to run job | 22:33 |
| arosen | ah okay. | 22:33 |
| anteaya | morning jhesketh | 22:33 |
| arosen | thanks. | 22:33 |
| clarkb | arosen: ok so it is running in jenkins but gearman doesn't report it as running | 22:33 |
| *** jerryz has quit IRC | 22:34 | |
| arosen | you can tell that from this? | 22:34 |
| clarkb | arosen: well you said jenkins ui shows it as running | 22:34 |
| clarkb | arosen: and that output says not jobs of that name running | 22:34 |
| arosen | ah okay. | 22:34 |
| clarkb | on the gearman side | 22:34 |
| *** ramashri has joined #openstack-infra | 22:35 | |
| arosen | what name should i be looking for? | 22:35 |
| *** armax has quit IRC | 22:35 | |
| clarkb | the name in your layout.yaml. dsvm-tempest-full | 22:36 |
| clarkb | that is our key in gearman and in jenkins | 22:36 |
| *** praneshp has quit IRC | 22:36 | |
| *** armax has joined #openstack-infra | 22:36 | |
| mattoliverau | Morning | 22:36 |
| clarkb | mattoliverau: hi there | 22:37 |
| clarkb | arosen: another thing to try may be to remove the noop comms check. just to see if that triggers different behavior | 22:37 |
| clarkb | it shouldn't but... | 22:37 |
| *** asadough1 has joined #openstack-infra | 22:37 | |
| anteaya | morning mattoliverau | 22:38 |
| arosen | clarkb: from etc/zuul/layout.yaml right? | 22:38 |
| clarkb | arosen: yup | 22:38 |
| *** openstackgerrit has quit IRC | 22:38 | |
| *** jlk has quit IRC | 22:38 | |
| *** asadoughi has quit IRC | 22:38 | |
| *** bhuvan has quit IRC | 22:38 | |
| *** oanufriev has quit IRC | 22:38 | |
| clarkb | arosen: and do the zuul reload after | 22:38 |
| arosen | yup | 22:38 |
| *** openstackgerrit_ has joined #openstack-infra | 22:39 | |
| *** bknudson has quit IRC | 22:39 | |
| arosen | clarkb: Eeee... It still posted: noop-check-communication SUCCESS in 0s | 22:39 |
| clarkb | arosen: ok good, I think that means zuul has an old config cached. That may mean you have a bad layout.yaml for some reason | 22:39 |
| clarkb | arosen: you should be able to do a zuul layout-validate or some such to have it check and spit out errors | 22:40 |
| *** dangers is now known as dangers_away | 22:40 | |
| *** openstackgerrit_ is now known as openstackgerrit | 22:40 | |
| *** gyee has quit IRC | 22:40 | |
| arosen | clarkb: if i stop zuul i should see it stop voting right? | 22:41 |
| *** daya_k has quit IRC | 22:41 | |
| clarkb | arosen: correct | 22:41 |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 22:41 |
| *** bhuvan has joined #openstack-infra | 22:42 | |
| arosen | clarkb: hrm zuul only has these commands: {enqueue,promote,show} | 22:43 |
| *** pballand has quit IRC | 22:44 | |
| *** dims_ has joined #openstack-infra | 22:44 | |
| *** praneshp has joined #openstack-infra | 22:45 | |
| asselin | arosen, btw, my script will check the zuul layout: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/manifests/master.pp#L291 | 22:45 |
| asselin | /usr/local/bin/zuul-server -t | 22:45 |
| *** dannywilson has joined #openstack-infra | 22:45 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 22:46 |
| *** persia has quit IRC | 22:47 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: lsb_release is missing for debian setup https://review.openstack.org/119245 | 22:48 |
| *** emagana has quit IRC | 22:48 | |
| *** emagana has joined #openstack-infra | 22:49 | |
| *** persia_ has joined #openstack-infra | 22:49 | |
| *** persia_ has quit IRC | 22:49 | |
| *** persia_ has joined #openstack-infra | 22:49 | |
| *** dannywilson has quit IRC | 22:49 | |
| *** roaet has joined #openstack-infra | 22:50 | |
| roaet | howdy infra folks | 22:51 |
| roaet | anyone here? | 22:51 |
| clarkb | roaet: hello | 22:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow ssh from the new puppetmaster https://review.openstack.org/120922 | 22:51 |
| roaet | clarkb: have you been having any issues with your instance builds? | 22:51 |
| roaet | as of the last... mmm.. 2 -3 hours? | 22:51 |
| roaet | with jenkins? | 22:51 |
| clarkb | roaet: can you be more specific? are you talking about the nodes managed by nodepool or the tempest tests run in those nodes? | 22:52 |
| *** rainya has quit IRC | 22:52 | |
| roaet | err... not even sure. | 22:52 |
| arosen | clarkb: asselin awesome it's wroking now :) | 22:52 |
| arosen | clarkb: gotta wait for your patch to merge first i guess though. | 22:52 |
| jeblair | Notice: /Stage[main]/Openstack_project::Base/Ssh_authorized_key[puppet-remote-2014-05-24]/ensure: removed | 22:52 |
| jeblair | Notice: /Stage[main]/Openstack_project::Base/Ssh_authorized_key[puppet-remote-2014-09-11]/ensure: created | 22:53 |
| *** rainya has joined #openstack-infra | 22:53 | |
| nibalizer | jeblair: excellent | 22:53 |
| roaet | I'm a racker checking if you're having any problems with your instance builds. <-- clarkb | 22:53 |
| *** emagana has quit IRC | 22:53 | |
| clarkb | roaet: ah ok. in that case probably the nodepool nodes /me looks at graphite really quickly | 22:53 |
| JayF | roaet: Who are you? | 22:53 |
| *** emagana has joined #openstack-infra | 22:53 | |
| clarkb | roaet: according to graphite dfw is having trouble. I can dig up logs for you shortly | 22:54 |
| clarkb | roaet: ord and iad look healthy though | 22:54 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Added the ability to run jeepyb not only on the host where the Gerrit installed. https://review.openstack.org/116936 | 22:54 |
| roaet | JayF: i'm Justin Hammond | 22:54 |
| jeblair | puppetmaster.openstack.org : ok=1 changed=0 unreachable=0 failed=0 | 22:54 |
| *** dimtruck is now known as zz_dimtruck | 22:54 | |
| jeblair | nibalizer, clarkb, mordred: ^ yay! | 22:54 |
| roaet | just bein' helpful really. I don't know if I'm supposed to be asking directly if you're having troubles. | 22:54 |
| roaet | as a dev | 22:54 |
| clarkb | roaet: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-91bbfd9d-b7ce-4ace-be40-4a2de8b4a7f5) so we seem to be seeing some of that | 22:55 |
| * nibalizer does happy dance | 22:55 | |
| roaet | yes. you should have been receiving a lot of those. | 22:55 |
| jeblair | nibalizer: roll the dice -- which server should i try first? | 22:55 |
| nibalizer | hrm, the fact that the puppet3master itself can run puppet without OMG FAILURE implies that the base->template->server path functions | 22:56 |
| nibalizer | jeblair: uh something unimportant | 22:56 |
| clarkb | roaet: and only in dfw | 22:56 |
| nibalizer | maybe paste? | 22:56 |
| jeblair | paste it is | 22:56 |
| clarkb | roaet: anything else you would like from our end? | 22:56 |
| roaet | yes. it's only dfw. | 22:56 |
| roaet | can you describe the situation at the time during those build failures? | 22:56 |
| roaet | It was incredibly bursty, just wondering what was happening to create such traffic. | 22:57 |
| nibalizer | jeblair: hrm i wonder if puppetdb is going to be a problem for us | 22:57 |
| jeblair | err: Could not load downloaded file /var/lib/puppet/lib/puppet/type/staging_file.rb: undefined method `newparams' for Puppet::Type::Staging_file:Class | 22:57 |
| jeblair | nibalizer: that's the only error ^ | 22:57 |
| clarkb | sure. so we have this daemon called nodepool that talks to our test systems to get test slave demand then calculates from that demand how many nodes to boot in each of our cloud regions | 22:57 |
| jeblair | info: Loading downloaded plugin /var/lib/puppet/lib/puppet/type/staging_file.rb | 22:58 |
| jeblair | err: Could not load downloaded file /var/lib/puppet/lib/puppet/type/staging_file.rb: undefined method `newparams' for Puppet::Type::Staging_file:Class | 22:58 |
| jeblair | nibalizer: ^ more context | 22:58 |
| roaet | clarkb: it also appears to happen weekly-ish | 22:58 |
| clarkb | roaet: we do rate limit limit requests on our end but you may still get bursts if we are getting nodes that error quickly keep trying to rebuild | 22:58 |
| roaet | i appreciate the traffic, it's a unique use-case that is helping with load and stuff | 22:59 |
| nibalizer | jeblair: hrm, does that come from a module or core? | 22:59 |
| *** jpich has joined #openstack-infra | 22:59 | |
| jeblair | roaet: happy to oblige! in fact, we'd be happy to do more ;) | 22:59 |
| clarkb | roaet: basically nodepool is as eager as it can be in attempting to fulfill the demand from the test systems. Though it is constrained by quota's set on our side and API rate limits that we set on our side too (iirc we set the first value to our ramquota/rampernode and second value to the suggested rate limit from you guys?) | 22:59 |
| roaet | jeblair: how do you mean? | 22:59 |
| roaet | clarkb: the biggest help for stability purposes would be to build requesting ipv6 | 23:00 |
| *** erlon has quit IRC | 23:00 | |
| roaet | but tbh, we should be able to handle this load. | 23:00 |
| fungi | insane quantities of scrollback. i skimmed and saw promising puppet3 progress. anything else important of which i should be aware? | 23:00 |
| *** rainya has quit IRC | 23:00 | |
| roaet | So you're helping us find new things. | 23:00 |
| clarkb | roaet: you mean use ipv6 endpoints instead of ipv4? | 23:00 |
| roaet | clarkb: that would be great. | 23:00 |
| clarkb | roaet: I am sure we are happy to do that, but today we just ask keystone I think | 23:00 |
| *** emagana has quit IRC | 23:00 | |
| roaet | clarkb: I mean for the test instances that you spin up, only request ipv6 | 23:01 |
| clarkb | roaet: oh that hrm | 23:01 |
| clarkb | jeblair: fungi ^ I think we can get away with that | 23:01 |
| *** emagana has joined #openstack-infra | 23:01 | |
| nibalizer | jeblair: can you find / -name 'staging_file.rb' on the puppet3master ? | 23:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix ruby-191-prep to install bundler https://review.openstack.org/120116 | 23:01 |
| nibalizer | i dont have that file at all on my vm which is supposed to look like your vm | 23:01 |
| roaet | clarkb: but again, that is something that could really help the stability, but we need your usecase to performance test | 23:01 |
| jeblair | we need to modify nodepool to support it. i'm ok with that, we've just been doing lowest common denominator right now tho | 23:01 |
| jeblair | clarkb: i'm assuming jenkins supports it | 23:02 |
| roaet | I'm a neutron dev and this is all very interesting. | 23:02 |
| clarkb | jeblair: does nodepool only grab the ipv4 address to do the initial ssh connection? | 23:02 |
| clarkb | jeblair: oh good point | 23:02 |
| fungi | yeah, it's something i've generally thought would be cool (for someone else with time to work on making happen) | 23:02 |
| jeblair | clarkb: yep (nodepool ipv4; it doesn't even have a db field for v6) | 23:02 |
| jeblair | but i'm ++ on doing that; shouldn't be super hard | 23:02 |
| jeblair | nibalizer: a second run completed without error | 23:03 |
| clarkb | roaet: so something we can do, it just won't happen today | 23:03 |
| nibalizer | jeblair: hrmmm weird | 23:03 |
| roaet | jeblair: clarkb could you keep me in the loop on it, just in case we run into ipv6 issues I will be able to help you directly (again, neutron dev) | 23:03 |
| jeblair | nibalizer: and yes: /etc/puppet/modules/staging/lib/puppet/type/staging_file.rb (on pupeptmaster) | 23:03 |
| nibalizer | well maybe that makes sense | 23:03 |
| roaet | jeblair: clarkb: justin.hammond@rackspace.com | 23:03 |
| clarkb | roaet: definitely. if you are curious the code that will need updating lives at https://git.openstack.org/cgit/openstack-infra/nodepool | 23:03 |
| *** ianw has quit IRC | 23:04 | |
| nibalizer | jeblair: so with --noop it ran successfully and didn't show any diffs of files? | 23:04 |
| *** ianw has joined #openstack-infra | 23:04 | |
| roaet | excellent. what would you say is the schedule of nodepool? | 23:04 |
| jeblair | nibalizer: it diffed, it was mostly puppet internal stuff that diffed | 23:04 |
| roaet | these, runs that cause a lot of traffic, how often, etc? | 23:04 |
| clarkb | roaet: right now we are kind of focused on feature freeze hand holding and upgrading our puppet master before puppet2.7 is EOLed at the end of the month | 23:05 |
| jeblair | nibalizer: second time i ran it for real it worked | 23:05 |
| clarkb | roaet: but changes to nodepool tend to happen relatively quickly | 23:05 |
| jeblair | cleanly | 23:05 |
| clarkb | roaet: I can try hacking on this early next week | 23:05 |
| *** emagana has quit IRC | 23:05 | |
| roaet | clarkb: oh no sweat. I was actually wondering how often nodepool does it's worker 'check' thing. | 23:05 |
| roaet | Since this has happened 3 thursdays in a row | 23:05 |
| roaet | I assume something is scheduled | 23:05 |
| *** emagana has joined #openstack-infra | 23:06 | |
| clarkb | roaet: I think that may be more related to demand from the test system | 23:06 |
| nibalizer | jeblair: thats really awesome | 23:06 |
| *** dalgaaf_ has quit IRC | 23:06 | |
| clarkb | though we have been flat out since freeze. I am not sure why it would happen every thursday | 23:06 |
| jeblair | nibalizer: there is now a /var/lib/puppet/lib/puppet/type/staging_file.rb on paste.o.o | 23:06 |
| clarkb | roaet: basically nodepool spins up the nodes, and before giving them to jenkins sshes into them and does a quick sanity check | 23:06 |
| roaet | Alright. I appreciate your help. I'm sorry that the system was a bit shaky. Infra though is a really intense user. | 23:07 |
| jeblair | nibalizer: so i think it self-corrected | 23:07 |
| clarkb | roaet: so that should happen for all nodes pretty continuous | 23:07 |
| clarkb | roaet: we are happy to help shake out all the bugs :) | 23:07 |
| clarkb | roaet: and you guys have been really responsive so it works well | 23:07 |
| jeblair | paste.openstack.org is moved to puppetmaster | 23:07 |
| nibalizer | jeblair: that error wasn't it not downlaoding i think, i think it downloaded but couldn't load it because some internal api was incompatible | 23:08 |
| nibalizer | but the second run it had all puppet 3 libs loaded so the file loaded fine maybe? | 23:08 |
| roaet | Has infra filed a ticket at all with rackspace or has it not become that serious? | 23:08 |
| nibalizer | jeblair: did you run it without --noop ? | 23:08 |
| jeblair | nibalizer: i'd like to do pbx.o.o now, because it's a less-important centos6 server | 23:08 |
| jeblair | nibalizer: yes | 23:08 |
| clarkb | roaet: I don't think we have filed anything for this particular issue. we run with 4 cloud regions and nodepool somewhat transparently shifts load for us so that we don't really notice | 23:08 |
| nibalizer | jeblair: sounds good | 23:08 |
| nibalizer | this is a glorious day | 23:08 |
| *** bhuvan_ has joined #openstack-infra | 23:09 | |
| jeblair | nibalizer: same thing about staging_file, but no other red flags | 23:09 |
| jeblair | will run for real now? | 23:10 |
| nibalizer | ++ | 23:10 |
| fungi | having caught back up on scrollback, yes jenkins seems to be fine connecting to slaves over ipv6... if you check established sockets i believe you'll see the slave agent connection preferring ipv6 addresses... double-checking that now | 23:10 |
| jeblair | pbx.openstack.org is moved to puppetmaster | 23:10 |
| jeblair | clarkb: want to grab some hosts? | 23:11 |
| clarkb | jeblair: sure | 23:11 |
| *** bhuvan has quit IRC | 23:11 | |
| jeblair | clarkb: want to take logstash*? | 23:11 |
| *** mfer has joined #openstack-infra | 23:11 | |
| clarkb | jeblair: how about the logstash* nodes to start | 23:11 |
| clarkb | :) | 23:11 |
| fungi | nope, my bad. looks like it's still defaulting to ipv4 | 23:11 |
| fungi | we could test that pretty easily though | 23:11 |
| clarkb | jeblair: remind me of what I am doing. login to node and `puppet agent --test --server puppetmaster.openstack.org --noop` and run without noop if clean? | 23:12 |
| nibalizer | be extra careful when watching file diffs | 23:12 |
| clarkb | jeblair: and finally edit the two hostfiles lists on the masters if things are clean? | 23:12 |
| nibalizer | since variable scoping is more strict in puppet 3 | 23:12 |
| jeblair | clarkb: yep. and yep. | 23:12 |
| fungi | oh, wait, nodepool is registering those with jenkins as raw v4 addresses. i should look at the long-running workers | 23:12 |
| jeblair | nibalizer: ack. clarkb: you will see some boilerplate puppet difs as it diffs its own internal modules, etc. | 23:12 |
| fungi | but still, no dice | 23:12 |
| clarkb | nibalizer: jeblair gotcha | 23:12 |
| *** mfer has quit IRC | 23:13 | |
| fungi | i suppose we could try registering a slave in jenkins-dev with only a v6 address at some point and confirm it keeps communication up | 23:13 |
| clarkb | fungi: ya that is what I was thinking a first step would be | 23:13 |
| clarkb | fungi: and if that works then update nodepol | 23:13 |
| jeblair | clarkb: we should run an editor in screen. 1 sec. | 23:13 |
| clarkb | jeblair: kk | 23:14 |
| fungi | in which case it's just a matter of convincing nodepool of the bits it will need | 23:14 |
| fungi | roaet: how do you tell nova boot you only want a v6 address? | 23:14 |
| mordred | fungi: which cloud? | 23:14 |
| anteaya | ianw: jenkins is unhappy on this patch: https://review.openstack.org/#/c/117396/ | 23:14 |
| mordred | fungi: because the way you ask for an IP address at all is different in different places | 23:14 |
| *** Guest54173 is now known as antonym | 23:15 | |
| fungi | mordred: well, yes, granted | 23:15 |
| roaet | fungi: i actually don't know how to do that off hand haha | 23:15 |
| jeblair | clarkb: su to root and 'screen -x' on both puppetmasters for the hostfile editing | 23:15 |
| jeblair | pypi.dfw.openstack.org is moved | 23:15 |
| *** antonym is now known as Guest35655 | 23:15 | |
| fungi | roaet: it sounded like that's what you were suggesting | 23:15 |
| roaet | fungi: that is what I'm suggesting, I just need to remember the command | 23:15 |
| fungi | unless i misunderstood | 23:15 |
| clarkb | jeblair: will do | 23:15 |
| jeblair | fungi, mordred: let me know if you want to play with moving some puppet hosts | 23:15 |
| fungi | roaet: ahh, cool | 23:15 |
| clarkb | logstash.o.o is not clean. moving onto workers before writing any puppet | 23:16 |
| roaet | fungi: I think, though, the ipv6 thing shouldn't be a large priority. | 23:16 |
| fungi | jeblair: i'm happy to give one a whirl, though evening is encoraching | 23:16 |
| mordred | jeblair: I may have missed the task ...tl;dr ? | 23:16 |
| nibalizer | clarkb: paste error for me? | 23:16 |
| roaet | I think we can fix stuff on our end to prevent you from doing extra work | 23:16 |
| jeblair | mordred: log into a host and run puppet agent --test --server puppetmaster.openstack.org --noop | 23:16 |
| *** yamamoto has joined #openstack-infra | 23:16 | |
| jeblair | mordred: see if it will apply clean or needs work | 23:16 |
| mordred | ah! is there an etherpad list? | 23:17 |
| *** Sincler has quit IRC | 23:17 | |
| clarkb | nibalizer: http://paste.openstack.org/show/110363/ | 23:17 |
| jeblair | mordred: let's make one here: https://etherpad.openstack.org/p/infra-puppet3 | 23:17 |
| fungi | roaet: well, it was already an interesting use case anyway, because we've talked with at least one potential donor provider who has basically no globally routable ipv4 assignment but would be willing to heap ipv6 addresses on us | 23:17 |
| *** dannywilson has joined #openstack-infra | 23:17 | |
| roaet | oh | 23:17 |
| roaet | ipv6 is remarkably available ;) | 23:17 |
| fungi | so we'd already pondered the requirements to some extent | 23:17 |
| roaet | I will ask around and get the information to you asap, got an e-mail I can send to? | 23:18 |
| roaet | feel free to pm it if need be | 23:18 |
| fungi | roaet: nah, if it's just a matter of reading novaclient's manpage i can likely manage to find out how to not request v4 addressing | 23:18 |
| clarkb | logstash-worker01 looks good. I am going to spot check 11 and 20 as well since they were built at differenttimes | 23:18 |
| roaet | fungi: I'm honestly not even sure at this point if it is possible. it really should be though | 23:19 |
| jeblair | pypi.iad.openstack.org is moved to puppetmaster | 23:20 |
| mordred | jogo: ^^ ? | 23:20 |
| fungi | roaet: back when i used to work at a service provider i was the network architect charged with v6ing all the things, so i'm definitely interested in seeing it be a first-class use case in openstack. just haven't had free cycles to devote to helping much in that direction | 23:20 |
| *** shashankhegde has quit IRC | 23:20 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 23:20 |
| *** shashankhegde has joined #openstack-infra | 23:21 | |
| roaet | fungi: I think it's possible but, and this is my opinion, our neutron really assumed everyone wanted ipv4.. so it just gives them out. I don't know how to make it not do that now that I'm lookin at the code. | 23:21 |
| jeblair | pypi.ord.openstack.org is moved to puppetmaster | 23:21 |
| *** yamamoto has quit IRC | 23:21 | |
| fungi | roaet: cool. i'm sure i'm not the only one who feels, deep down, like that's sort of amusing | 23:21 |
| roaet | i find it amusing and I helped write it | 23:22 |
| clarkb | oh boy emacs /me reads him a manual | 23:22 |
| jeblair | clarkb: you type in the letters you want to show up on the screen :) | 23:22 |
| mordred | roaet: funny thing is - I _do_ want to be able to just get nodes from clouds and not have to tell them to give me an ip address | 23:22 |
| mordred | roaet: but also, sometimes, I don't | 23:22 |
| clarkb | jeblair: :P | 23:22 |
| fungi | jeblair: so which lines in the etherpad correspond to the individual server moves to new puppet? 27-32? | 23:23 |
| jeblair | mordred: the cert on pypi.region-b.geo-1.openstack.org seems to have server problems | 23:23 |
| jeblair | er cert problems | 23:23 |
| mordred | jeblair: I don't know what that menas? | 23:23 |
| roaet | mordred: you want to have a node with no connectivity at first then add connectivity later? | 23:23 |
| jeblair | mordred: puppet does not work because it has no cert on hpcloud pypi | 23:23 |
| mordred | jeblair: o_O | 23:24 |
| jeblair | info: Creating a new SSL key for pypi.openstacklocal | 23:24 |
| jeblair | mordred: it looks like a hostname problem | 23:24 |
| mordred | oh - yeah | 23:24 |
| mordred | the openstacklocal thing | 23:24 |
| mordred | sigh | 23:24 |
| jeblair | certname=pypi.openstacklocal | 23:25 |
| jeblair | in puppet.conf | 23:25 |
| clarkb | jeblair: I can let you flush the buffers when you are happy | 23:25 |
| *** MaxV has quit IRC | 23:25 | |
| mordred | clarkb: when I test one and I want to move it over, how do I do that? | 23:25 |
| jeblair | mordred: we have root owned screen sessions on both puppetmasters | 23:25 |
| clarkb | mordred: login to both puppet masters and screen -x as root. This will bring up the common editors | 23:25 |
| mordred | ah | 23:25 |
| jeblair | hitech | 23:25 |
| clarkb | I am going to let jeblair ^X^S | 23:25 |
| clarkb | to avoid conflicts flushing | 23:25 |
| fungi | i bet that's the "pypi" that's been spamming cron | 23:26 |
| clarkb | nibalizer: any thoughts on that error? I am goin to look at it now | 23:26 |
| *** stevemar has quit IRC | 23:26 | |
| nibalizer | clarkb: ya its easy fix | 23:26 |
| sdague | clarkb: your bash-fu is weak :) | 23:26 |
| clarkb | nibalizer: oh you want to push it? | 23:26 |
| sdague | https://review.openstack.org/#/c/120929/1/devstack-vm-gate.sh - else if | 23:26 |
| clarkb | sdague: aroo? | 23:26 |
| clarkb | oh man | 23:26 |
| clarkb | that will learn me | 23:26 |
| nibalizer | review incomming | 23:26 |
| sdague | I so had to call you out on that :) | 23:26 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Fixing variable access in logstash module https://review.openstack.org/120937 | 23:26 |
| nibalizer | clarkb: ^^ | 23:26 |
| * clarkb fixes | 23:26 | |
| jeblair | mordred: so any idea about the openstacklocal thing? | 23:26 |
| clarkb | sdague: wait elif is a thing isn't it? | 23:27 |
| jeblair | mordred: i assume if i try to fix it, puppet will still think that's the fqdn and fix it back | 23:27 |
| clarkb | sdague: too many things going on at once | 23:27 |
| fungi | hrm, speaking of cronspam, the git remote updates on the puppetmasters are generating stdout to e-mail every 15 minutes | 23:27 |
| nibalizer | jeblair: what does `facter fqdn` show ? | 23:27 |
| clarkb | clearly it exploded so must not be | 23:27 |
| nibalizer | probably you've got the fqdn on the box borked | 23:27 |
| jeblair | nibalizer: pypi.openstacklocal | 23:27 |
| sdague | clarkb: oh, maybe it was your : | 23:27 |
| roaet | thanks all. I'll keep you in the loop. | 23:27 |
| mordred | jeblair: no - you should be able to set the certname in puppet.conf, fix /etc/hosts, and then run puppet and it should set it all to be the right thing | 23:27 |
| jeblair | nibalizer: by "you" i'm betting you mean "hpcloud" | 23:27 |
| *** roaet has left #openstack-infra | 23:27 | |
| clarkb | sdague: yup thats it | 23:28 |
| nibalizer | jeblair: indeed :) | 23:28 |
| sdague | clarkb: yeh | 23:28 |
| jeblair | mordred: okay, i'll try that a few times :) | 23:28 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Install PyYaml for test-matrix.py https://review.openstack.org/120929 | 23:28 |
| clarkb | nibalizer: so is that not just a warning then? | 23:28 |
| jeblair | mordred: what exactly should i put in /etc/hosts? | 23:28 |
| jeblair | mordred: should i look up the public ip addr and put it in there? | 23:29 |
| nibalizer | clarkb: yea idunno, i thought they were deprecation errors i guess they're hard failures now; gg puppet | 23:29 |
| mordred | jeblair: yes | 23:29 |
| clarkb | mordred: jeblair fungi https://review.openstack.org/#/c/120937/ can that get a quick review? should hopefully free up logstash.o.o too | 23:29 |
| *** camunoz has joined #openstack-infra | 23:30 | |
| mordred | jeblair: 15.126.223.24 | 23:30 |
| mordred | clarkb: once we noop, if that works, are we running again without noop? | 23:30 |
| clarkb | jeblair: mordred: I am going to remove my nodes from the etherpad list when they are done | 23:31 |
| nibalizer | clarkb: so likely we'll need https://review.openstack.org/#/c/107243/ as well | 23:31 |
| clarkb | mordred: yes run without noop if noop is good | 23:31 |
| clarkb | nibalizer: yup | 23:31 |
| nibalizer | but if we dont want to touch gerrit today, thats A OK with me :P | 23:31 |
| jeblair | -certname=pypi.region-b.geo-1.openstack.org | 23:31 |
| jeblair | +certname=pypi.openstacklocal | 23:31 |
| jeblair | :/ | 23:32 |
| jeblair | facter fqdn now reports blank | 23:32 |
| mordred | wow | 23:32 |
| jeblair | probably because THE HOST DOES NOT KNOW ITS OWN IP ADDRESS | 23:32 |
| mordred | maybe put it in on the first 127.0.0.1 line? | 23:32 |
| jeblair | so adding the ip address of pypi.region-b.geo-1.openstack.org to /etc/hosts won't work | 23:32 |
| jeblair | there is so much wrong with that | 23:32 |
| *** r-daneel has quit IRC | 23:33 | |
| mordred | indeed. | 23:33 |
| nibalizer | convincing servers to rename themselves has always been very hard for me | 23:33 |
| *** dims_ has quit IRC | 23:34 | |
| jeblair | okay, also the host has an abbreviated hostname | 23:34 |
| openstackgerrit | Jeffrey Zhang proposed a change to openstack-infra/config: Add keystone-formula project https://review.openstack.org/120324 | 23:35 |
| asselin | I set min-ready:0 in nodepool. Is that legal? Would it possible result in a zuul not registered error? | 23:35 |
| jeblair | mordred: okay, i think setting /etc/hostname correctly causes facter fqdn to return the right thing without anything in /etc/hosts | 23:35 |
| *** baoli has joined #openstack-infra | 23:36 | |
| fungi | asselin: yes, the jobs won't be registered in zuul until the a jenkins master has at least one slave which can run them | 23:36 |
| mordred | jeblair: excellent | 23:36 |
| mordred | jeblair: perhaps we shoudl add that cantrip to launch_node | 23:37 |
| *** oomichi has joined #openstack-infra | 23:37 | |
| jeblair | mordred: yeah i think so. we might have it in nodepool | 23:37 |
| *** nelsnelson has quit IRC | 23:37 | |
| jeblair | pypi.region-b.geo-1.openstack.org is moved | 23:37 |
| fungi | clarkb: jeblair: mordred: are we doing lines 27-32 one machine at a time, or just doing recon on most of them first? | 23:37 |
| clarkb | note git.o.o is a scary one | 23:37 |
| jeblair | oh you know what, i haven't been upgrading puppet | 23:38 |
| clarkb | oh I wasn't updating the apt pin | 23:38 |
| clarkb | jeblair: :) neither have I. I think this is a good first step without that | 23:38 |
| mordred | ++ | 23:38 |
| fungi | i'll give jenkins-dev a shot before i duck out for the evening | 23:38 |
| clarkb | then we decouple a couple of things | 23:38 |
| fungi | huh... pretty massive diff on the --test --noop | 23:40 |
| fungi | oh | 23:40 |
| jamielennox | anteaya: are you around? | 23:40 |
| fungi | looks like puppet's not been running on this node for ages | 23:40 |
| clarkb | fungi: run it again if its just /var/lib/puppet stuff. puppet updates itself that way | 23:40 |
| clarkb | fungi: well its also diff with the contents of puppet3 things | 23:41 |
| jeblair | fungi: i think i'm seeing the same thing on pypi.slave | 23:41 |
| fungi | yeah, i have a puppet agent process on here from april | 23:41 |
| fungi | with a defunct zombie child | 23:41 |
| clarkb | wow | 23:41 |
| fungi | so i guess the implied pre-step is "make sure the server is up to date with the old puppetmaster" | 23:42 |
| jeblair | i have no idea what the problem is on pypi.slave :/ | 23:42 |
| clarkb | git01 updated jeepyb when I ran it | 23:42 |
| jeblair | Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://ci-puppetmaster.openstack.org/pluginfacts: Error 400 on SERVER: Not authorized to call find on /file_metadata/pluginfacts with {:source_permissions=>"use", :links=>"manage"} | 23:42 |
| clarkb | whcih should be safe so I am going to move git01 through git05 | 23:42 |
| fungi | clarkb: i think i merged a jeepyb change just a little while ago | 23:42 |
| mordred | jeblair: zoikes | 23:42 |
| clarkb | fungi: that would explain it | 23:43 |
| fungi | oh, puppet can't update here because it's failing to install nodepool | 23:43 |
| clarkb | jeblair: that is sayig puppet://ci-puppetmaster? | 23:43 |
| fungi | pkg_resources.VersionConflict: (pbr 0.5.23 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('pbr>=0.6,!=0.7,<1.0')) | 23:43 |
| nibalizer | another task to put on the pile is have the new puppetmaster report to puppetdb, because I think thats disabled right now | 23:43 |
| jeblair | clarkb: yeah, i'm running it against the old master and it's barfing | 23:43 |
| nibalizer | maybe not, idunno | 23:44 |
| jeblair | it seems like it's refusing to talk to either one | 23:44 |
| clarkb | ok I was just notified football is on tv so I am going to run home. will resume from there | 23:45 |
| jeblair | i think it has tho wrong cert | 23:45 |
| mordred | what are the ramifications of an empty serveradmin in an apache conf file? | 23:45 |
| fungi | okay, after some pip surgery, nodepool can't install because mysql dev packages aren't installed on jenkins-dev | 23:46 |
| jeblair | oh, no it does have the right certs | 23:47 |
| *** rainya has joined #openstack-infra | 23:47 | |
| nibalizer | mordred: http://httpd.apache.org/docs/2.2/mod/core.html#serveradmin | 23:47 |
| nibalizer | did puppet drop it in the diff? | 23:47 |
| *** rainya has quit IRC | 23:47 | |
| mordred | yeah | 23:48 |
| mordred | but only in puppet 3 | 23:48 |
| nibalizer | what node? | 23:48 |
| mordred | storyboard | 23:48 |
| mordred | + ServerAdmin | 23:48 |
| mordred | + | 23:48 |
| nibalizer | i think we should fix it in the puppet code, though the ramifications of that are probably zilch | 23:49 |
| mordred | nibalizer: is there a sensible fix? | 23:50 |
| *** baoli has quit IRC | 23:50 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixing variable access in logstash module https://review.openstack.org/120937 | 23:50 |
| nibalizer | mordred: im investigating | 23:51 |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Create repo for keystoneclient-federation https://review.openstack.org/120670 | 23:51 |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Add keystoneclient-kerberos repository https://review.openstack.org/120261 | 23:51 |
| nibalizer | if you run it in --noop agianst the old puppetmaster does it also remove the serveradmin directive? | 23:51 |
| jeblair | oh! | 23:52 |
| jeblair | pypi.slave is running puppet 3.6 | 23:52 |
| anteaya | interesting effect of being in a white hot rage with no end in sight, people that care about me find things to apologize to me for, things they would never apologize for normally | 23:52 |
| jeblair | i will stop trying to get it to talk to the old master | 23:52 |
| fungi | clarkb: so you say run the --noop against the new puppetmaster twice? | 23:52 |
| *** dims_ has joined #openstack-infra | 23:52 | |
| anteaya | I've had two apolgies in the last hour | 23:53 |
| anteaya | I have to say I am surprised | 23:53 |
| * anteaya goes back to reviews | 23:53 | |
| fungi | clarkb: yep, that seemed to do it | 23:53 |
| fungi | and it's /usr/local/bin/puppet-inventory we're removing/adding servers to on each master? | 23:54 |
| jeblair | fungi: no, screen -x as root on both masters | 23:54 |
| jeblair | fungi: they are editing /etc/ansible/hostfile | 23:54 |
| *** dims__ has joined #openstack-infra | 23:55 | |
| fungi | thanks. was trying to infer that from the etherpad and was probably staring right at it | 23:55 |
| sandywalsh | anteaya: thanks, do you mean in the commit message first-line? | 23:56 |
| jeblair | my guess is that the long-running slaves ended up picking up the puppet3 transition we did for the workers | 23:56 |
| anteaya | sandywalsh: well new-project is a good topic for all of them | 23:56 |
| anteaya | sandywalsh: and something in the commit message for stacktach-sandbox, title is best | 23:57 |
| anteaya | sandywalsh: does that make sense? | 23:57 |
| sandywalsh | anteaya: right, but when you say "topic", do you mean "commit message" or something else? | 23:57 |
| anteaya | sandywalsh: topic is usually the git branch of the commit | 23:57 |
| fungi | are we changing the puppetdb-terminus and facter pins when we change the top pin? | 23:57 |
| sandywalsh | anteaya: I'll fix up the sandbox one though, thanks for that | 23:57 |
| anteaya | sandywalsh: which can be set with git review -t new-project | 23:58 |
| sandywalsh | anteaya: ah, the topic branch ... gotcha | 23:58 |
| jeblair | fungi: we would, but we've been skipping the puppet3 client side upgrade right now | 23:58 |
| *** dims_ has quit IRC | 23:58 | |
| anteaya | sandywalsh: or in the gui, click the little icon to the right of the topic text box | 23:58 |
| fungi | and the top pin is changing from 2.7.* to 3.* right? | 23:58 |
| jeblair | fungi: yep | 23:58 |
| sandywalsh | anteaya: k, well, I'm nearly done (only a couple more repos to go up ... think I still need to change them, or just note for future reference?) | 23:59 |
| *** emagana has quit IRC | 23:59 | |
| fungi | jeblair: have you been changing the pin still and just not upgrading the client yet but still adding to the new puppetmaster? | 23:59 |
| jeblair | fungi: no i haven't been changing the pin at all | 23:59 |
| fungi | or skipping the pin edit as well? | 23:59 |
| anteaya | sandywalsh: you can change the ones that are up, use the gui | 23:59 |
| fungi | okay, i'll follow suit | 23:59 |
| *** emagana has joined #openstack-infra | 23:59 | |
| anteaya | sandywalsh: and if you need to submit a new patch use git review -t new-topic | 23:59 |
| sandywalsh | anteaya: k | 23:59 |
| jeblair | fungi: so no pin changes or puppet upgrades, just changing the puppetmaster | 23:59 |
| anteaya | plus you can use that command for new repos | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
