| mordred | jeblair: I can confirm log streaming | 00:00 |
|---|---|---|
| jeblair | whee! | 00:00 |
| mordred | jeblair: both of those seemed unhappy | 00:01 |
| jeblair | mordred: i agree; i have switched nodepool back | 00:01 |
| *** jamesmcarthur has joined #openstack-infra | 00:05 | |
| *** lakshmiS has joined #openstack-infra | 00:05 | |
| *** dimtruck is now known as zz_dimtruck | 00:06 | |
| jeblair | oh i wonder if $HOME is not set | 00:07 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Make sorting consistent on smaller devices https://review.openstack.org/318985 | 00:07 |
| *** flepied1 has quit IRC | 00:07 | |
| *** pilgrimstack has quit IRC | 00:07 | |
| *** Swami has quit IRC | 00:10 | |
| *** thorst has joined #openstack-infra | 00:11 | |
| *** mixos has quit IRC | 00:11 | |
| *** thorst_ has joined #openstack-infra | 00:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME https://review.openstack.org/318987 | 00:12 |
| jeblair | look at all the things we're learning | 00:13 |
| *** sflanigan has joined #openstack-infra | 00:13 | |
| *** sflanigan has joined #openstack-infra | 00:13 | |
| *** salv-orlando has joined #openstack-infra | 00:13 | |
| clarkb | are you using the shell module? | 00:13 |
| clarkb | I want to say that shpuld set $HOME | 00:13 |
| clarkb | but command wont | 00:14 |
| jeblair | clarkb: no, the zuul_runner module (so that we get redirected timestamped console logs) | 00:14 |
| clarkb | ah | 00:14 |
| clarkb | this is why the d-g stuff uses shell over command iirc | 00:14 |
| *** Qiming has quit IRC | 00:16 | |
| *** thorst has quit IRC | 00:16 | |
| *** csomerville has joined #openstack-infra | 00:16 | |
| *** sridhar_ram_ has joined #openstack-infra | 00:17 | |
| *** salv-orlando has quit IRC | 00:17 | |
| clarkb | jeblair: you may also need to set localr stuff | 00:18 |
| notmorgan | jeblair, mordred: http://paste.openstack.org/show/497820/ | 00:18 |
| clarkb | as libc default is very often not utf8 friendly | 00:18 |
| notmorgan | jeblair, mordred: that can be dropped into .ssh/config and it should do exactly what you asked for, if you want to pre-populate the ECDSA keys it would be possible to make a global entry for them somewhere on thesystme | 00:19 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Move prepare-node to openstack-repos https://review.openstack.org/318990 | 00:19 |
| notmorgan | jeblair, mordred: and/or move to stricthostkeychecking yes (if keys are pre-populated to a file) | 00:19 |
| jeblair | clarkb: hrm. maybe we should make this a login shell? what do you think would be better -- we set the minimal parameters explicitly, or we punt and source things like a login shell before running anything? | 00:19 |
| * notmorgan left HashKnownHosts for simplicity sake | 00:20 | |
| clarkb | jeblair I like punting but then distros may argue over what values shpuld be | 00:20 |
| *** r-mibu has quit IRC | 00:20 | |
| jeblair | notmorgan: yes, thanks for thinking of the hash thing -- that's really helpful on systems like these | 00:20 |
| *** kzaitsev_mb has quit IRC | 00:20 | |
| *** EricGonc_ has quit IRC | 00:20 | |
| clarkb | notmorgan: ecdsa? there still potentially be dragons there | 00:20 |
| *** sdake has quit IRC | 00:21 | |
| notmorgan | clarkb: hostkeys are default to ECDSA | 00:21 |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 00:21 |
| notmorgan | but can be rsa | 00:21 |
| clarkb | maybe centos6 was the last one to only dsa and rsa | 00:21 |
| notmorgan | clarkb: (this is server keys vs. user keys) | 00:21 |
| *** sdake has joined #openstack-infra | 00:21 | |
| clarkb | notmorgan: we definitely had hosts that didmt ecdsa but I may be thinking of centos6 | 00:21 |
| clarkb | yes I know | 00:21 |
| notmorgan | clarkb: wouldn't matter though | 00:21 |
| clarkb | ok | 00:21 |
| notmorgan | if the keys are pre-populated to a central known file we could stricthostkeycheck | 00:21 |
| *** sdake has quit IRC | 00:21 | |
| notmorgan | to eys | 00:22 |
| notmorgan | yes* | 00:22 |
| notmorgan | otherwise ask is really the only "safe" option to use | 00:22 |
| notmorgan | or "no" | 00:22 |
| clarkb | notmorgan: there is no good way with opemstack to strict check | 00:22 |
| *** pilgrimstack has joined #openstack-infra | 00:22 | |
| jeblair | notmorgan: what's 'ask'? | 00:22 |
| notmorgan | clarkb: well this would be for known hosts like .o.o | 00:23 |
| clarkb | because you cant get hostkey out of banf | 00:23 |
| notmorgan | clarkb: vs throw-aways | 00:23 |
| prometheanfire | can't git-review? | 00:23 |
| *** jamesmcarthur has quit IRC | 00:23 | |
| notmorgan | jeblair: the default behaviro "HEY found this unknown key, do you want to add it?" | 00:23 |
| notmorgan | jeblair: and error if the key changes from the known host | 00:23 |
| jeblair | is that the same as 'yes'? | 00:23 |
| clarkb | notmorgan: ya I think its a good idea to have something just pointing out the first one is always going to have some risk involved | 00:23 |
| notmorgan | jeblair: yes requires the key to be there, no asking | 00:23 |
| jeblair | oh ok | 00:24 |
| notmorgan | jeblair: if the key isn't there it fails | 00:24 |
| clarkb | mordred has asked for a feature in nova to handle this better | 00:24 |
| notmorgan | clarkb: and that config basically says anything you're not reaching for via a hostname of *.openstack.org will not store hostkeys and not check | 00:25 |
| notmorgan | clarkb: we could include lots of ips as well for known-long-lived things | 00:25 |
| jeblair | notmorgan: cool, so i think we can use 'ask' when we get to a point where we have a source for host keys for our semi-permanent servers. but we don't now, so we still probably want 'no' for openstack.org (otherwise, we will need to do a lot of manual intervention when bringing up a new launcher) | 00:25 |
| mordred | clarkb: yah. I need to update that spec | 00:25 |
| notmorgan | clarkb: but that is extending beyond the very simplle case. | 00:25 |
| *** Sukhdev has joined #openstack-infra | 00:25 | |
| notmorgan | jeblair: sure. easy enough. | 00:25 |
| *** r-mibu has joined #openstack-infra | 00:25 | |
| prometheanfire | oh, it did work | 00:25 |
| prometheanfire | wonder why I didn't see it | 00:25 |
| jeblair | notmorgan: (we only have host keys for a few servers so far; we can probably add more to hiera moderately easily) | 00:26 |
| notmorgan | jeblair: i'll create a special entry for ones we care about next. let me figure out where to put this in system-config? | 00:26 |
| mnaser | (not sure if im tuned in but, if only puppet exported resources actually worked properly) | 00:26 |
| notmorgan | jeblair: the idea will be "openstack.org will store host keys", non-o.o will not store at all, and we'll have one we can add hosts to if we want hostkeychecking | 00:27 |
| ianw | hmmm ... http://imgur.com/oM4E2RR | 00:27 |
| *** salv-orlando has joined #openstack-infra | 00:27 | |
| jeblair | notmorgan: yep | 00:27 |
| notmorgan | jeblair: cool. i'll brew up some puppet for this. long term we def. want to pre-populate host-keys for know semi-permanent/permanent things. and do strict checking | 00:28 |
| jeblair | ianw: is that supposed to have cycles? | 00:28 |
| ianw | jeblair : i would say no :) | 00:28 |
| mordred | mnaser: :) | 00:28 |
| *** pvaneck has quit IRC | 00:28 | |
| mnaser | o/ mordred | 00:28 |
| jeblair | ianw: neat! | 00:28 |
| mnaser | i hear you took this thing called days off | 00:28 |
| *** salv-orlando has quit IRC | 00:30 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Remove node-devstack from nodepool https://review.openstack.org/318994 | 00:30 |
| prometheanfire | ianw: jeblair: I don't think cycles mater for dib, since actual ordering is done other ways | 00:31 |
| ianw | yeah, the requirements are I guess in different phases | 00:31 |
| prometheanfire | ianw: jeblair: element-deps is just a collector, not really a dep-graph | 00:31 |
| *** ddieterly has joined #openstack-infra | 00:31 | |
| *** jamielennox|away is now known as jamielennox | 00:31 | |
| prometheanfire | ianw: check my last two reviews, they remove node-devstack | 00:32 |
| *** mixos has joined #openstack-infra | 00:32 | |
| *** sdake has joined #openstack-infra | 00:32 | |
| *** sdake has quit IRC | 00:32 | |
| *** sarob has quit IRC | 00:32 | |
| *** piet has quit IRC | 00:36 | |
| *** ffrank has quit IRC | 00:36 | |
| *** sarob has joined #openstack-infra | 00:39 | |
| *** bpokorny_ has joined #openstack-infra | 00:39 | |
| *** Apoorva has quit IRC | 00:40 | |
| *** tqtran has quit IRC | 00:41 | |
| *** andreykurilin has quit IRC | 00:42 | |
| *** baoli has joined #openstack-infra | 00:43 | |
| *** andreykurilin__ has quit IRC | 00:43 | |
| *** bpokorny has quit IRC | 00:44 | |
| *** sarob has quit IRC | 00:44 | |
| *** bpokorny_ has quit IRC | 00:44 | |
| *** amotoki has joined #openstack-infra | 00:46 | |
| *** amotoki has quit IRC | 00:50 | |
| *** jamesmcarthur has joined #openstack-infra | 00:51 | |
| *** mixos has quit IRC | 00:53 | |
| *** ddieterly has quit IRC | 01:01 | |
| *** baoli has quit IRC | 01:02 | |
| *** zhurong has joined #openstack-infra | 01:02 | |
| *** jamesmcarthur has quit IRC | 01:03 | |
| *** mixos has joined #openstack-infra | 01:06 | |
| *** amotoki has joined #openstack-infra | 01:07 | |
| *** thorst_ has quit IRC | 01:07 | |
| *** ddieterly has joined #openstack-infra | 01:08 | |
| *** thorst has joined #openstack-infra | 01:08 | |
| *** EricGonczer_ has joined #openstack-infra | 01:08 | |
| *** amotoki_ has joined #openstack-infra | 01:10 | |
| *** sarob has joined #openstack-infra | 01:10 | |
| *** amotoki has quit IRC | 01:11 | |
| *** Apoorva has joined #openstack-infra | 01:12 | |
| *** hichihara has joined #openstack-infra | 01:14 | |
| *** sarob has quit IRC | 01:15 | |
| *** sdake has joined #openstack-infra | 01:15 | |
| *** ddieterly is now known as ddieterly[away] | 01:15 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/system-config: Provide customized /etc/ssh/ssh_config https://review.openstack.org/319005 | 01:16 |
| *** thorst has quit IRC | 01:16 | |
| *** Qiming has joined #openstack-infra | 01:16 | |
| *** sarob has joined #openstack-infra | 01:16 | |
| *** baoli has joined #openstack-infra | 01:17 | |
| notmorgan | jeblair, mordred, clarkb: ^ there is the 1st pass, but i def need to 2x check a couple things before it's ready | 01:18 |
| notmorgan | jeblair, mordred, clarkb: i did not find a good pre-made module to cover complex ssh_config options (very basic ones only). I'll circle up and poke at a few of the puppet-experts we have on that and make sure i'm not doing something absurdly silly. | 01:19 |
| notmorgan | also... what nodes should get customized ssh things? | 01:19 |
| *** jianghuaw has quit IRC | 01:20 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 01:20 |
| *** sarob has quit IRC | 01:22 | |
| *** baoli has quit IRC | 01:23 | |
| *** sarob has joined #openstack-infra | 01:23 | |
| jhesketh | jeblair: finished reviewing your zuul changes fyi | 01:27 |
| *** jamesmcarthur has joined #openstack-infra | 01:28 | |
| *** baoli has joined #openstack-infra | 01:28 | |
| *** Guest60864 is now known as jgriffith | 01:29 | |
| ianw | greghaynes / SpamapS : you might like to look at https://etherpad.openstack.org/p/gentoo-ci when around. i think it could fit great with the booting test we want to get in | 01:30 |
| clarkb | and Nakato has a new ps for zuul cloner change | 01:30 |
| clarkb | will.have to try rereviewing tonight | 01:30 |
| *** sflanigan has quit IRC | 01:31 | |
| prometheanfire | ianw: have gentoo be that test :P that'll help get eyes on this lol | 01:31 |
| *** flepied has joined #openstack-infra | 01:31 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** salv-orlando has joined #openstack-infra | 01:37 | |
| *** ddieterly[away] is now known as ddieterly | 01:40 | |
| *** ddieterly is now known as ddieterly[away] | 01:40 | |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Fix H703 test https://review.openstack.org/319014 | 01:41 |
| *** salv-orlando has quit IRC | 01:41 | |
| *** Clint has left #openstack-infra | 01:42 | |
| *** Apoorva has quit IRC | 01:44 | |
| *** baoli has quit IRC | 01:46 | |
| *** thorst has joined #openstack-infra | 01:47 | |
| *** thorst has quit IRC | 01:47 | |
| *** thorst has joined #openstack-infra | 01:47 | |
| *** jamesmcarthur has joined #openstack-infra | 01:48 | |
| *** baoli has joined #openstack-infra | 01:53 | |
| *** jamesmcarthur has quit IRC | 01:53 | |
| *** eliqiao has joined #openstack-infra | 01:54 | |
| *** thorst has quit IRC | 01:56 | |
| ianw | prometheanfire: i think https://imgur.com/n37xfpp represents the conclusion i've come to pulling this apart | 01:57 |
| *** piet has joined #openstack-infra | 01:57 | |
| * ianw goes to get a sandwich and resists temptation to make dib output graphviz | 01:57 | |
| *** baoli has quit IRC | 01:58 | |
| prometheanfire | ianw: Sam-I-Am knows all about sandwiches | 01:59 |
| *** jamesmcarthur has joined #openstack-infra | 02:03 | |
| *** gomarivera has joined #openstack-infra | 02:03 | |
| *** banix has joined #openstack-infra | 02:06 | |
| *** gomarivera has quit IRC | 02:07 | |
| *** xiangxinyong has quit IRC | 02:11 | |
| *** xiangxinyong has joined #openstack-infra | 02:11 | |
| *** lakshmiS has quit IRC | 02:12 | |
| *** Sukhdev has quit IRC | 02:16 | |
| *** piet has quit IRC | 02:21 | |
| *** _prad_ has quit IRC | 02:23 | |
| *** vhosakot has joined #openstack-infra | 02:23 | |
| *** sridhar_ram_ has quit IRC | 02:25 | |
| *** amotoki_ has quit IRC | 02:27 | |
| *** xiangxinyong has quit IRC | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** x00350071 has joined #openstack-infra | 02:28 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add session param doc of get_recent_failed_runs() https://review.openstack.org/319026 | 02:29 |
| prometheanfire | ianw: cache-bindep depended on openstack_repos before the repos had a dep on node-devstack | 02:29 |
| *** sarob has quit IRC | 02:29 | |
| *** yonglihe has joined #openstack-infra | 02:30 | |
| *** amotoki has joined #openstack-infra | 02:30 | |
| prometheanfire | ianw: should I change openstack_repos to dep on the new prepare-node element at the same time I create the element? | 02:31 |
| *** flepied has quit IRC | 02:35 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add optional start_date parameter to get_recent*() methods https://review.openstack.org/307416 | 02:35 |
| *** gildub has quit IRC | 02:38 | |
| *** ddieterly[away] has quit IRC | 02:41 | |
| *** vhosakot has quit IRC | 02:42 | |
| *** vhosakot has joined #openstack-infra | 02:43 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Move prepare-node to it's own element https://review.openstack.org/319030 | 02:44 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add session param doc of get_recent_failed_runs() https://review.openstack.org/319026 | 02:44 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add link to published release notes in readme https://review.openstack.org/318433 | 02:45 |
| *** jpr has joined #openstack-infra | 02:46 | |
| *** jamesmcarthur has joined #openstack-infra | 02:47 | |
| *** jpr1 has joined #openstack-infra | 02:47 | |
| *** csomerville has quit IRC | 02:49 | |
| *** jpr has quit IRC | 02:51 | |
| *** tlian has quit IRC | 02:51 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Remove node-devstack from nodepool https://review.openstack.org/318994 | 02:51 |
| *** yuanying has quit IRC | 02:51 | |
| prometheanfire | ianw: ok, think those two reviews clean it up like your graph shows | 02:52 |
| *** piet_ has joined #openstack-infra | 02:53 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** shashank_hegde has joined #openstack-infra | 02:54 | |
| *** thorst has joined #openstack-infra | 02:54 | |
| *** dims_ has quit IRC | 02:55 | |
| *** woodster_ has quit IRC | 02:58 | |
| *** dims has joined #openstack-infra | 03:00 | |
| notmorgan | mnaser: mordred is bad at taking days off | 03:00 |
| notmorgan | mnaser: he totally should be better at it | 03:00 |
| prometheanfire | notmorgan: we are all bad at that | 03:00 |
| notmorgan | prometheanfire: but mordred is *really* bad at it | 03:01 |
| *** thorst has quit IRC | 03:01 | |
| *** yuanying has joined #openstack-infra | 03:01 | |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| openstackgerrit | Ethan Lynn proposed openstack-infra/project-config: Add gate-senlin-dsvm-tempest-api job to senlin's experimental queue https://review.openstack.org/316544 | 03:06 |
| *** salv-orlando has joined #openstack-infra | 03:07 | |
| *** piet_ has quit IRC | 03:11 | |
| *** piet_ has joined #openstack-infra | 03:11 | |
| *** salv-orlando has quit IRC | 03:12 | |
| *** fawadkhaliq has joined #openstack-infra | 03:12 | |
| *** lakshmiS has joined #openstack-infra | 03:13 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Remove cap on oslo.db version https://review.openstack.org/318199 | 03:14 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Move oslo.concurrency to test-requirements https://review.openstack.org/318771 | 03:14 |
| *** salv-orlando has joined #openstack-infra | 03:16 | |
| *** EricGonczer_ has quit IRC | 03:18 | |
| *** salv-orlando has quit IRC | 03:18 | |
| *** lakshmiS has quit IRC | 03:19 | |
| *** psachin has joined #openstack-infra | 03:23 | |
| *** jpr1 has quit IRC | 03:29 | |
| *** greghaynes has quit IRC | 03:31 | |
| *** bkero has quit IRC | 03:31 | |
| *** ramishra has quit IRC | 03:35 | |
| *** ramishra has joined #openstack-infra | 03:36 | |
| *** fawadkhaliq has quit IRC | 03:38 | |
| *** ramishra has quit IRC | 03:38 | |
| *** ramishra has joined #openstack-infra | 03:39 | |
| *** amotoki_ has joined #openstack-infra | 03:39 | |
| *** ramishra has quit IRC | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:41 | |
| *** amotoki has quit IRC | 03:41 | |
| *** banix has quit IRC | 03:43 | |
| *** ramishra has joined #openstack-infra | 03:44 | |
| *** yuanying has quit IRC | 03:45 | |
| *** yuanying has joined #openstack-infra | 03:45 | |
| *** yuanying has quit IRC | 03:47 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| *** vhosakot has quit IRC | 03:48 | |
| *** harlowja_at_home has joined #openstack-infra | 03:49 | |
| *** x00350071 is now known as xiangxinyong | 03:50 | |
| *** shashank_hegde has quit IRC | 03:53 | |
| *** piet_ has quit IRC | 03:53 | |
| *** amotoki_ has quit IRC | 03:53 | |
| *** shashank_hegde has joined #openstack-infra | 03:53 | |
| *** Sukhdev has joined #openstack-infra | 03:54 | |
| *** amotoki has joined #openstack-infra | 03:58 | |
| *** jamesmcarthur has joined #openstack-infra | 03:58 | |
| *** armax has quit IRC | 03:58 | |
| *** thorst has joined #openstack-infra | 03:59 | |
| *** sdake has quit IRC | 04:00 | |
| *** oanson has joined #openstack-infra | 04:01 | |
| *** shashank_hegde has quit IRC | 04:03 | |
| *** psachin has quit IRC | 04:04 | |
| *** piet_ has joined #openstack-infra | 04:06 | |
| *** thorst has quit IRC | 04:06 | |
| *** harlowja_at_home has quit IRC | 04:07 | |
| *** shashank_hegde has joined #openstack-infra | 04:10 | |
| *** asselin__ has joined #openstack-infra | 04:11 | |
| *** links has joined #openstack-infra | 04:13 | |
| *** asselin has quit IRC | 04:15 | |
| *** ccamacho1 has joined #openstack-infra | 04:15 | |
| *** ccamacho has quit IRC | 04:15 | |
| *** fawadkhaliq has joined #openstack-infra | 04:16 | |
| *** amotoki has quit IRC | 04:17 | |
| *** psachin has joined #openstack-infra | 04:17 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Make sorting consistent on smaller devices https://review.openstack.org/318985 | 04:17 |
| *** bhunter71 has quit IRC | 04:20 | |
| *** flepied has joined #openstack-infra | 04:20 | |
| *** piet_ has quit IRC | 04:23 | |
| *** piet has joined #openstack-infra | 04:23 | |
| *** Sukhdev has quit IRC | 04:25 | |
| *** Sukhdev has joined #openstack-infra | 04:26 | |
| *** sdake has joined #openstack-infra | 04:28 | |
| *** gomarivera has joined #openstack-infra | 04:29 | |
| *** sdake_ has joined #openstack-infra | 04:30 | |
| *** yamahata has joined #openstack-infra | 04:33 | |
| *** gyee has quit IRC | 04:33 | |
| *** yamamoto has quit IRC | 04:34 | |
| *** sdake has quit IRC | 04:34 | |
| *** gomarivera has quit IRC | 04:34 | |
| *** amotoki has joined #openstack-infra | 04:34 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add cloud-initramfs-growroot for Precise https://review.openstack.org/318461 | 04:34 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add release to pkg-map https://review.openstack.org/319049 | 04:34 |
| *** salv-orlando has joined #openstack-infra | 04:37 | |
| *** mrmartin has joined #openstack-infra | 04:37 | |
| *** javeriak has joined #openstack-infra | 04:38 | |
| *** jamesmcarthur has quit IRC | 04:38 | |
| *** jcoufal has joined #openstack-infra | 04:39 | |
| *** gildub has joined #openstack-infra | 04:41 | |
| *** shashank_hegde has quit IRC | 04:43 | |
| *** jamesmcarthur has joined #openstack-infra | 04:45 | |
| *** shashank_hegde has joined #openstack-infra | 04:46 | |
| *** oanson has quit IRC | 04:46 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 04:49 |
| *** mixos has quit IRC | 04:51 | |
| *** jamesmcarthur has quit IRC | 04:53 | |
| *** salv-orlando has quit IRC | 04:53 | |
| *** mrmartin has quit IRC | 04:56 | |
| *** javeriak has quit IRC | 04:56 | |
| *** javeriak has joined #openstack-infra | 04:58 | |
| *** kushal has joined #openstack-infra | 04:59 | |
| *** piet has quit IRC | 05:01 | |
| *** rlandy has quit IRC | 05:01 | |
| *** jaosorior has joined #openstack-infra | 05:02 | |
| *** thorst has joined #openstack-infra | 05:03 | |
| openstackgerrit | Madhuri Kumari proposed openstack-infra/project-config: Add new project Higginsclient to openstack repo https://review.openstack.org/317699 | 05:03 |
| *** Nakato has quit IRC | 05:07 | |
| *** thorst has quit IRC | 05:11 | |
| *** timl0101 has quit IRC | 05:12 | |
| *** javeriak_ has joined #openstack-infra | 05:13 | |
| *** fawadkhaliq has quit IRC | 05:14 | |
| *** javeriak has quit IRC | 05:15 | |
| *** yamamoto has joined #openstack-infra | 05:19 | |
| *** flepied has quit IRC | 05:21 | |
| openstackgerrit | Madhuri Kumari proposed openstack-infra/project-config: Change Magnum Description https://review.openstack.org/319053 | 05:30 |
| *** Sukhdev has quit IRC | 05:33 | |
| openstackgerrit | Madhuri Kumari proposed openstack-infra/project-config: Change Magnum Description https://review.openstack.org/319053 | 05:33 |
| *** salv-orlando has joined #openstack-infra | 05:40 | |
| *** sree has joined #openstack-infra | 05:41 | |
| *** rcernin has joined #openstack-infra | 05:48 | |
| *** Qiming has quit IRC | 06:00 | |
| *** greghaynes has joined #openstack-infra | 06:02 | |
| *** Qiming has joined #openstack-infra | 06:02 | |
| *** YorikSar_ has quit IRC | 06:03 | |
| *** YorikSar has joined #openstack-infra | 06:05 | |
| *** thorst has joined #openstack-infra | 06:09 | |
| *** amotoki has quit IRC | 06:13 | |
| *** salv-orl_ has joined #openstack-infra | 06:14 | |
| *** kushal has quit IRC | 06:14 | |
| *** aeng has quit IRC | 06:15 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: [WIP] Add: lvm element https://review.openstack.org/316529 | 06:16 |
| *** thorst has quit IRC | 06:16 | |
| *** salv-orlando has quit IRC | 06:17 | |
| *** rossella_s has quit IRC | 06:24 | |
| *** fawadkhaliq has joined #openstack-infra | 06:24 | |
| *** rossella_s has joined #openstack-infra | 06:24 | |
| *** flepied has joined #openstack-infra | 06:25 | |
| *** david-lyle_ has joined #openstack-infra | 06:26 | |
| *** SlickNik has quit IRC | 06:26 | |
| *** bmjen has quit IRC | 06:26 | |
| *** timburke has quit IRC | 06:26 | |
| *** bmjen has joined #openstack-infra | 06:26 | |
| *** SlickNik has joined #openstack-infra | 06:26 | |
| *** szaher has quit IRC | 06:27 | |
| *** ptoohill has quit IRC | 06:28 | |
| *** david-lyle has quit IRC | 06:28 | |
| *** boris-42 has quit IRC | 06:28 | |
| *** SamYaple has quit IRC | 06:28 | |
| *** LaT0rtue has joined #openstack-infra | 06:29 | |
| *** boris-42 has joined #openstack-infra | 06:29 | |
| *** SamYaple has joined #openstack-infra | 06:29 | |
| *** szaher has joined #openstack-infra | 06:29 | |
| *** flepied has quit IRC | 06:30 | |
| *** flepied has joined #openstack-infra | 06:31 | |
| *** Nakato has joined #openstack-infra | 06:31 | |
| *** ptoohill has joined #openstack-infra | 06:32 | |
| *** timburke has joined #openstack-infra | 06:32 | |
| *** jiggalator has joined #openstack-infra | 06:32 | |
| *** shardy has joined #openstack-infra | 06:35 | |
| *** netsin has quit IRC | 06:35 | |
| *** mrmartin has joined #openstack-infra | 06:36 | |
| *** mrmartin has quit IRC | 06:38 | |
| *** jokke_ has quit IRC | 06:45 | |
| *** jokke_ has joined #openstack-infra | 06:45 | |
| *** sflanigan has joined #openstack-infra | 06:46 | |
| *** salv-orl_ has quit IRC | 06:47 | |
| *** arxcruz has joined #openstack-infra | 06:52 | |
| *** nadya has joined #openstack-infra | 06:58 | |
| *** jaosorior has quit IRC | 07:00 | |
| *** jaosorior has joined #openstack-infra | 07:01 | |
| *** nadya has quit IRC | 07:01 | |
| *** jtomasek has quit IRC | 07:01 | |
| *** mrmartin has joined #openstack-infra | 07:10 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-generic-switch-dsvm: Enable q-l3 https://review.openstack.org/319085 | 07:11 |
| *** kushal has joined #openstack-infra | 07:13 | |
| *** thorst has joined #openstack-infra | 07:14 | |
| yolanda | good morning | 07:14 |
| *** zeih has joined #openstack-infra | 07:15 | |
| *** ihrachys has joined #openstack-infra | 07:19 | |
| *** thorst has quit IRC | 07:22 | |
| dmellado | morning yolanda ;) | 07:24 |
| *** hichihara has quit IRC | 07:24 | |
| *** akaszuba has joined #openstack-infra | 07:25 | |
| *** mikelk has joined #openstack-infra | 07:25 | |
| *** esikachev has joined #openstack-infra | 07:26 | |
| *** mrmartin has quit IRC | 07:27 | |
| *** ihrachys has quit IRC | 07:28 | |
| *** Qiming has quit IRC | 07:29 | |
| *** jlanoux has joined #openstack-infra | 07:29 | |
| *** vincentll has joined #openstack-infra | 07:30 | |
| *** salv-orlando has joined #openstack-infra | 07:32 | |
| *** slaweq has joined #openstack-infra | 07:33 | |
| *** bauzas is now known as bauwser | 07:34 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add volume quotas support https://review.openstack.org/319096 | 07:36 |
| *** savihou has joined #openstack-infra | 07:36 | |
| *** matrohon has joined #openstack-infra | 07:39 | |
| *** yamahata has quit IRC | 07:41 | |
| *** sarob has joined #openstack-infra | 07:42 | |
| *** sarob has quit IRC | 07:47 | |
| *** Qiming has joined #openstack-infra | 07:47 | |
| *** dtantsur|afk is now known as dtantsur | 07:47 | |
| *** javeriak_ has quit IRC | 07:48 | |
| *** electrofelix has joined #openstack-infra | 07:49 | |
| *** jamielennox is now known as jamielennox|away | 07:49 | |
| *** electrofelix has quit IRC | 07:50 | |
| *** javeriak has joined #openstack-infra | 07:56 | |
| *** Qiming has quit IRC | 07:56 | |
| *** markvoelker has joined #openstack-infra | 07:56 | |
| *** nadya has joined #openstack-infra | 07:58 | |
| *** ihrachys has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** nadya has quit IRC | 08:02 | |
| *** ihrachys_ has joined #openstack-infra | 08:02 | |
| *** daemontool has joined #openstack-infra | 08:03 | |
| *** ihrachy__ has joined #openstack-infra | 08:04 | |
| *** ihrachys has quit IRC | 08:05 | |
| *** Qiming has joined #openstack-infra | 08:05 | |
| *** akaszuba has quit IRC | 08:05 | |
| *** markvoelker has quit IRC | 08:05 | |
| *** ihrachy__ has quit IRC | 08:05 | |
| *** zhurong_ has joined #openstack-infra | 08:07 | |
| *** ifarkas has joined #openstack-infra | 08:07 | |
| *** ihrachys_ has quit IRC | 08:07 | |
| *** gildub has quit IRC | 08:08 | |
| *** shashank_hegde has quit IRC | 08:10 | |
| *** zhurong has quit IRC | 08:10 | |
| *** DevBox has joined #openstack-infra | 08:10 | |
| *** chem`` has joined #openstack-infra | 08:12 | |
| *** chem` has quit IRC | 08:14 | |
| *** amotoki has joined #openstack-infra | 08:15 | |
| *** salv-orlando has quit IRC | 08:15 | |
| *** jistr has joined #openstack-infra | 08:15 | |
| *** mrmartin has joined #openstack-infra | 08:16 | |
| *** salv-orlando has joined #openstack-infra | 08:16 | |
| *** javeriak has quit IRC | 08:19 | |
| *** thorst has joined #openstack-infra | 08:19 | |
| *** asettle has joined #openstack-infra | 08:20 | |
| *** mrmartin has quit IRC | 08:20 | |
| *** alex_xu has joined #openstack-infra | 08:21 | |
| *** thorst has quit IRC | 08:26 | |
| *** markusry has joined #openstack-infra | 08:32 | |
| *** lucas-afk is now known as lucasagomes | 08:32 | |
| *** sflanigan has quit IRC | 08:40 | |
| *** nadya has joined #openstack-infra | 08:41 | |
| *** derekh has joined #openstack-infra | 08:44 | |
| *** nadya has quit IRC | 08:48 | |
| *** mhickey has joined #openstack-infra | 08:56 | |
| *** YorikSar has quit IRC | 08:58 | |
| *** YorikSar has joined #openstack-infra | 09:00 | |
| *** Na3iL has joined #openstack-infra | 09:00 | |
| *** daemontool has quit IRC | 09:05 | |
| *** daemontool has joined #openstack-infra | 09:05 | |
| *** jyuso1 has quit IRC | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:06 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add virtualbmc from source non-voting test https://review.openstack.org/315646 | 09:07 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add ironic cleaning job https://review.openstack.org/304473 | 09:11 |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add network quotas support https://review.openstack.org/319144 | 09:12 |
| *** dmk0202 has joined #openstack-infra | 09:19 | |
| *** Ravikiran_K has joined #openstack-infra | 09:20 | |
| *** thorst has joined #openstack-infra | 09:24 | |
| *** sambetts|afk is now known as sambetts | 09:24 | |
| *** Na3iL has quit IRC | 09:31 | |
| *** thorst has quit IRC | 09:31 | |
| *** savihou has quit IRC | 09:33 | |
| *** Na3iL has joined #openstack-infra | 09:33 | |
| *** jtomasek_ has joined #openstack-infra | 09:34 | |
| *** zhurong_ has quit IRC | 09:36 | |
| *** matrohon has quit IRC | 09:36 | |
| *** mhickey has quit IRC | 09:37 | |
| *** BobBall_AWOL is now known as BobBall | 09:38 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update 60-setup-testenvs to match what we use https://review.openstack.org/315290 | 09:39 |
| openstackgerrit | chenying proposed openstack-infra/project-config: Add smaug fullstack job https://review.openstack.org/317566 | 09:42 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Disable automated cleaning feature in ironic-inspector jobs https://review.openstack.org/319165 | 09:44 |
| openstackgerrit | Shu Muto proposed openstack-infra/project-config: Add Zaqar-UI translation settings https://review.openstack.org/319166 | 09:46 |
| *** ccamacho1 is now known as ccamacho | 09:51 | |
| *** devananda has quit IRC | 09:52 | |
| *** nadya has joined #openstack-infra | 09:56 | |
| *** nadya has quit IRC | 10:00 | |
| *** nadya has joined #openstack-infra | 10:02 | |
| *** camunoz_ has quit IRC | 10:02 | |
| *** devananda has joined #openstack-infra | 10:03 | |
| *** abregman|nb is now known as abregman|afk | 10:05 | |
| *** amotoki has quit IRC | 10:05 | |
| *** sshnaidm has quit IRC | 10:06 | |
| *** sarob has joined #openstack-infra | 10:09 | |
| *** amotoki has joined #openstack-infra | 10:13 | |
| *** sdague has joined #openstack-infra | 10:14 | |
| *** sarob has quit IRC | 10:14 | |
| *** ihrachys has joined #openstack-infra | 10:15 | |
| *** LaT0rtue has quit IRC | 10:19 | |
| *** Qiming has quit IRC | 10:19 | |
| *** ihrachys has quit IRC | 10:20 | |
| *** rhallisey has joined #openstack-infra | 10:21 | |
| *** gildub has joined #openstack-infra | 10:21 | |
| *** javeriak has joined #openstack-infra | 10:23 | |
| *** rossella_s has quit IRC | 10:24 | |
| *** rossella_s has joined #openstack-infra | 10:24 | |
| *** ihrachys has joined #openstack-infra | 10:25 | |
| *** claudiub has joined #openstack-infra | 10:27 | |
| *** thorst has joined #openstack-infra | 10:28 | |
| *** nadya has quit IRC | 10:29 | |
| *** ihrachys has quit IRC | 10:31 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Allow the mirror server to be publicly accessible https://review.openstack.org/319188 | 10:33 |
| *** amotoki has quit IRC | 10:35 | |
| *** thorst has quit IRC | 10:36 | |
| *** csaba has quit IRC | 10:43 | |
| *** alaski is now known as lascii | 10:43 | |
| *** _amrith_ is now known as amrith | 10:48 | |
| *** LaT0rtue has joined #openstack-infra | 10:52 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:56 | |
| *** javeriak has quit IRC | 10:59 | |
| *** jaosorior has quit IRC | 11:04 | |
| *** nadya has joined #openstack-infra | 11:11 | |
| *** zzzeek has quit IRC | 11:12 | |
| *** kushal has quit IRC | 11:12 | |
| *** Qiming has joined #openstack-infra | 11:13 | |
| *** gildub has quit IRC | 11:16 | |
| *** nadya has quit IRC | 11:16 | |
| *** gildub has joined #openstack-infra | 11:16 | |
| *** Na3iL has quit IRC | 11:24 | |
| *** nadya has joined #openstack-infra | 11:24 | |
| *** Ravikiran_K has quit IRC | 11:26 | |
| *** dguitarbite has quit IRC | 11:28 | |
| *** sagausi has joined #openstack-infra | 11:29 | |
| *** ccamacho is now known as ccamacho|lunch | 11:33 | |
| *** daemontool has quit IRC | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** jpr has joined #openstack-infra | 11:38 | |
| robcresswell | Any chance of some eyes on this? https://review.openstack.org/#/c/316730/ It's giving us a ton of false failures right now. Temporary fix. | 11:39 |
| robcresswell | "us" being Horizon. | 11:39 |
| *** daemontool has joined #openstack-infra | 11:41 | |
| *** baoli has joined #openstack-infra | 11:43 | |
| *** baoli_ has joined #openstack-infra | 11:44 | |
| Zara | we see similar problems with patches for the storyboard webclient; I don't feel confident that I understand the possible side-effects of upping the timeout, so I won't +1, but I'm lurking. | 11:45 |
| *** weshay has joined #openstack-infra | 11:46 | |
| *** sree has quit IRC | 11:46 | |
| *** baoli has quit IRC | 11:47 | |
| *** xenthree3 has joined #openstack-infra | 11:49 | |
| *** xenthree3 has left #openstack-infra | 11:49 | |
| *** ihrachys has joined #openstack-infra | 11:49 | |
| *** EricGonczer_ has joined #openstack-infra | 11:52 | |
| *** banix has joined #openstack-infra | 11:52 | |
| *** banix has quit IRC | 11:54 | |
| *** javeriak has joined #openstack-infra | 11:54 | |
| *** raildo-afk is now known as raildo | 11:55 | |
| *** rfolco has joined #openstack-infra | 11:56 | |
| *** sagausi has quit IRC | 11:57 | |
| *** thorst has joined #openstack-infra | 12:00 | |
| *** markusry has quit IRC | 12:01 | |
| *** julim has joined #openstack-infra | 12:02 | |
| *** eliqiao has quit IRC | 12:02 | |
| *** lucasagomes is now known as lucas-hungry | 12:02 | |
| *** deadnull_ has joined #openstack-infra | 12:03 | |
| *** flwang has quit IRC | 12:03 | |
| *** deadnull_ has quit IRC | 12:03 | |
| *** vincentll has quit IRC | 12:04 | |
| *** deadnull_ has joined #openstack-infra | 12:04 | |
| *** eliqiao has joined #openstack-infra | 12:05 | |
| *** jtomasek_ has quit IRC | 12:05 | |
| *** deadnull_ has quit IRC | 12:05 | |
| *** claudiub has quit IRC | 12:05 | |
| *** jtomasek_ has joined #openstack-infra | 12:05 | |
| *** deadnull_ has joined #openstack-infra | 12:05 | |
| *** aysyd has joined #openstack-infra | 12:06 | |
| *** claudiub has joined #openstack-infra | 12:06 | |
| kamsz | Shrews: is it possible to update quotas through shade? v've seen PR in ansible that leverages shade, but i'm having a hard time finding method in the documentation that was used | 12:07 |
| *** EricGonczer_ has quit IRC | 12:08 | |
| *** vincentll has joined #openstack-infra | 12:10 | |
| Shrews | kamsz: not yet. GheRivero has some patches up to add support. I just need to get around to reviewing them | 12:11 |
| *** ddieterly has joined #openstack-infra | 12:11 | |
| *** flwang has joined #openstack-infra | 12:11 | |
| *** EricGonczer_ has joined #openstack-infra | 12:13 | |
| *** daemontool_ has joined #openstack-infra | 12:14 | |
| *** ddieterly is now known as ddieterly[away] | 12:14 | |
| *** dprince has joined #openstack-infra | 12:14 | |
| kamsz | Shrews: alright, thanks | 12:15 |
| *** banix has joined #openstack-infra | 12:15 | |
| *** jtomasek_ has quit IRC | 12:16 | |
| *** ayoung has quit IRC | 12:16 | |
| *** jtomasek has joined #openstack-infra | 12:16 | |
| *** daemontool has quit IRC | 12:17 | |
| *** piet has joined #openstack-infra | 12:18 | |
| *** flepied has quit IRC | 12:19 | |
| *** amotoki has joined #openstack-infra | 12:22 | |
| *** psilvad has joined #openstack-infra | 12:24 | |
| *** trown|outtypewww is now known as trown | 12:25 | |
| GheRivero | kamsz: https://review.openstack.org/#/q/status:open+project:openstack-infra/shade+branch:master+topic:quotas | 12:26 |
| *** haypo has joined #openstack-infra | 12:26 | |
| haypo | hi. my change https://review.openstack.org/#/c/316890/ "Build also wheel packages for Python 3" was merged, but I see that gate-trove-python34-db still has to compile a lot of wheel packages: http://logs.openstack.org/91/313791/13/check/gate-trove-python34-db/fa9497b/tox/py34-1.log.txt | 12:27 |
| *** sylwesterB has left #openstack-infra | 12:27 | |
| *** banix has quit IRC | 12:27 | |
| haypo | does someone know if my change jenkins/scripts/wheel-build.sh is already effective or not? | 12:27 |
| GheRivero | there is a ansible module in progress for the compute quotas, and for the network and volume, I hope to have something today | 12:27 |
| haypo | does someone know what triggers the wheel mirror job? in http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/mako/ i only see Mako-1.0.4-py2-none-any.whl with the timestamp 2016-04-03 | 12:31 |
| *** deadnull_ has quit IRC | 12:31 | |
| *** amotoki has quit IRC | 12:32 | |
| *** salv-orl_ has joined #openstack-infra | 12:33 | |
| clarkb | haypo I think it runs periodically like once per day | 12:34 |
| clarkb | and probably has to succeed before the afs volume will publish changes | 12:36 |
| *** pilgrimstack1 has joined #openstack-infra | 12:36 | |
| *** sarob has joined #openstack-infra | 12:36 | |
| haypo | clarkb: are logs of this job public? i would like to check if it was executed and if it succeeded :) | 12:37 |
| *** flepied has joined #openstack-infra | 12:37 | |
| haypo | clarkb: this job would reduce the duration of most python3 job by 5-10 minutes! | 12:37 |
| *** EricGonczer_ has quit IRC | 12:37 | |
| *** salv-orlando has quit IRC | 12:37 | |
| haypo | (at least, for trove, i expect that the job will be 7 minutes shorter) | 12:38 |
| clarkb | yes should be under http://logs.openstack.org/periodic | 12:38 |
| *** pilgrimstack has quit IRC | 12:38 | |
| odyssey4me | yolanda If you have a moment, could you please take a look at https://review.openstack.org/317931 & https://review.openstack.org/318836 ? | 12:39 |
| *** markusry has joined #openstack-infra | 12:40 | |
| *** binbincong has joined #openstack-infra | 12:40 | |
| *** piet has quit IRC | 12:40 | |
| *** sarob has quit IRC | 12:41 | |
| robcresswell | Zara: Ah fair enough. Yeah, I left my thoughts as a comment on the patch. I'm not terribly happy about it, but need to get the gate moving for the contributors while the integration test guys figure out a more stable solution. | 12:41 |
| *** bhunter71 has joined #openstack-infra | 12:42 | |
| *** gordc has joined #openstack-infra | 12:43 | |
| *** piet has joined #openstack-infra | 12:43 | |
| *** ddieterly[away] is now known as ddieterly | 12:43 | |
| haypo | clarkb: oh, "pip3: command not found" :) it explains why nothing was built :) | 12:44 |
| *** rlandy has joined #openstack-infra | 12:45 | |
| Zara | robcresswell: yeah, I want to see how it goes since it affects us too, so I'll keep watching! :) I just don't know enough about the gate in general to know if lengthening test-times will cause anyone else problems, so I'd need to learn more about that first. our tests have been failing all over the place today, grr. | 12:46 |
| *** _prad_ has joined #openstack-infra | 12:46 | |
| *** marcusvrn_ has joined #openstack-infra | 12:47 | |
| *** Na3iL has joined #openstack-infra | 12:49 | |
| *** julim has quit IRC | 12:49 | |
| *** savihou has joined #openstack-infra | 12:54 | |
| *** piet has quit IRC | 12:55 | |
| *** matt-borland has joined #openstack-infra | 12:55 | |
| *** salv-orl_ has quit IRC | 12:55 | |
| *** psachin has quit IRC | 12:56 | |
| *** julim has joined #openstack-infra | 12:57 | |
| *** salv-orlando has joined #openstack-infra | 12:58 | |
| *** tlian has joined #openstack-infra | 12:58 | |
| *** annegentle has joined #openstack-infra | 12:59 | |
| *** ddieterly has quit IRC | 12:59 | |
| *** lucas-hungry is now known as lucasagomes | 13:00 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos https://review.openstack.org/319234 | 13:04 |
| *** Hal1 has quit IRC | 13:07 | |
| *** Hal1 has joined #openstack-infra | 13:07 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Fix wheel-build.sh for Python 3 https://review.openstack.org/319236 | 13:07 |
| *** javeriak has quit IRC | 13:08 | |
| openstackgerrit | Martin André proposed openstack-infra/project-config: Add tripleo-validations project to tripleo team https://review.openstack.org/319237 | 13:08 |
| *** daemontool_ has quit IRC | 13:09 | |
| *** xyang1 has joined #openstack-infra | 13:10 | |
| *** gildub has quit IRC | 13:11 | |
| odyssey4me | thanks yolanda | 13:11 |
| yolanda | odyssey4me, np | 13:12 |
| openstackgerrit | Vitaly Kramskikh proposed openstack-infra/project-config: Switch npm-lint to non-voting for fuel-ui https://review.openstack.org/319242 | 13:12 |
| vkramskikh | krotscheck: ^ I think we should disable voting for fuel-ui linting job until all the issues (I know at least 3) with NPM are resolved - we're almost blocked this week | 13:13 |
| fungi | haypo: i think you need `python3 -m pip ...` instead of `pip3 ...` | 13:14 |
| haypo | fungi: hey, see 15:07 < openstackgerrit> Victor Stinner proposed openstack-infra/project-config: Fix wheel-build.sh for Python 3 https://review.openstack.org/319236 | 13:15 |
| *** _ari_ has joined #openstack-infra | 13:15 | |
| fungi | haypo: ahh, yep, looks like that's exactly what you did | 13:15 |
| haypo | fungi: the problem is that i'm unable to fix if python3 and/or pip for python3 are installed | 13:16 |
| haypo | fungi: yep | 13:16 |
| haypo | fungi: some weeks ago, i read that "pip" command should not be used, it can lead to issues, especially with virtualenv | 13:16 |
| haypo | fungi: "python -m pip" (python2 -m pip, python2.7 -m pip, etc.) is preferred | 13:16 |
| fungi | haypo: there is no "pip for python 3" | 13:17 |
| fungi | haypo: pip3 is just a distro fabrication (a wrapper to invoke pip under a non-default copy of the interpreter) | 13:17 |
| *** vincentll has quit IRC | 13:18 | |
| haypo | fungi: pip should be installed once for python 2, and once for python 3, no? | 13:18 |
| *** vincentll has joined #openstack-infra | 13:18 | |
| fungi | nope | 13:18 |
| pabelanger | morning | 13:18 |
| haypo | fungi: i'm talking about /usr/lib/python3.4/site-packages/pip/ | 13:18 |
| fungi | haypo: right, we don't use distro packaged pip | 13:18 |
| fungi | haypo: we have pip installed via the bootstrap script for it. ends up in /usr/local | 13:19 |
| haypo | fungi: if you only install pip for python 2, python3 -m pip will fail, no? | 13:19 |
| fungi | haypo: we don't install pip "for" any specific interpreter | 13:19 |
| *** salv-orl_ has joined #openstack-infra | 13:19 | |
| haypo | fungi: i'm asking if pip is also installed for python 3 (i don't care of /usr/bin/pip*) | 13:19 |
| *** mdrabe has joined #openstack-infra | 13:20 | |
| fungi | pip is pure python and works under both python and python3 interpreters just fine | 13:20 |
| haypo | fungi: ah. do you mean that "python2 -m pip" command can be used to install a package for python 3 ? | 13:20 |
| fungi | pip is _installed_ for any python interpreter you care to call it under | 13:20 |
| fungi | haypo: don't use "python2" (python 2.x is called "python" according to the upstream python community, and python 3.x is called "python3") | 13:22 |
| *** salv-orlando has quit IRC | 13:22 | |
| haypo | "python" is sometimes "python3", like in virtual environements :) | 13:22 |
| pabelanger | I am cleaning up jenkins01 | 13:22 |
| haypo | fungi: for my change, i'm ok to write "python" if you prefer | 13:23 |
| pabelanger | there is a large number of ready nodes attached to the server | 13:23 |
| pabelanger | over 120 nodes | 13:23 |
| fungi | some distros made the regrettable decision to symlink /usr/bin/python to python3 in their installations, and yes it can be a link to python3 in a virtualenv but that's a rarified environmen | 13:23 |
| fungi | rarefied | 13:23 |
| pabelanger | I've stopped jenkins01 and in the process of deleting the nodes from nodepool | 13:23 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Cleanup old Trainers and Evoque meetings https://review.openstack.org/317420 | 13:24 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Make networking-midonet meeting every two weeks https://review.openstack.org/317282 | 13:24 |
| *** javeriak has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Cinder-Nova-API-interaction meeting series https://review.openstack.org/317038 | 13:24 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add the new Requirements team meeting https://review.openstack.org/318043 | 13:25 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove the compass meeting https://review.openstack.org/318899 | 13:25 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove cue meeting https://review.openstack.org/318908 | 13:25 |
| *** esikachev has quit IRC | 13:25 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove hierarchical-multitenancy-meeting https://review.openstack.org/318912 | 13:25 |
| *** Na3iL has quit IRC | 13:26 | |
| pabelanger | jenkins01.o.o started again | 13:26 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove mercador meeting https://review.openstack.org/318915 | 13:27 |
| odyssey4me | pabelanger hrm, I've clearly not quite done https://review.openstack.org/319234 right - can you advise how better to do it? I didn't want to have this be enforced on all roles | 13:28 |
| *** mhickey has joined #openstack-infra | 13:28 | |
| *** cloudtrainme has joined #openstack-infra | 13:29 | |
| *** amitgandhinz has joined #openstack-infra | 13:29 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: poppy: Add meeting_id, chair name, & better description https://review.openstack.org/318918 | 13:29 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add chair IRC nicks to Ironic meetings https://review.openstack.org/318924 | 13:29 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Enable returning run id in get_test_runs_by_status_for_run_ids() https://review.openstack.org/312298 | 13:29 |
| pabelanger | odyssey4me: you are missing the change for JJB. So zuul cannot find the job in jenkins. | 13:30 |
| odyssey4me | pabelanger it needs a change in jenkins/ clearly, but I wasn't sure where | 13:31 |
| pabelanger | odyssey4me: you'll need to add the template to ansible-role-jobs.yaml | 13:31 |
| *** akshai has joined #openstack-infra | 13:31 | |
| pabelanger | odyssey4me: should be able to use ansible-role-jobs template | 13:31 |
| *** dansmith is now known as superdan | 13:31 | |
| *** anteaya has joined #openstack-infra | 13:31 | |
| anteaya | I was on #openstick-infra for a minute | 13:32 |
| anteaya | it is a lot quieter in there | 13:32 |
| odyssey4me | lol | 13:32 |
| *** chuck__ is now known as zul | 13:33 | |
| *** ddieterly has joined #openstack-infra | 13:33 | |
| *** ayoung has joined #openstack-infra | 13:33 | |
| Zara | hehe | 13:33 |
| anteaya | :) | 13:34 |
| *** thorst has quit IRC | 13:34 | |
| odyssey4me | pabelanger would it be better to just add '{name}-announce-release' to ansible-role-jobs for the release queue - wouldn't that then apply the release announcements to all roles, and thus put you in a fix for the ansible-role-* repositories? | 13:34 |
| *** piet has joined #openstack-infra | 13:34 | |
| *** rodrigods has quit IRC | 13:34 | |
| odyssey4me | I suppose for it to do anything you need to actually use the release process? | 13:34 |
| *** rodrigods has joined #openstack-infra | 13:34 | |
| *** piet has quit IRC | 13:34 | |
| pabelanger | odyssey4me: add where? | 13:35 |
| odyssey4me | pabelanger if it's added to jenkins/jobs/ansible-role-jobs.yaml for the ansible-role-jobs group | 13:36 |
| pabelanger | odyssey4me: if you add it to JJB, you are just saying create these jobs, which for the most part is fine. Add it to zuul, says you want to run it (the job) for these projects. So, your layout.yaml is correct (you have limited it to specific projects). You just need to generate all the jenkins jobs for all ansible roles. | 13:37 |
| pabelanger | odyssey4me: Yup, add '{name}-announce-release' and you should be fine | 13:37 |
| pabelanger | to ansible-role-jobs group in JJB | 13:38 |
| *** salv-orl_ has quit IRC | 13:38 | |
| anteaya | zuul's eth0 outbound is spiking and queue lengths are growing even though the top of the gate is 7 minutes away from being finished: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=403&rra_id=all | 13:39 |
| pabelanger | #status log I've stopped jenkins01.o.o, it doesn't appear to be working properly. Nodes attach to jenkins but are not launched by nodepool. I believe zl01 might be the issue | 13:39 |
| *** pabelanger has quit IRC | 13:39 | |
| *** pabelanger has joined #openstack-infra | 13:39 | |
| pabelanger | #status log I've stopped jenkins01.o.o, it doesn't appear to be working properly. Nodes attach to jenkins but are not launched by nodepool. I believe zl01 might be the issue | 13:40 |
| openstackstatus | pabelanger: finished logging | 13:40 |
| *** thorst has joined #openstack-infra | 13:40 | |
| pabelanger | jeblair: when you have a moment^ if you have some thoughts about jenkins01 | 13:40 |
| anteaya | zl01 doesn't seem to have a presence in cacti | 13:41 |
| odyssey4me | pabelanger oh hang on, I need corresponding jobs in jenkins/jobs/projects.yaml right? | 13:41 |
| pabelanger | odyssey4me: Right, but adding it into ansible-role-jobs.yaml is the easier way. Otherwise, you need to manually add the announce-release to specific projects | 13:42 |
| *** ddieterly is now known as ddieterly[away] | 13:42 | |
| pabelanger | anteaya: Ya, jeblair is still testing the system. And likely overlooked by mistake | 13:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos https://review.openstack.org/319234 | 13:43 |
| odyssey4me | pabelanger ^ like that then? | 13:44 |
| *** daemontool has joined #openstack-infra | 13:44 | |
| *** thorst has quit IRC | 13:44 | |
| anteaya | pabelanger: okay | 13:44 |
| pabelanger | odyssey4me: if you go that route, you need to add it to the other projects you've tagged in zuul too | 13:44 |
| pabelanger | odyssey4me: moving it to ansible-role-jobs.yaml, it will be a single line patch. | 13:45 |
| anteaya | zuul queued events have dropped | 13:45 |
| *** thorst has joined #openstack-infra | 13:46 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Fix wheel-build.sh for Python 3 https://review.openstack.org/319236 | 13:46 |
| *** links has quit IRC | 13:46 | |
| *** thorst has quit IRC | 13:47 | |
| *** thorst has joined #openstack-infra | 13:47 | |
| odyssey4me | pabelanger ah, so just add to ansible-role-jobs, then remove the 'release-announce-jobs' template entries per-repo because it's included with the 'ansible-role-jobs' template, and also there's no need for the 'release-announce-jobs' macro at all? | 13:48 |
| pabelanger | odyssey4me: I'll comment on the review | 13:48 |
| odyssey4me | pabelanger thanks | 13:49 |
| *** dtardivel has joined #openstack-infra | 13:49 | |
| *** kgiusti has joined #openstack-infra | 13:49 | |
| pabelanger | odyssey4me: comment left | 13:51 |
| odyssey4me | pabelanger replied - that particular repo is a special case because it doesn't use ansible-role-jobs for test execution | 13:52 |
| *** zhurong has joined #openstack-infra | 13:52 | |
| krotscheck | vkramskikh: Well, the hope was that https://review.openstack.org/#/c/318875/ would land. | 13:53 |
| krotscheck | vkramskikh: because it's not just you. | 13:54 |
| pabelanger | odyssey4me: replied | 13:54 |
| krotscheck | But no cores have looked at it. | 13:54 |
| krotscheck | Has anyone figured out what is up with our mirrors returning partial files? | 13:54 |
| openstackgerrit | Kirill Bespalov proposed openstack-infra/project-config: add reno jobs for oslo.messaging https://review.openstack.org/319259 | 13:55 |
| odyssey4me | pabelanger Is it not already in ansible-role-jobs? https://review.openstack.org/#/c/319234/2/jenkins/jobs/ansible-role-jobs.yaml | 13:55 |
| krotscheck | infra-core: I'd like to disable (and unblock) the javascript jobs by temporarily switching back over to the public npm mirrors. https://review.openstack.org/#/c/318875/ | 13:55 |
| *** esikachev has joined #openstack-infra | 13:55 | |
| pabelanger | odyssey4me: Oh, ya. Some how I missed that | 13:55 |
| pabelanger | odyssey4me: I blame not having coffee | 13:55 |
| odyssey4me | :) | 13:55 |
| pabelanger | odyssey4me: jenkins should be happy now | 13:55 |
| * odyssey4me hands coffee to pabelanger | 13:56 | |
| *** sarob has joined #openstack-infra | 13:56 | |
| *** sarob has quit IRC | 13:56 | |
| *** kushal has joined #openstack-infra | 13:56 | |
| *** sarob has joined #openstack-infra | 13:57 | |
| *** thorst has quit IRC | 13:57 | |
| *** ddieterly[away] is now known as ddieterly | 13:58 | |
| vkramskikh | krotscheck: but without mirrors there will be ECONNREFUSED error and that "module not found" error in npm 3.5.2 from Xenial, so the job will still be not functional | 13:58 |
| *** thorst has joined #openstack-infra | 13:58 | |
| krotscheck | vkramskikh: You didn't have problems last week, right? | 13:58 |
| krotscheck | vkramskikh: Xenial went in 3 weeks ago. | 13:58 |
| *** _sarob has joined #openstack-infra | 13:58 | |
| *** ccamacho|lunch is now known as ccamacho | 14:00 | |
| vkramskikh | I had that ECONNREFUSED error. ok, I'll put workflow -1 on my patch and see if your patch resolves the issues | 14:00 |
| krotscheck | vkramskikh: We can do a quick check, just a minute | 14:00 |
| *** sarob has quit IRC | 14:01 | |
| krotscheck | vkramskikh: Updated 318687 | 14:01 |
| vkramskikh | thanks, let's see | 14:03 |
| *** rlandy has quit IRC | 14:05 | |
| *** daemontool has quit IRC | 14:06 | |
| *** jiggalator has quit IRC | 14:08 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:09 | |
| *** _sarob has quit IRC | 14:11 | |
| jeblair | pabelanger: i reassigned all nodes that nodepool sent to zuul to jenkins01, because nodepool can't handle a target being removed from its configuration | 14:12 |
| jeblair | pabelanger: they should get cleaned up eventually | 14:12 |
| pabelanger | jeblair: okay, so safe to bring jenkins01 back online now? | 14:13 |
| *** hemanthm|afk is now known as hemanthm | 14:14 | |
| jeblair | pabelanger: i think so -- if the only thing you saw was nodes in 'nodepool list' that didn't show up on jenkins itself, and those nodes had 'zl01' as the manager, then yeah, should be ok | 14:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting https://review.openstack.org/319273 | 14:14 |
| pabelanger | jeblair: okay, starting. Will monitor | 14:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting https://review.openstack.org/319273 | 14:16 |
| vkramskikh | krotscheck: it passed, i'll try a few more times | 14:17 |
| krotscheck | That works | 14:18 |
| Shrews | jeblair: any tips on trying to see the actual playbooks with your change in 318397? i mean, it looks fine, but would like to see the actual output somehow | 14:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting https://review.openstack.org/319273 | 14:20 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting https://review.openstack.org/319273 | 14:21 |
| jeblair | Shrews: so far, i've only done so by running it for real, and launching a job, after commenting out the bit that deletes the JobDir on completion. i think we shoul probably a) add an option to avoid deleting jobdirs :) and b) maybe make some kind of utility that will take a jjb config and construct the jobdir for it. that may take some minor refactoring. | 14:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos https://review.openstack.org/319234 | 14:22 |
| jeblair | (i think that will be a useful utility to have in v3 as well | 14:22 |
| odyssey4me | sorry about the spam, I edited the wrong patch :/ | 14:22 |
| *** yamamoto has joined #openstack-infra | 14:22 | |
| *** yamamot__ has joined #openstack-infra | 14:23 | |
| *** akshai has quit IRC | 14:23 | |
| Shrews | jeblair: ack | 14:23 |
| pabelanger | jeblair: if you want to take a quick peak at jenkins01, we've attached over 40 nodes now, but none are being used | 14:23 |
| *** regXboi has joined #openstack-infra | 14:24 | |
| *** sree has joined #openstack-infra | 14:24 | |
| *** pilgrimstack1 has quit IRC | 14:24 | |
| jeblair | pabelanger: did you start/stop jenkins itself? | 14:25 |
| pabelanger | jeblair: I did, I put it into shutdown and clean up the nodes this morning | 14:25 |
| pabelanger | jeblair: and just started jenkins01 now | 14:25 |
| jeblair | pabelanger: ah, it is running jobs, but nodepool isn't getting zmq events from it | 14:25 |
| jeblair | pabelanger: we should try restarting nodepool | 14:26 |
| jeblair | pabelanger: you want to do that? | 14:26 |
| pabelanger | jeblair: sure, I can | 14:26 |
| jeblair | (this is why we're never touching zmq again) | 14:26 |
| jeblair | there it goes | 14:27 |
| pabelanger | okay, stop / start on nodepool | 14:27 |
| jeblair | pabelanger: i see "Received: onCompleted" from jenkins01 in the nodepool log now | 14:27 |
| fungi | pabelanger: yeah, if you see a jenkins master with lots of in-use or disabled nodes which nodepool thinks are still in a ready state, that's generally a sign that something (network hiccup?) caused the zmq connection to go dead | 14:27 |
| *** yamamoto has quit IRC | 14:27 | |
| pabelanger | fungi: okay, good to know | 14:29 |
| *** Na3iL has joined #openstack-infra | 14:29 | |
| pabelanger | jeblair: fungi: will nodepool bleed of the current ready nodes on jenkins01 after some time? | 14:30 |
| jeblair | pabelanger: they'll get used and nodepool will get the complete event and delete them; ones that are completed already will take longer to atrophy; they may be worth a manual cleanup in an hour or so (when they will be easier to spot) | 14:31 |
| *** akshai has joined #openstack-infra | 14:31 | |
| jeblair | pabelanger: in the mean time, nodepool will under-provision since it thinks it has more nodes ready; if we are in dire straights, it might be good to manually clean up now. | 14:31 |
| pabelanger | jeblair: Okay, I have a shell script I can run quickly to delete the ready nodes | 14:32 |
| pabelanger | will take a minute | 14:32 |
| *** tonytan4ever has joined #openstack-infra | 14:32 | |
| jeblair | pabelanger: well, just realize that some of the nodes that nodepool thinks are ready are running jobs right now | 14:32 |
| jeblair | you have to either wait an hour or correlate it with what jenkins says | 14:32 |
| pabelanger | jeblair: okay, I'll wait for a bit | 14:33 |
| zaro | morning | 14:33 |
| mtreinish | jeblair, nibalizer, pabelanger: if you get a sec can you take a look at: https://review.openstack.org/312308 | 14:33 |
| fungi | pabelanger: yeah, i usually wait a couple hours and then start deleting any nodes which nodepool says have been in a "ready" state since around the time of the restart or longer | 14:34 |
| mtreinish | it adds the setup bits need for openstack-health to use elastic-recheck (per https://review.openstack.org/311919 ) | 14:34 |
| fungi | pabelanger: depending on how node-starved we are, i sometimes give it 3 hours since there are some ironic/tripleo jobs which run in the 2.5-hour neighborhood | 14:34 |
| *** banix has joined #openstack-infra | 14:36 | |
| scottda | Hi Infra Team. Cinder is wanting to enable multi-backend for Tempest single-node Devstack tests, a one-line change. Where is the best place to do this, in nodepool? | 14:40 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: net-generic-switch: Do not create Neutron debug probes https://review.openstack.org/318739 | 14:40 |
| *** trown is now known as trown|afk | 14:42 | |
| *** zeih has quit IRC | 14:43 | |
| *** zeih has joined #openstack-infra | 14:44 | |
| *** daemontool has joined #openstack-infra | 14:44 | |
| *** javeriak has quit IRC | 14:44 | |
| *** hongbin has joined #openstack-infra | 14:45 | |
| *** ramishra has quit IRC | 14:45 | |
| *** ramishra has joined #openstack-infra | 14:47 | |
| *** mtanino has joined #openstack-infra | 14:48 | |
| *** zeih has quit IRC | 14:49 | |
| pabelanger | jeblair: Do we change any of the default MTU setting on afsd? I am trying to debug some AFS failures in tripleo-cloud-test-rh1 which I think are related to the NAT in tripleo | 14:49 |
| zigo | fungi: jeblair: pabelanger: After 2 days, it still looks like some jenkins have the old version of the pkgos-build-openstack-pkg-tools script, as the check job works, but the gate one doesn't. Please see: https://review.openstack.org/#/c/318500/ | 14:49 |
| zigo | Both ScotK and Igot Belikov are saying it's probably a Jenkins that has an outdated version of the job. | 14:50 |
| jeblair | pabelanger: you're not running afsd on a pnat are you? | 14:50 |
| zigo | Since we don't have access to Jenkins anymore, I can't check... | 14:50 |
| *** inc0 has joined #openstack-infra | 14:51 | |
| *** kien-ha has joined #openstack-infra | 14:53 | |
| pabelanger | jeblair: the AFS client appears to be behind a RHEL box doing the NAT translation. That's how much I currently know | 14:53 |
| jeblair | pabelanger: does it have a public ip? | 14:54 |
| pabelanger | jeblair: yes, 66.187.229.129 you'll have credentials | 14:55 |
| *** kushal has quit IRC | 14:55 | |
| jeblair | pabelanger: so it's straight nat, not pnat? | 14:55 |
| pabelanger | jeblair: I believe to, let me confirm | 14:56 |
| pabelanger | jeblair: Yes, a straight nat | 14:57 |
| *** nwkarsten has joined #openstack-infra | 14:58 | |
| jeblair | pabelanger: what's the problem you're seeing? | 14:58 |
| *** arxcruz has quit IRC | 14:59 | |
| *** EricGonczer_ has joined #openstack-infra | 15:00 | |
| pabelanger | jeblair: afs is operating slow. A basic ls on /afs/openstack simply hangs. Here is a tcpdump of an attempt: https://paste.fedoraproject.org/368914/14637544/ | 15:00 |
| fungi | zigo: that example got its job definition from jenkins02. i'll check its configuration state | 15:01 |
| zigo | fungi: cheers! | 15:01 |
| fungi | betting we're timing out trying to apply a large set of job updates on our slower jenkins masters for some days now | 15:01 |
| *** EricGonc_ has joined #openstack-infra | 15:01 | |
| fungi | -rw-r--r-- 1 jenkins nogroup 4046 May 14 01:11 /var/lib/jenkins/jobs/pkgdeb-build-openstack-pkg-tools/config.xml | 15:02 |
| pabelanger | jeblair: okay, here is some more info about NAT: http://paste.openstack.org/show/497929/ | 15:02 |
| fungi | so it's been 6 days since that job was last modified there | 15:02 |
| pabelanger | jeblair: some ports are forwarded | 15:02 |
| *** tonytan4ever has quit IRC | 15:03 | |
| *** zhurong has quit IRC | 15:03 | |
| fungi | i'll start a jjb update on jenkins02 in a root screen session and check the other masters for similar signs of outdatedness | 15:03 |
| jeblair | pabelanger: i'm sorry. we only work with servers that are on the internet. that server is not on the internet. i would like to reiterate that i think it is past time for us to work on moving the tripleo cloud to a third-party ci model. | 15:04 |
| *** xarses has quit IRC | 15:04 | |
| fungi | oh, nice, broken jjb cache on jenkins02 | 15:04 |
| fungi | that might explain it | 15:05 |
| *** EricGonczer_ has quit IRC | 15:05 | |
| pabelanger | jeblair: okay, I am unsure how to proceed with moving tripleo to third party | 15:05 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Revert "Test with latest master heat" https://review.openstack.org/309051 | 15:05 |
| *** cloudtrainme has quit IRC | 15:06 | |
| *** javeriak has joined #openstack-infra | 15:06 | |
| fungi | pabelanger: tripleo sysadmins use puppet-openstackci to deploy zuul/jenkins/nodepool somewhere in their network and point it at a tenant in the cloud they're providing? | 15:07 |
| *** cody-somerville has joined #openstack-infra | 15:07 | |
| pabelanger | fungi: jeblair: Right, but they don't want to leave. That is the issue | 15:08 |
| jeblair | pabelanger: i think we have two choices -- we can either work with the tripleo folks on a timeline and help them move operation of the zuul/nodepool component internally (like what fungi said). or we can just say "we're going to turn it off in 4 months". i'd prefer the former, but i don't have time to drive it. i'm happy to help where i can though. | 15:08 |
| fungi | pabelanger: and hopefully they don't turn it into a rathole to get those puppet modules working on rhel simply so that they can avoid installing non-rh distros inside their environment, or else it will drag on forever | 15:08 |
| pabelanger | right, so I think it will have to come to that. Impose some sort of migration date | 15:09 |
| *** eventingmonkey has joined #openstack-infra | 15:10 | |
| *** asettle has quit IRC | 15:12 | |
| *** woodster_ has joined #openstack-infra | 15:12 | |
| zigo | fungi: Thanks a lot for your check... | 15:12 |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 15:12 |
| fungi | zigo: this will take a while to clean up, but i'll let you know once it's on track everywhere | 15:12 |
| zigo | fungi: Please ping me when I should re-attempt to do a "recheck". | 15:12 |
| fungi | will do! | 15:13 |
| zigo | Cheers. | 15:13 |
| openstackgerrit | Merged openstack-dev/hacking: Fix H703 test https://review.openstack.org/319014 | 15:13 |
| *** yamahata has joined #openstack-infra | 15:14 | |
| fungi | all the other jenkins masters seem to be reasonably up to date. i'm going to put jenkins02 into prepare for shutdown so that it doesn't run any new jobs until the jjb updates complete. that should get it over with sooner and also avoid using stale job configs in the meantime | 15:15 |
| *** akshai has quit IRC | 15:15 | |
| zigo | Good. | 15:16 |
| zigo | I just wonder why I got *always* the same result. Bad luck and always got to jenkins02 ? :P | 15:17 |
| fungi | well, it may not be the _only_ problem | 15:17 |
| fungi | but with this out of the way we can narrow the issue down more easily | 15:17 |
| *** markvoelker has joined #openstack-infra | 15:18 | |
| *** markvoelker has quit IRC | 15:18 | |
| *** markvoelker has joined #openstack-infra | 15:18 | |
| fungi | zigo: okay, jenkins02 is in prepare for shutdown mode now, so it shouldn't pick up any new jobs until we bring it back into full service. if you recheck that it will hopefully pass and if not then we can move on to investigating whatever the next issue is with it | 15:19 |
| *** xarses has joined #openstack-infra | 15:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH https://review.openstack.org/318975 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command https://review.openstack.org/318982 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP https://review.openstack.org/318979 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: several fixes https://review.openstack.org/318397 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers https://review.openstack.org/318411 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME https://review.openstack.org/318987 | 15:20 |
| *** akshai has joined #openstack-infra | 15:21 | |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 15:24 |
| *** shashank_hegde has joined #openstack-infra | 15:25 | |
| *** vincentll has quit IRC | 15:25 | |
| *** cloudtrainme has joined #openstack-infra | 15:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Estimate job runtimes internally https://review.openstack.org/318966 | 15:26 |
| *** roxanagh_ has joined #openstack-infra | 15:26 | |
| zigo | fungi: Ok, doing the recheck. Thanks again. | 15:26 |
| *** ri0 has joined #openstack-infra | 15:28 | |
| *** Sukhdev has joined #openstack-infra | 15:28 | |
| *** kushal has joined #openstack-infra | 15:28 | |
| *** Sukhdev has quit IRC | 15:29 | |
| *** Sukhdev has joined #openstack-infra | 15:30 | |
| *** daemontool_ has joined #openstack-infra | 15:31 | |
| *** lakshmiS has joined #openstack-infra | 15:31 | |
| *** sree has quit IRC | 15:33 | |
| *** Qiming has quit IRC | 15:33 | |
| *** mixos has joined #openstack-infra | 15:33 | |
| *** daemontool has quit IRC | 15:33 | |
| *** DevBox has quit IRC | 15:35 | |
| *** armax has joined #openstack-infra | 15:35 | |
| *** jsavak has joined #openstack-infra | 15:36 | |
| *** roxanagh_ has quit IRC | 15:37 | |
| *** mixos has quit IRC | 15:37 | |
| *** bkero has joined #openstack-infra | 15:38 | |
| *** mikelk has quit IRC | 15:38 | |
| *** jamesmcarthur has joined #openstack-infra | 15:39 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository https://review.openstack.org/319322 | 15:40 |
| *** jamesmcarthur has quit IRC | 15:41 | |
| *** kushal has quit IRC | 15:45 | |
| zaro | fungi: was able to figure out how to force running online re-index | 15:46 |
| *** ri0 has quit IRC | 15:46 | |
| *** jamesmcarthur has joined #openstack-infra | 15:47 | |
| *** Swami has joined #openstack-infra | 15:48 | |
| fungi | zaro: oh? what bit of magic were we missing last time we tried? | 15:48 |
| zaro | fungi: do you happen to know the number of cores on review.o.o? | 15:49 |
| fungi | zaro: i'm willing to give it a shot in the upcoming rename maintenance, but will still plan for an offline reindex in case online still doesn't do what we need | 15:49 |
| jeblair | --for-realzies ? :) | 15:49 |
| *** Nakato has quit IRC | 15:49 | |
| zaro | fungi: there's a groovy script that luca wrote to do it, https://gerrit.googlesource.com/plugins/scripts/+/master/admin/reindexer-1.0.groovy | 15:50 |
| fungi | zaro: `grep -c ^processor /proc/cpuinfo` says "16" | 15:50 |
| zaro | fungi: it works but the only problem is that it doesn't respect gerrit access rules | 15:51 |
| *** rcernin has quit IRC | 15:51 | |
| *** asettle has joined #openstack-infra | 15:52 | |
| krotscheck | Any other infra-cores able to look at https://review.openstack.org/#/c/318875/ ? Most of the JS jobs are borked because mirrors. | 15:52 |
| vkramskikh | https://review.openstack.org/#/c/318687/ 5 successful runs so far, so +1 for merge of ^ | 15:53 |
| zaro | fungi: i'm not sure if it's possible to enforce access rules from a script but i'm pretty sure not. | 15:53 |
| krotscheck | jeblair: Was that log from yesterday useful? | 15:53 |
| *** markvoelker has quit IRC | 15:54 | |
| *** jamesmcarthur has quit IRC | 15:54 | |
| jeblair | krotscheck: sorry i missed that. | 15:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add timeout to get_host_info task https://review.openstack.org/317937 | 15:55 |
| krotscheck | jeblair: No worries. I'm guessing the segfault problems you've run into are taking up your time? | 15:55 |
| jeblair | krotscheck: nope, i'm working on zuul | 15:55 |
| krotscheck | jeblair: Ah. hrm. Is anyone working on figuring out why we got partial downloads? | 15:56 |
| *** inc0 has quit IRC | 15:56 | |
| jeblair | krotscheck: i do not know the answer to that | 15:56 |
| *** asettle has quit IRC | 15:56 | |
| krotscheck | Alrightey | 15:56 |
| fungi | krotscheck: i'm thinking at this point the only additional info we're going to get out of apache on the random segfaults is to run with coredumps enabled and load those into a debugger to get backtraces | 15:56 |
| krotscheck | fungi: Hrm. This sounds like something better done on a non-infra vm. | 15:57 |
| fungi | krotscheck: i don't know whether we can reasonably recreate it without building a custom load generator otherwise | 15:57 |
| krotscheck | fungi: I can generate load, but I may not be able to generate infra-levels of load. | 15:58 |
| fungi | we're not doing anything particularly fancy with our apache configuration and have a fairly minimal set of standard modules in play | 15:58 |
| *** jamesmcarthur has joined #openstack-infra | 15:58 | |
| krotscheck | fungi: I'd be willing ot help, but I have zero experience reading coredumps. Or, well, anything below the web tier, really. | 15:59 |
| *** sdake_ has quit IRC | 15:59 | |
| *** Apoorva has joined #openstack-infra | 15:59 | |
| fungi | yeah, i've got the apache docs in hand and am going to give it a whirl here shortly. just want to make sure it has my undivided attention in case it causes any unexpected service disruption | 16:00 |
| *** cody-somerville has quit IRC | 16:00 | |
| *** sdake has joined #openstack-infra | 16:01 | |
| *** salv-orlando has joined #openstack-infra | 16:02 | |
| openstackgerrit | Mário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances https://review.openstack.org/319333 | 16:02 |
| *** slaweq has quit IRC | 16:03 | |
| *** cody-somerville has joined #openstack-infra | 16:03 | |
| *** cody-somerville has quit IRC | 16:03 | |
| *** cody-somerville has joined #openstack-infra | 16:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily start using the public registry again https://review.openstack.org/318875 | 16:03 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository https://review.openstack.org/319335 | 16:03 |
| *** vhosakot has joined #openstack-infra | 16:03 | |
| krotscheck | fungi: Alright, need me to get some load generation thingies handy? | 16:03 |
| fungi | krotscheck: it seems to happen with sufficient regularity on mirror.ord.rax.o.o; i'm going to see if it's happening elsewhere before i plan | 16:05 |
| zigo | fungi: It's looking green now. | 16:05 |
| zigo | fungi: How can I check that my post job really published something in tarballs.o.o ? | 16:05 |
| fungi | zigo: did the change merge? | 16:06 |
| zigo | fungi: Yup. | 16:06 |
| *** nadya has quit IRC | 16:06 | |
| *** salv-orlando has quit IRC | 16:06 | |
| *** e0ne has quit IRC | 16:06 | |
| *** gyee has joined #openstack-infra | 16:07 | |
| *** jlanoux has quit IRC | 16:07 | |
| fungi | zigo: so first is to look at the new branch tip | 16:07 |
| *** daemontool has joined #openstack-infra | 16:07 | |
| fungi | zigo: http://git.openstack.org/cgit/openstack/deb-openstack-pkg-tools/log/?h=debian/mitaka says the change merged without needing a merge commit, so we'll take an abbreviation of its git sha | 16:08 |
| fungi | zigo: in this case the first 7 characters of the sha are (amusingly) "220dead" | 16:08 |
| zigo | LOL ! :) | 16:08 |
| fungi | zigo: if it ended up needing a merge commit, you'd want to use teh sha of the merge commit instead | 16:08 |
| *** daemontool_ has quit IRC | 16:09 | |
| fungi | zigo: next go to http://status.openstack.org/zuul/ or parse the json from the zuul status api looking for that abbreviated sha | 16:09 |
| *** Sukhdev has quit IRC | 16:09 | |
| *** Na3iL has quit IRC | 16:10 | |
| fungi | zigo: if we're running a backlog in the post pipeline, it may be queued up there so you'd have to make sure to wait until its jobs run | 16:10 |
| zigo | I don't see it in the queue. | 16:11 |
| *** bpokorny has joined #openstack-infra | 16:11 | |
| zigo | Wouldn't the post job show in gerrit as well? | 16:11 |
| *** Sukhdev has joined #openstack-infra | 16:11 | |
| fungi | yeah, which is a little odd. i guess it's possible its jobs completed already | 16:11 |
| fungi | no, post pipeline results don't get added in gerrit comments but we can go find the logs | 16:11 |
| *** zz_dimtruck is now known as dimtruck | 16:12 | |
| fungi | zigo: look under http://logs.openstack.org/22/220dead31374edabced0d890bfaf2f900da4b66c/ | 16:12 |
| *** Sukhdev_ has joined #openstack-infra | 16:12 | |
| fungi | that's http://logs.openstack.org/ followed by the first two characters of the git sha and then the full git sha | 16:12 |
| *** sdake has quit IRC | 16:12 | |
| zigo | Looks like it uploaded the files ! :) | 16:12 |
| zigo | 2016-05-20 16:07:40.246 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1.dsc' | 16:12 |
| zigo | 2016-05-20 16:07:40.362 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1.tar.xz' | 16:12 |
| zigo | 2016-05-20 16:07:40.441 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1_all.deb' | 16:12 |
| zigo | 2016-05-20 16:07:40.520 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1_amd64.changes' | 16:12 |
| zigo | :) | 16:12 |
| fungi | i think since the debian-jessie nodes are only used for this project so far, you got a node assignment right away and the job completed | 16:13 |
| fungi | that's why it wasn't in the post pipeline backlog | 16:13 |
| *** javeriak has quit IRC | 16:13 | |
| zigo | Oh, nice ! :) | 16:13 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add grenade jobs to check pipeline for Ironic https://review.openstack.org/319336 | 16:13 |
| zigo | I got a VIP priority ! :) | 16:13 |
| pabelanger | http://tarballs.openstack.org/%7bproject%7d/ | 16:13 |
| fungi | so anyway, next take a look at... | 16:14 |
| pabelanger | must be a typo some place | 16:14 |
| fungi | yeah | 16:14 |
| fungi | was about to point that out | 16:14 |
| fungi | so we have an improper macro expansion happening there | 16:14 |
| *** Na3iL has joined #openstack-infra | 16:14 | |
| *** sridhar_ram has joined #openstack-infra | 16:14 | |
| zigo | Oh... | 16:14 |
| fungi | we wanted it to replace {project} with the short project name | 16:14 |
| zigo | Also, I didn't expect to have "uploads" there, just its content. | 16:14 |
| zigo | I guess that's not really a problem though. | 16:15 |
| *** Sukhdev has quit IRC | 16:15 | |
| fungi | right, the uploads thing is a shortcoming of hierarchy preservation in the scp publisher plugin, there are some options/workarounds to address that later | 16:15 |
| fungi | but for now let's get the macro expansion problem solved | 16:16 |
| *** Sukhdev_ has quit IRC | 16:16 | |
| zigo | So, what about the {project} macro expansion? | 16:16 |
| zigo | How to fix? | 16:16 |
| fungi | yeah, i'm looking to see how we address that in the normal tarball publisher | 16:17 |
| zigo | cheers | 16:17 |
| *** dmk0202 has quit IRC | 16:17 | |
| *** jamesmcarthur has quit IRC | 16:17 | |
| *** jamesmcarthur has joined #openstack-infra | 16:18 | |
| *** roxanagh_ has joined #openstack-infra | 16:18 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 16:18 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist https://review.openstack.org/319337 | 16:18 |
| fungi | zigo: so one option is to do like the javascript jobs do... http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/javascript.yaml#n72 | 16:19 |
| fungi | basically pass '{name}' (which expands from the short project name from the jenkins/jobs/projects.yaml) | 16:19 |
| *** ddieterly is now known as ddieterly[away] | 16:19 | |
| *** claudiub|2 has joined #openstack-infra | 16:20 | |
| zigo | fungi: So, use {name} instead of {project} ? | 16:21 |
| *** ddieterly[away] is now known as ddieterly | 16:23 | |
| fungi | zigo: i don't think {name} will expand to the parent project from within the macro (it might). i was suggesting have your job at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/deb-openstack-pkg-tools.yaml#n54 pass project: '{name}' as a parameter under the - deb-pkgs: publisher | 16:23 |
| *** claudiub has quit IRC | 16:24 | |
| *** vsaienko has joined #openstack-infra | 16:24 | |
| fungi | zigo: though it's also possible that only works from within a job-template rather than a job. we should be able to tell from the results of teh xml comparison job running on the fix you propose | 16:25 |
| vsaienko | hello infra team, please help to merge https://review.openstack.org/#/c/318521/ | 16:25 |
| *** salv-orlando has joined #openstack-infra | 16:25 | |
| *** salv-orlando has quit IRC | 16:26 | |
| *** salv-orlando has joined #openstack-infra | 16:26 | |
| vsaienko | project-config core, please help to merge https://review.openstack.org/#/c/304473/ the patch adds job to test ironic cleaning | 16:26 |
| *** daemontool has quit IRC | 16:27 | |
| *** daemontool_ has joined #openstack-infra | 16:27 | |
| *** shashank_hegde has quit IRC | 16:28 | |
| *** johnny__ has joined #openstack-infra | 16:29 | |
| zaro | fungi: ok. i figured out how to enforce access rules on the gerrit reindex script, if you want to give it a try on next rename i'll prepare the changes. | 16:29 |
| *** jamesmcarthur has quit IRC | 16:29 | |
| *** jpr has quit IRC | 16:29 | |
| *** Hal1 has quit IRC | 16:30 | |
| fungi | zaro: yes, please! | 16:31 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Fix {project} expansion https://review.openstack.org/319343 | 16:32 |
| *** nwkarsten has quit IRC | 16:33 | |
| zigo | fungi: Is that what you were talkinga bout/ | 16:34 |
| zigo | ? | 16:34 |
| *** lakshmiS has quit IRC | 16:34 | |
| fungi | zigo: almost, but needs more spaces. see inline comment | 16:36 |
| *** annegentle has quit IRC | 16:37 | |
| *** tonytan4ever has joined #openstack-infra | 16:37 | |
| *** HeOS has joined #openstack-infra | 16:37 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Fix {project} expansion https://review.openstack.org/319343 | 16:37 |
| fungi | zigo: yep, that's what i was suggesting | 16:38 |
| zigo | :) | 16:38 |
| fungi | zigo: now once it runs the xml comparison job, we can take a look at the actual jenkins job configuration created by this | 16:39 |
| zigo | I'm not sure I understood why the 2 more spaces were needed, but I fully trust! :) | 16:39 |
| zigo | Right. | 16:39 |
| *** jamesmcarthur has joined #openstack-infra | 16:40 | |
| *** cloudtrainme has quit IRC | 16:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add test of post cloner with cached repo https://review.openstack.org/319347 | 16:41 |
| *** camunoz has quit IRC | 16:41 | |
| *** esikachev has quit IRC | 16:42 | |
| *** camunoz has joined #openstack-infra | 16:42 | |
| *** rajinir has quit IRC | 16:42 | |
| *** trown|afk is now known as trown | 16:43 | |
| fungi | zigo: easier to demonstrate than to explain in this case http://paste.openstack.org/show/497943/ | 16:43 |
| zigo | oh, indeed ! :) | 16:44 |
| *** jsavak has quit IRC | 16:44 | |
| *** flepied1 has joined #openstack-infra | 16:44 | |
| fungi | zigo: the flat dict in the first case vs the nested dict in the second | 16:44 |
| zigo | Got it. | 16:45 |
| *** roxanagh_ has quit IRC | 16:45 | |
| fungi | this is one reason why some people despise yaml | 16:45 |
| *** roxanagh_ has joined #openstack-infra | 16:45 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Batch calls to list_floating_ips https://review.openstack.org/315697 | 16:45 |
| *** flepied has quit IRC | 16:45 | |
| fungi | this particular quirk doesn't bother me, but it does bother a lot of people | 16:45 |
| *** tonytan4ever has quit IRC | 16:46 | |
| *** ddieterly is now known as ddieterly[away] | 16:46 | |
| zigo | Well, one got to know about it, and once one does, that's ok-ish, I guess... | 16:47 |
| openstackgerrit | Merged openstack-infra/shade: Be more precise in our detection of provider networks https://review.openstack.org/318880 | 16:47 |
| clarkb | zaro: wow gerrit's gerrit is switching to use notedb | 16:48 |
| *** abregman|afk has quit IRC | 16:48 | |
| *** jamesmcarthur has quit IRC | 16:50 | |
| notmorgan | clarkb: ? | 16:50 |
| notmorgan | clarkb: oh wow | 16:50 |
| clarkb | notmorgan: getting rid of the sql db and replacing it with content in git | 16:50 |
| *** cloudtrainme has joined #openstack-infra | 16:51 | |
| notmorgan | clarkb: interesting choice. | 16:51 |
| *** asettle has joined #openstack-infra | 16:51 | |
| fungi | in git notes, specifically (hence the name, "notedb") | 16:51 |
| notmorgan | clarkb: i.. hmm. very interestin | 16:51 |
| Shrews | i fear anything with "notes" in the name | 16:51 |
| *** asettle has quit IRC | 16:51 | |
| notmorgan | Shrews: lol ++ | 16:51 |
| *** yamamot__ has quit IRC | 16:51 | |
| *** kien-ha has quit IRC | 16:51 | |
| fungi | gerrit's gerrit is switching to use lotus notes! | 16:51 |
| *** tonytan4ever has joined #openstack-infra | 16:51 | |
| vsaienko | fungi, I'm sorry for bothering you. Could you please help to merge https://review.openstack.org/#/c/318521/ it is a blocker for Ironic team on grenade stuff | 16:52 |
| notmorgan | Shrews: lets use notedb for nodepool! (JK don't throw anything at me :P ) | 16:52 |
| jeblair | clarkb, jhesketh: i left a couple of suggestions on nakato's post change that i would like you to look over: https://review.openstack.org/293194 | 16:52 |
| fungi | vsaienko: is the goal there just to shorten the job runtime? | 16:53 |
| openstackgerrit | Brent Eagles proposed openstack-infra/tripleo-ci: Exit with non-zero status on create and update errors https://review.openstack.org/319352 | 16:54 |
| fungi | vsaienko: jroll: any significant risk of introducing an upgrade regression which would have been caught by a tempest test but isn't exposed by grenade's smoke tests? | 16:54 |
| *** banix has quit IRC | 16:55 | |
| vsaienko | fungi, yes it will speed up the job. There is an example http://logs.openstack.org/09/311109/39/experimental/gate-grenade-dsvm-ironic/b14cce8/ | 16:55 |
| *** lucasagomes is now known as lucas-afk | 16:55 | |
| *** kus has joined #openstack-infra | 16:55 | |
| vsaienko | fungi, we launching smoke tests which do actually the same thing (boot/delete instance) in grenade.sh. There is no need to launch extra tests from devstack-gate.sh | 16:56 |
| *** jamesmcarthur has joined #openstack-infra | 16:56 | |
| *** jpr has joined #openstack-infra | 16:59 | |
| *** pblaho has quit IRC | 16:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Install build-essential instead of just gcc https://review.openstack.org/317085 | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Remove quotes from zuul.conf launcher sites https://review.openstack.org/317617 | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Add /usr/local/bin to zuul-launcher path https://review.openstack.org/317141 | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Add launcher logging config file to zuul.conf https://review.openstack.org/317090 | 17:00 |
| *** jsavak has joined #openstack-infra | 17:00 | |
| *** alexey_weyl has joined #openstack-infra | 17:01 | |
| *** eventingmonkey has quit IRC | 17:01 | |
| alexey_weyl | Hi | 17:02 |
| alexey_weyl | I have pushed a change to gerrit a few days ago, and I would like please if someone could approve it | 17:02 |
| alexey_weyl | https://review.openstack.org/#/c/317851/ | 17:02 |
| *** derekh has quit IRC | 17:03 | |
| clarkb | jeblair: I agree with the fetch thing, I expect that that has some potentially nasty and later unfun to debug corner cases | 17:03 |
| clarkb | jeblair: an explicit fetch should be relatively quick if the update has already pulled it | 17:03 |
| jeblair | clarkb: ok, that's reassuring, thanks :) | 17:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository https://review.openstack.org/319322 | 17:04 |
| clarkb | commented on the change with that and suggested we go ahead and address it now before someone hsa a bad weekend debugging it in 2 years | 17:05 |
| *** sambetts is now known as sambetts|afk | 17:05 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST for Ironic grenade https://review.openstack.org/318521 | 17:06 |
| openstackgerrit | Brian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin https://review.openstack.org/319368 | 17:07 |
| *** jamesmcarthur has quit IRC | 17:08 | |
| openstackgerrit | Merged openstack-infra/shade: Add initial setup for magnum in shade https://review.openstack.org/243239 | 17:08 |
| clarkb | vsaienko: why can't https://review.openstack.org/#/c/304473/ just be part of the existing job? | 17:09 |
| *** jistr has quit IRC | 17:09 | |
| *** alexey_weyl has quit IRC | 17:09 | |
| *** jamesmcarthur has joined #openstack-infra | 17:10 | |
| *** alexey_weyl has joined #openstack-infra | 17:10 | |
| *** jsavak has quit IRC | 17:10 | |
| *** tpsilva has joined #openstack-infra | 17:11 | |
| *** shashank_hegde has joined #openstack-infra | 17:13 | |
| *** jtomasek has quit IRC | 17:14 | |
| *** ccamacho has quit IRC | 17:15 | |
| zaro | clarkb: yup, exciting times, i'm wondering what the migration might look like. | 17:15 |
| *** ccamacho has joined #openstack-infra | 17:16 | |
| clarkb | zaro: ya, judging by the thread people are worried about perofmrnace too | 17:17 |
| clarkb | but neat to see it happening | 17:17 |
| *** inc0 has joined #openstack-infra | 17:18 | |
| *** tonytan4ever has quit IRC | 17:18 | |
| notmorgan | jeblair: FYI, about to push some restructuring changes for nodepool up. Adding an abstraction for nodedb (so we can move to nodezk but not need to drop nodedb immediately at the same time), among a few other things. | 17:18 |
| notmorgan | jeblair: will let you know when i get to that point. | 17:19 |
| jeblair | notmorgan: cool, thanks | 17:19 |
| *** alexey_weyl has quit IRC | 17:19 | |
| openstackgerrit | benny@us.ibm.com proposed openstack-infra/project-config: New project request for SwiftOnHPSS https://review.openstack.org/316098 | 17:20 |
| *** shashank_hegde has quit IRC | 17:20 | |
| pleia2 | fungi: looks like no one took an action item to announce the upcoming renames 2 weeks from now, shall I in case folks need to get some others queued up? | 17:20 |
| pleia2 | I can send a follow-up reminder the week of | 17:20 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Add a build for Gerrit groovy-provider plugin https://review.openstack.org/319380 | 17:22 |
| *** dtantsur is now known as dtantsur|afk | 17:22 | |
| *** david-lyle_ is now known as david-lyle | 17:22 | |
| pleia2 | and to everyone: sorry for being away-ish since summit (especially w/ reviews), but happy to say my book is pretty much done now, just some dangling feedback out there, and I've even been able to make some patches to the openstack puppet modules even if I have been less infra-y :) | 17:22 |
| pleia2 | so not wasted time away from day to day incidents, I should be able to help infra-cloud more now | 17:23 |
| jeblair | pleia2: welcome back! | 17:23 |
| pleia2 | we fixed several bugs in mitaka/stable | 17:23 |
| pleia2 | re: puppet | 17:23 |
| *** inc0_ has joined #openstack-infra | 17:24 | |
| pleia2 | jeblair: thanks :) | 17:24 |
| clarkb | https://review.openstack.org/#/c/312647/ should save a minute or two per devstack-gate run | 17:24 |
| *** ddieterly[away] is now known as ddieterly | 17:24 | |
| *** inc0 has quit IRC | 17:25 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Add a build for Gerrit groovy-provider plugin https://review.openstack.org/319380 | 17:25 |
| mtreinish | pleia2: any thoughts on: https://review.openstack.org/312308 | 17:25 |
| zaro | pleia2: ^ that may help the rename, force online reindex | 17:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Ops repo to OpenStack-Ansible https://review.openstack.org/319381 | 17:26 |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST_ALL_PLUGINS for Ironic grenade https://review.openstack.org/318521 | 17:27 |
| *** Na3iL has quit IRC | 17:27 | |
| pleia2 | mtreinish: you don't waste any time :) | 17:27 |
| *** roxanagh_ has quit IRC | 17:27 | |
| clarkb | fungi: jeblair can you double check my bash on https://review.openstack.org/#/c/296171/2/functions.sh ? if that comment looks right to you I will go ahead and push up a new patchset | 17:27 |
| pleia2 | zaro: ooh, neat | 17:28 |
| *** jsavak has joined #openstack-infra | 17:28 | |
| mtreinish | pleia2: heh | 17:28 |
| jeblair | clarkb: agreed | 17:29 |
| *** vhosakot has quit IRC | 17:29 | |
| clarkb | pabelanger: the WIP on https://review.openstack.org/#/c/311815/ probably hides it from most reviewers fwiw | 17:30 |
| clarkb | pabelanger: I wil lreview it nowish | 17:30 |
| pleia2 | mtreinish: so we're not adding a second instance of running e-r on static.o.o, just the source+config so health can use it? | 17:30 |
| mtreinish | pleia2: yeah it just needs the source and the python interface available on health.o.o so the api server can leverage the queries in the repo | 17:31 |
| mtreinish | pleia2: the patch using it is here: https://review.openstack.org/311919 | 17:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Make reproduce function quiet https://review.openstack.org/296171 | 17:32 |
| mtreinish | (I just noticed I put the wrong changeid in the commit msg) | 17:32 |
| clarkb | jeblair: ^ ok new patchset there that should make it quieter | 17:32 |
| *** roxanagh_ has joined #openstack-infra | 17:32 | |
| jeblair | +3 | 17:34 |
| *** nwkarsten has joined #openstack-infra | 17:34 | |
| *** nwkarsten has quit IRC | 17:34 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add a meeting_id to openstack-community-office-hours https://review.openstack.org/319385 | 17:34 |
| *** jamesmcarthur has quit IRC | 17:34 | |
| *** jamesmcarthur has joined #openstack-infra | 17:35 | |
| pleia2 | mtreinish: does it actually manipulate it at all? wondering if we can use the existing e-r instance on status | 17:35 |
| *** salv-orlando has quit IRC | 17:36 | |
| *** daemontool has joined #openstack-infra | 17:37 | |
| clarkb | pabelanger: mtreinish I think https://review.openstack.org/#/c/311815/5 needs an element-deps with package-installs in it | 17:37 |
| pleia2 | oh, I guess this would all be on health.o.o, just the dashboard is on status | 17:37 |
| mtreinish | pleia2: it does, the o-h api server is actually querying es. It's just using e-r as a lib (and a source of queries) The other instances that run e-r are either a daemon or a cron job creating a json blob | 17:37 |
| clarkb | pabelanger: mtreinish it may work for us because we already include package-installs into our dep list but if someone else doesn't this won't work | 17:37 |
| mtreinish | pleia2: yeah | 17:38 |
| clarkb | pabelanger: you volunteered to babysit, would you like that change to be updated or have a followup change to include the deps | 17:38 |
| pleia2 | mtreinish: thanks, +2A | 17:38 |
| *** jerryz has joined #openstack-infra | 17:39 | |
| mtreinish | pleia2: I thought about publishing e-r on pypi for this, but then there is the data files in python packaging annoyance so I decided not to bother | 17:39 |
| mtreinish | pleia2: cool thanks | 17:39 |
| *** jamesmcarthur has quit IRC | 17:39 | |
| *** daemontool_ has quit IRC | 17:39 | |
| mtreinish | clarkb: hmm, do we expect anyone else to use the stackviz stuff? | 17:39 |
| pleia2 | mtreinish: heh, yeah, that's fair | 17:40 |
| clarkb | mtreinish: possibly? the elements should function on their own | 17:40 |
| *** shashank_hegde has joined #openstack-infra | 17:40 | |
| mtreinish | clarkb: sure, that's fair | 17:41 |
| clarkb | armax: regXboi re https://review.openstack.org/#/c/314079/ I am not sure if that should be abandoned or merged. I think I saw recent mail that the debug command would be removed completely? | 17:41 |
| *** esikachev has joined #openstack-infra | 17:41 | |
| armax | clarkb: we started a long process of deprecation | 17:42 |
| regXboi | armax: I assume I can abandon this then? | 17:42 |
| armax | clarkb: for the neutron-debug command | 17:42 |
| *** gomarivera has joined #openstack-infra | 17:42 | |
| *** sdague has quit IRC | 17:43 | |
| armax | clarkb, regXboi it will eventually need to merge | 17:43 |
| armax | once neutron-debug is replaced for good, but I don’t see the urgency atm | 17:43 |
| clarkb | armax: regXboi ok if it is a long way out I think we should abandon that change to get it out of the way. When we are ready to remove neutron-debug or replcae it or whatever we can push a new change or restore and update | 17:44 |
| *** akshai has quit IRC | 17:44 | |
| regXboi | and so abandoned | 17:46 |
| *** e0ne has joined #openstack-infra | 17:46 | |
| *** akshai has joined #openstack-infra | 17:47 | |
| clarkb | vsaienko: I will have a comment on https://review.openstack.org/#/c/316662/2 shortly as well | 17:48 |
| fungi | pleia2: yeah, rename announcement would be good. sorry forgot to action anyone since we were crunched on time at the end of the meeting slot | 17:49 |
| pleia2 | fungi: ok, will do :) | 17:49 |
| *** ddieterly is now known as ddieterly[away] | 17:51 | |
| *** kzaitsev_mb has quit IRC | 17:51 | |
| fungi | clarkb: yep, 296171 lgtm | 17:51 |
| *** yamamoto has joined #openstack-infra | 17:52 | |
| armax | clarkb: thanks | 17:52 |
| clarkb | therve: are you around to have a quick chat about https://review.openstack.org/#/c/286136/4/functions.sh ? | 17:54 |
| *** markvoelker has joined #openstack-infra | 17:54 | |
| *** sdague has joined #openstack-infra | 17:55 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Move nodedb to nodestore.nodedb https://review.openstack.org/319390 | 17:56 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Move constants to nodestore module https://review.openstack.org/319391 | 17:56 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Create pivot point for nodestate storage https://review.openstack.org/319392 | 17:56 |
| *** lakshmiS has joined #openstack-infra | 17:57 | |
| *** mtanino has quit IRC | 17:57 | |
| *** chem`` has quit IRC | 17:57 | |
| openstackgerrit | Merged openstack-infra/puppet-openstack_health: Add elastic recheck to api server https://review.openstack.org/312308 | 17:57 |
| notmorgan | jeblair, Shrews: ^ there is the starting place for the nodepool changes. working thorugh some of the ZK specific things and changes around that for better parsing. | 17:58 |
| notmorgan | but that structure will allow us to transition from sql->zk more easily | 17:58 |
| *** markvoelker has quit IRC | 17:59 | |
| *** vhosakot has joined #openstack-infra | 17:59 | |
| *** yamamoto has quit IRC | 18:00 | |
| *** jpr has quit IRC | 18:01 | |
| *** cody-somerville has quit IRC | 18:01 | |
| *** nwkarsten has joined #openstack-infra | 18:01 | |
| *** jcoufal has quit IRC | 18:01 | |
| openstackgerrit | Mário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances https://review.openstack.org/319395 | 18:02 |
| clarkb | notmorgan: jeblair I am confused, are we completely replacing sql with zk or just the bits that require shared state (eg image builds)? | 18:02 |
| notmorgan | clarkb: in nodepool the structure of everything could easily lend itself to being 100% zk | 18:03 |
| *** sridhar_ram has quit IRC | 18:03 | |
| clarkb | right thats the question, what is your intention | 18:03 |
| notmorgan | clarkb: my intention is 100% zk | 18:03 |
| clarkb | because aiui we were keeping sql but skimming your changes you are replacing sql | 18:03 |
| notmorgan | clarkb: unless there is a good reason for keeping SQL with this type of structure. | 18:05 |
| clarkb | notmorgan: I am not sure if there is, but iirc the spec was limited to image building | 18:05 |
| notmorgan | clarkb: i could do it as a split, but it felt... awkward? | 18:05 |
| clarkb | so just making sure I understand | 18:05 |
| notmorgan | clarkb: i spent a good amount of time looking at the structure of the data before writing this though. | 18:05 |
| notmorgan | spec might need an update too. | 18:05 |
| notmorgan | then | 18:05 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Setup a tox pep8 check and update test-requirements.txt https://review.openstack.org/319398 | 18:05 |
| clarkb | and yes generally the data is resrouce tracking, it isn't rickly relational where SQL would really shine | 18:06 |
| clarkb | *richly | 18:06 |
| notmorgan | clarkb: exactly. | 18:06 |
| notmorgan | clarkb: which is why the split between sql and zk felt so awkward | 18:07 |
| *** nadya has joined #openstack-infra | 18:07 | |
| *** chem`` has joined #openstack-infra | 18:07 | |
| *** jcoufal has joined #openstack-infra | 18:07 | |
| notmorgan | it is almost purely hierarchical as well. <image>/<node based on image>|snapshot|etc | 18:07 |
| *** oanson has joined #openstack-infra | 18:07 | |
| notmorgan | with some levels for proviers too. | 18:07 |
| notmorgan | or it could be attrs on the node. | 18:08 |
| clarkb | notmorgan: I think the one place where this could be a potential problem is if/when we start treating other resources like floating IPs and volumes and networks and routers etc as things nodepool should know about | 18:08 |
| notmorgan | but it is def only somewhat relational | 18:08 |
| clarkb | because then you would be tracking relations between things | 18:08 |
| notmorgan | perhaps. | 18:08 |
| clarkb | we have long avoided that because the theory is openstack provides that management for us but reality is it often does not work | 18:08 |
| notmorgan | clarkb: hmm. | 18:09 |
| *** piet has joined #openstack-infra | 18:09 | |
| clarkb | notmorgan: one concrete example of something that would possibly benefit from sql is if we managed a pool of preallocated fips directly | 18:10 |
| notmorgan | clarkb: i'll have to think on that a bit - it may justify a split. | 18:10 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Install the Gerrit delete-project plugin https://review.openstack.org/319400 | 18:10 |
| clarkb | notmorgan: rather than boot instance, create and attach fip, delete fip, delete instance | 18:10 |
| notmorgan | clarkb: it also may be a narrow enough dataset that relationality and normalized data isn't a huge win | 18:11 |
| clarkb | notmorgan: have full quota of fips preallocated (or lazily allocate them), then assign and unassign as we boot and delete nodes | 18:11 |
| *** ifarkas has quit IRC | 18:11 | |
| notmorgan | clarkb: it depends, if w'ere not using FKs and updating the rows directly - it's about the same overhead as locking a node and setting an attr | 18:11 |
| clarkb | notmorgan: yes that is possible too, we are talking a relatively small number of resources unless computing fundamentally change stomorrow | 18:11 |
| notmorgan | clarkb: if we lean on RDBMS technology more, sure. | 18:11 |
| *** nadya has quit IRC | 18:11 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 18:12 |
| notmorgan | clarkb: i almost think a /<provider>/fips/ | 18:12 |
| notmorgan | makes sense in this case | 18:12 |
| notmorgan | *almost* | 18:12 |
| notmorgan | i'll mull on it a bit | 18:12 |
| clarkb | notmorgan: you have to track what instance owns what ip at any given time | 18:12 |
| clarkb | notmorgan: beucase neutron and nova can't actually do that reliably today | 18:12 |
| notmorgan | clarkb: right. but that may in fact be ok. it depends on how we want to do that with sql | 18:12 |
| notmorgan | it's a difference is how we do that. | 18:13 |
| *** sarob has joined #openstack-infra | 18:13 | |
| notmorgan | if the number is low, non-normal form might be as elegant/easier. | 18:13 |
| notmorgan | i'll ponder some before driving to much further here. | 18:13 |
| *** nadya has joined #openstack-infra | 18:13 | |
| *** mtanino has joined #openstack-infra | 18:14 | |
| clarkb | notmorgan: we have also discussed possibly using volume backed instances but apparently cinder and nova do a good job of tracking that cooperation | 18:14 |
| clarkb | that may be a another similar case but probably not due to cloud holding our hand properly | 18:14 |
| notmorgan | right | 18:14 |
| openstackgerrit | amrith proposed openstack-infra/project-config: Adding new repository trove-image-builder https://review.openstack.org/312805 | 18:15 |
| *** mikelk has joined #openstack-infra | 18:15 | |
| clarkb | notmorgan: and probably the most complicated set of relations would be if we went nodepool managed networks, subnets, routers, dhcp servers (and maybe security groups) | 18:16 |
| *** esikachev has quit IRC | 18:17 | |
| notmorgan | clarkb: i think it comes down to the question of how likely that is | 18:17 |
| clarkb | notmorgan: it is something we struggled with when hpcloud was a thing | 18:18 |
| notmorgan | clarkb: if we're going to track FIPs and maybe volumes -- i'm thinking non-normalized is fine | 18:18 |
| clarkb | notmorgan: we used a workaround to scale up by having a different logical provider per network then manually preprovisioned everything | 18:18 |
| notmorgan | if we really thing we should do networks etc and highly complex stuff | 18:18 |
| notmorgan | i'm inclined to say we should keep some level of relationship | 18:18 |
| clarkb | so I am not sure we know how we would like to best solve that | 18:18 |
| clarkb | I think we do know what we had was subpar | 18:18 |
| *** nadya has quit IRC | 18:18 | |
| notmorgan | s/relationship/relational engine/ | 18:19 |
| openstackgerrit | amrith proposed openstack-infra/project-config: Adding new repository trove-image-builder https://review.openstack.org/312805 | 18:19 |
| notmorgan | but we def don't want to just do "oh we'll update these rows with references" because thats no different than what we're doing now. | 18:19 |
| notmorgan | erm, s/now/without relational db | 18:19 |
| fungi | even then, i doubt scaling that (non-normal form lookups and updating secondary indices) in zk would result in anywhere near the performance issues with scaling call volume on the openstack api side | 18:20 |
| notmorgan | clarkb: it comes down to what the ops look like. | 18:20 |
| notmorgan | fungi: ++ | 18:20 |
| notmorgan | and the desired direction | 18:20 |
| fungi | so while it might be inefficient, i have a feeling it would be imperceptably less efficient held up against the other stuff nodepool has to wait on | 18:20 |
| clarkb | notmorgan: my initial impression is that the ops will be scale up/down related | 18:20 |
| clarkb | keep an accurate count of instances per network/subnet/router, then make and delete as necessary to maintain a ratio that achieves desired perfomrance | 18:21 |
| notmorgan | fungi: and i do like the hierarchical nature of the resource tracking. | 18:21 |
| clarkb | can be done without sql for sure | 18:21 |
| notmorgan | fungi: vs. the SQL model. | 18:21 |
| notmorgan | i also like that the data is mostly discoverable by just saying "get_children" | 18:22 |
| notmorgan | rather than needing to know the schema | 18:22 |
| clarkb | but also keep basic attributes for each, know what external net is attache dto each router (required for fips), know what subnets belong to which networks in order to make sure we don't make overlapping subnets and so on | 18:22 |
| notmorgan | not that i would want to do lots of get_children for genreral use-casee | 18:23 |
| *** eventingmonkey has joined #openstack-infra | 18:23 | |
| *** rossella_s has quit IRC | 18:23 | |
| notmorgan | clarkb: so again. is this something we're really looking to do? thats going to be what dictates my choices here more than anything | 18:24 |
| notmorgan | clarkb: and i don't really have a horse in this race. | 18:24 |
| *** rossella_s has joined #openstack-infra | 18:24 | |
| clarkb | notmorgan: it is something we need to solve if we can't get nova/neutron to reliably keep track of fips and if we want to support clouds like the old hpcloud | 18:25 |
| jeblair | woah | 18:25 |
| jeblair | okay, | 18:25 |
| clarkb | notmorgan: the hack we had for hpcloud works, it is just less than ideal. So we can limp along with that and worry about it if/when another cloud like that appears | 18:25 |
| jeblair | so, the approved spec is for zookeeper for image builds | 18:26 |
| clarkb | jeblair: yes that is what brought this up :) | 18:26 |
| notmorgan | and if i'm also pushing an update to the spec. | 18:26 |
| jeblair | there is a not-yet-approved spec to extend zk to everything in nodepool: https://review.openstack.org/305506 | 18:26 |
| notmorgan | i can narrow the scope of the pivot point easily | 18:26 |
| fungi | right, all of the performance discussion aside, that spec started small in scope for good reason | 18:26 |
| jeblair | it is the logical next step for this, and we should take it if the image build part works out | 18:26 |
| notmorgan | fungi: and i'm also looking at this needing to be a fairly large change in either case. | 18:27 |
| notmorgan | fungi: unfortunately | 18:27 |
| *** julim has quit IRC | 18:27 | |
| notmorgan | a lot of the work is internal code shuffling anyway. | 18:27 |
| jeblair | (honestly, at this point, i think that we *need* that for zuulv3, so i really hope it works out) | 18:27 |
| notmorgan | but narrowing down the pivot for building only is doable | 18:27 |
| notmorgan | but we need the pivot so we can work on this without doing a big bang change imo | 18:28 |
| * notmorgan doesn't like the big-bang changes. | 18:28 | |
| clarkb | notmorgan: no big bang chang ewould be required I odn't think | 18:28 |
| notmorgan | supporting both for a while will be required. | 18:28 |
| notmorgan | imo. | 18:28 |
| jeblair | i personally don't think that abstracting this is the right approach | 18:28 |
| clarkb | notmorgan: the image builds stuff is relatively limited in scope compared ot the rest of the state in nodepool | 18:28 |
| jeblair | because as was mentioned earlier, they are different ways of thinking about the problem | 18:29 |
| *** jcoufal has quit IRC | 18:29 | |
| jeblair | so i think we should start by thinking about the problem of image builds in the zookeeper way | 18:29 |
| jeblair | (and then later i think we should think about the problem of node builds in the zookeeper way) | 18:29 |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 18:29 |
| jeblair | but having said that, perhaps the best way to implement it is an abstraction | 18:29 |
| notmorgan | jeblair: i'll start again with a different scope in mind. | 18:30 |
| jeblair | so we should look at that and review it | 18:30 |
| *** dtardivel has quit IRC | 18:30 | |
| *** vhosakot has quit IRC | 18:30 | |
| *** dimtruck is now known as zz_dimtruck | 18:30 | |
| jeblair | notmorgan: i haven't looked at it at all, so i wouldn't ask you to change anything at this point :) | 18:30 |
| notmorgan | jeblair: fwiw, the split between building and "state" is pretty awkward. | 18:30 |
| *** Kaiyan has joined #openstack-infra | 18:30 | |
| notmorgan | jeblair: nah, it's fine this convo w/ clarkb has highlighted we can't do what i was proposing | 18:30 |
| notmorgan | jeblair: for now. | 18:31 |
| notmorgan | jeblair: so let me restart this with a different scope in mind. | 18:31 |
| jeblair | notmorgan: if you find something that does't jive with the spec, can you update the spec? | 18:32 |
| jeblair | i'd really like to work from that if possible | 18:32 |
| notmorgan | jeblair: yeah. it's not anything with the spec itself. | 18:32 |
| notmorgan | jeblair: just was looking at the whole data model. the way nodepool is built it's pretty tightly integrated atm. | 18:32 |
| jeblair | (i mean, it's intentionally not detailed as to how to accomplish the thing, but it should describe the end-state well enough that we shouldn't need to question the fundamentals after the fact) | 18:32 |
| clarkb | notmorgan: ya, I think the current data model is lacking when it comes to specific problems we have had though | 18:33 |
| notmorgan | jeblair: so i was trying to avoid pull apart the internal bits and split that up. | 18:33 |
| notmorgan | jeblair: to start, but look slike that is the first step. | 18:33 |
| clarkb | notmorgan: in particular floating IP (re)use and network management for clouds that require us to use >1 network | 18:33 |
| jeblair | clarkb: that's definitely out of scope for the zk builders | 18:33 |
| notmorgan | jeblair: i'll start reworking the datamodel so we can have the logical split we need. its just a change in the order of operations | 18:33 |
| jeblair | clarkb: but even so, i think it's out of scope in the future too. that is a shade problem | 18:34 |
| clarkb | jeblair: right but would likely factor into the replace sql | 18:34 |
| *** bpokorny has quit IRC | 18:34 | |
| *** zz_dimtruck is now known as dimtruck | 18:34 | |
| clarkb | jeblair: shade doesn't have persistent state and both of these problems require persistent state to be managed | 18:34 |
| notmorgan | jeblair: instead of rework to make ZK a possibility, it'll be rework data model, then rework to make zk hook where we need. | 18:34 |
| jeblair | clarkb: can you look over https://review.openstack.org/305506 ? | 18:34 |
| clarkb | jeblair: I was kind of hoping we could make progress on the first zk stuff before trying to throw darts at that | 18:35 |
| jeblair | notmorgan: it is definitely okay for us to have a flag-day transition. we don't need to support both things. all the data is ephemeral :) | 18:35 |
| clarkb | but I can review it sooner | 18:35 |
| openstackgerrit | David Lyle proposed openstack-infra/devstack-gate: Test patch https://review.openstack.org/319409 | 18:35 |
| jeblair | clarkb: i'm not asking anyone to implement that now. but i think it might help if we had an idea of where we wanted to go next. | 18:35 |
| *** davidgiluk has joined #openstack-infra | 18:35 | |
| *** jsavak has quit IRC | 18:36 | |
| jeblair | clarkb: i will say that in that vein, i'm not ready to say that we need to track floating ips, but, if we feel that tracking fips is something we might need to do in the future, it is worth making sure we can slot that into the proposed data model so that we don't back ourselves into a corner. | 18:36 |
| *** davidgiluk has left #openstack-infra | 18:36 | |
| clarkb | jeblair: to clarify we would only need to track fips if we want to reuse them | 18:36 |
| notmorgan | jeblair: no no, the pivot was to make this an easy "now use X" but not break sql until we change the config | 18:36 |
| clarkb | jeblair: rather than doing a full create, attach, detach, delete cycle for each fip | 18:37 |
| *** jsavak has joined #openstack-infra | 18:37 | |
| notmorgan | and if we needed to roll back we could then. thats all | 18:37 |
| clarkb | jeblair: if we tracked them then we could just attach, detach | 18:37 |
| *** jpr has joined #openstack-infra | 18:37 | |
| fungi | hrm, interesting correlation with the apache segfaults on our mirror servers. i consistently see them (roughly proportional to node quota/utilization) on all three mirrors in rax... and _nowhere_ else | 18:37 |
| fungi | leading me to wonder if this is something to do with xen domu timekeeping | 18:38 |
| fungi | seems to come and go in bursts maybe? hard to tell as the frequency even on the ord mirror is still pretty sparse | 18:39 |
| jeblair | clarkb: yeah, i think the whole port creation thing is supposed to make a lot of that better. i'd like for us to chase that to it's logical conclusion before we change course, but that is a conversation best left for later. i _do_ agree that we should allow ourselves the flexibility to do that if we need to, and i _think_ that if you look at the way i have described the zk stuff (for both builders and nodes), we should be able to ... | 18:39 |
| jeblair | ... handle it (after all, zookeeper is good at describing collections of things). | 18:39 |
| *** _sarob has joined #openstack-infra | 18:39 | |
| *** jed56 has quit IRC | 18:39 | |
| *** _prad_ has quit IRC | 18:39 | |
| fungi | but there are segfaults happening within a few seconds or minutes of each other and then hours pass before another incident | 18:40 |
| clarkb | fungi: is it worth trying to get a core dump? | 18:40 |
| jeblair | fungi: maybe they happen under memory/process/thread pressure? | 18:40 |
| *** pradk has quit IRC | 18:41 | |
| clarkb | jeblair: on the network management side I think that is something we do want and I don't know that it can go in shade. Basically point nodepool at cloud and once it has 100 instances (or whatever that scale factor is) on a router make a new router/network/subnet plumb it all and start distributing instances there | 18:41 |
| fungi | jeblair: maybe. though there's got to be something specific to rackspace's environment or maybe their images, since memory/process/thread pressure on dfw is way lower than, say, ovh | 18:41 |
| clarkb | jeblair: because if we do that then we can avoid the nastyness we had in the hpcloud configs which was 5 logical providers one per router/subnet/network | 18:41 |
| jeblair | notmorgan, clarkb: i think we need to keep in mind that we are replacing both the data store (sql) and the event system (gearman) in the image builders at the same time. that's why my intuition tells me that a slightly different way of thinking of things will be required, but like i said, i haven't written the code so i'm open. :) | 18:42 |
| *** sarob has quit IRC | 18:42 | |
| mtreinish | fungi, jeblair, clarkb: how long should it take for the pypi mirrors to sync? | 18:42 |
| *** abregman has joined #openstack-infra | 18:43 | |
| clarkb | mtreinish: its on a 5 minute loop | 18:43 |
| mtreinish | the openstack-health python jobs aren't seeing the latest subunit2sql release, it's been a few hours | 18:43 |
| mtreinish | http://logs.openstack.org/19/311919/14/check/gate-openstack-health-docs/a20a431/console.html#_2016-05-20_18_35_10_318 | 18:43 |
| jeblair | clarkb: yeah, i agree there. i think that is something we can support with zk, so i don't think we're backing ourselves into a corner. but i also would prefer not to solve it now since it isn't pressing. | 18:43 |
| mtreinish | https://pypi.python.org/pypi/subunit2sql/1.6.0 | 18:43 |
| *** vhosakot has joined #openstack-infra | 18:43 | |
| clarkb | jeblair: agreed on not solving it now, we currently have no clouds that require thinking about that again | 18:44 |
| jeblair | clarkb: aren't we lucky? :) | 18:44 |
| notmorgan | clarkb: yay? ;) | 18:44 |
| notmorgan | ok. thats fine. i'll work on splitting up the data model to start. | 18:44 |
| notmorgan | i just need to spend more time thinking about that split than i have (I was going to do that after moving to ZK, previously) | 18:45 |
| clarkb | mtreinish: I know the wheel mirror updates are/were broken due to missing pip3 | 18:45 |
| * notmorgan goes to stare at the db schema more. | 18:45 | |
| clarkb | mtreinish: but that is separate from pypi mirro | 18:45 |
| clarkb | fungi: ^ did the pip3 thing get addressed? | 18:45 |
| fungi | mtreinish: i don't see bandersnatch logging any update for subunit2sql | 18:45 |
| fungi | clarkb: i've +2'd haypo's proposed fix | 18:45 |
| clarkb | ooh tehre is a patch to review /me does this | 18:46 |
| fungi | not sure if it merged yet? | 18:46 |
| *** sdake has joined #openstack-infra | 18:46 | |
| fungi | we got into the weeds a little over "python2" vs "python" (probably just me being overly pedantic) | 18:46 |
| clarkb | doesn't look like it, just double checking if python3 is installed then will approve | 18:46 |
| fungi | yeah, i see a python3 executable on mirror-update.o.o | 18:47 |
| fungi | checked that already, should have mentioned it when reviewing | 18:47 |
| *** ihrachys has quit IRC | 18:47 | |
| mtreinish | fungi: hmm, well it's definitely on pypi | 18:47 |
| jeblair | mtreinish, fungi, dstufft: i think bandersnatch is stuck | 18:47 |
| clarkb | fungi: wheels are built ona different host | 18:47 |
| jeblair | mtreinish, fungi, dstufft: http://paste.openstack.org/show/497948/ | 18:48 |
| clarkb | wheel-mirrr-somethingorother.slave.openstack.org I think | 18:48 |
| * clarkb figures out the something or other | 18:48 | |
| fungi | jeblair: mtreinish: yeah, i see ERROR lines | 18:48 |
| fungi | about stale serials | 18:48 |
| *** tonytan4ever has joined #openstack-infra | 18:48 | |
| *** yamahata has quit IRC | 18:48 | |
| jeblair | i think that is bandersnatch stuck trying to sync that package, and it can't move past it | 18:48 |
| jeblair | (somehow, that warrants a successful exit code) | 18:48 |
| fungi | not stuck as in hung at least, just keeps trying to resume from there and doesn't ever get it | 18:49 |
| *** piet has quit IRC | 18:49 | |
| fungi | looks like it began at 01:15 utc today | 18:50 |
| *** dmk0202 has joined #openstack-infra | 18:50 | |
| clarkb | confirmed that python3 is on wheel-mirror-ubuntu-trusty-amd64.slave.openstack.org and wheel-mirror-centos-7-amd64.slave.openstack.org | 18:50 |
| fungi | though prior to that we were getting "ERROR: Stale serial for package awscli" around 21:05 utc yesterday | 18:51 |
| *** ddieterly[away] has quit IRC | 18:51 | |
| *** rajinir has joined #openstack-infra | 18:51 | |
| clarkb | jeblair: fungi: we haven't updated to the new version of bandersnatch have we? I want to say it requires a full resync | 18:51 |
| jeblair | clarkb: i have not done anything on that front | 18:51 |
| fungi | bandersnatch (1.9.dev0) | 18:52 |
| fungi | so no, we haven't yet | 18:52 |
| clarkb | is it possible the mirror protocol has changed on the pypi side too requiring an update? | 18:52 |
| fungi | i think we were needing to apply the rebased hash pathing patch to 1.10, but then that got included in 1.11 | 18:52 |
| jeblair | clarkb: i don't know the answer to that, but i know i have seen this before and have been unable to fix without pypi admin help. | 18:53 |
| *** ams__ has joined #openstack-infra | 18:54 | |
| fungi | could i guess be what this meant in the release 1.11 notes: "Fix an edge case: IO errors while marking off packages as “done” could result in crashing workers that would result in bandersnatch getting stuck." | 18:54 |
| *** tonytan_brb has joined #openstack-infra | 18:56 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Start using CINDER_VOLUME_CLEAR to replace CINDER_SECURE_DELETE https://review.openstack.org/240299 | 18:56 |
| jeblair | fungi: this is why we put the curl command in our bandersnatch script: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-07-02.log.html#t2014-07-02T18:31:21 | 18:57 |
| jeblair | i tried that manually though and it did not fix the problem | 18:57 |
| fungi | jeblair: dstufft: looks like this traceback preceded it: http://paste.openstack.org/show/497949/ | 18:57 |
| zigo | fungi: Looks like it didn't work, right? http://logs.openstack.org/43/319343/2/check/gate-project-config-compare-xml/04a0b72/console.html | 18:58 |
| dstufft | it's trying to get /simple/FlexGet/ | 18:58 |
| dstufft | instead of /simple/flexget/ | 18:58 |
| zigo | fungi: It's doing {name} instead of {project} without expansion, right? | 18:58 |
| dstufft | the first is a redirect | 18:58 |
| dstufft | (and redirects don't have X-PyPI-Serial_ | 18:58 |
| dstufft | that hasn't changed recently though... | 18:59 |
| jeblair | dstufft: would the recent changes to bandersnatch avoid this? (we aren't running that yet) | 18:59 |
| fungi | zigo: yep, that's what i thought might happen. i'll propose another patchset to that change with what i think is the solution, because that'll be easier than trying to describe it | 18:59 |
| dstufft | jeblair: yes, but only because they don't require /simple/ at all :] | 18:59 |
| dstufft | they only hit the JSON endpoint | 18:59 |
| dstufft | and the files | 19:00 |
| fungi | jeblair: dstufft: right, we're still on 2.9 because we haven't gotten around to upgrading and firing off a full resync yet | 19:00 |
| zigo | fungi: I don't even think we really need the project name, since all of that are just temporary folders for the repository pickup job. | 19:00 |
| fungi | we ended up putting apache reqrite workarounds on our mirrors in the meantime to deal with the . to - conversion | 19:00 |
| *** tonytan4ever has quit IRC | 19:00 | |
| zigo | fungi: We could as well just do deb-pkg/uploads/<SHA256>/ | 19:00 |
| zigo | fungi: Your thoughts? | 19:00 |
| dstufft | fungi: I guess 1.9 has jsut been broken for awhile and nobody noticed for... well I d'nt know why | 19:01 |
| zigo | IIRC, it's just monty that suggested to use {project} | 19:01 |
| dstufft | I don't see how this ever worked for projects like FlexGet | 19:01 |
| fungi | zigo: i guess that's an option. will this job end up running on additional repos, or only ever for that one repo? if the latter, we could probably even simplify further by moving that publisher macro directly into the job definition | 19:01 |
| haypo | oops, gate-project-config-layout just failed on my approved change https://review.openstack.org/#/c/319236/: "fatal: dumb http transport does not support --depth" on "Cloning into 'zuul'..." | 19:02 |
| haypo | what is this error? | 19:02 |
| *** ddieterly has joined #openstack-infra | 19:02 | |
| zigo | fungi: deb-openstack-pkg-tools is a special repo, because it builds itself, but other repos will only be different because they don't hold the scripts themselves and will just apt-get install openstack-pkg-tools. | 19:02 |
| zigo | fungi: So, I'm tempted to reply to you by "hum... half half..." :P | 19:03 |
| fungi | zigo: so their builds will use a separate job from this one? | 19:03 |
| zigo | Yup. | 19:03 |
| zigo | But one that will be very simillar. | 19:03 |
| jeblair | fungi, dstufft: i did cdn purges on flexget, and it worked. (i don't know if that's why, or if dstufft did something else behind the scenes) | 19:03 |
| zigo | The difference is apt-get install of the build scripts instead of a cp + chmod, but that's it. | 19:03 |
| jeblair | fungi, dstufft: if that did fix it, then i guess we should update our auto-purge thingy to also purge the normalized versions of those urls | 19:04 |
| zaro | fungi: thought i would ask you about the online reindex thing before doing more work since there's a few ways to make that work. | 19:04 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: New project: js-openstack-lib https://review.openstack.org/314312 | 19:04 |
| fungi | haypo: that looks like it may be a misleading error message following a silent network failure and fallback | 19:04 |
| zaro | fungi: basically there are 3 options to be able to use that functionality. | 19:05 |
| jeblair | mtreinish, fungi, dstufft: and if i grok bandersnatch correctly, we are now synced with where we should have been 18 hours ago, and the next run, in 10 minutes, should actually sync us up to the present. | 19:05 |
| haypo | fungi: ah maybe. should i just recheck? | 19:05 |
| jeblair | mtreinish, fungi, dstufft: you can check http://mirror.dfw.rax.openstack.org/pypi/last-modified for that | 19:05 |
| jeblair | dstufft: is there a handy function to normalize a name? | 19:06 |
| zaro | fungi: it's available now as the groovy script but in order for that to work we would need to install the groovy-provider plugin + the groovy indexing script. i wasn't sure whether you would be comfortable with that. | 19:06 |
| fungi | haypo: i think so. if we keep seeing it, then maybe we've changed something with zuul-cloner or run-layout.sh | 19:06 |
| haypo | fungi: ok | 19:06 |
| mtreinish | jeblair: ok, cool | 19:06 |
| fungi | haypo: unfortunately there's not much detail to go on there (the things it's cloning from should support "smart" hht transport/backend) | 19:07 |
| fungi | s/hht/http/ | 19:07 |
| zaro | fungi: the groovy-provider plugin is the thing that enables groovy in gerrit. | 19:08 |
| *** gordc has quit IRC | 19:08 | |
| fungi | jeblair: i _think_ the relevant pep includes an example python function. checking | 19:08 |
| zaro | fungi: an alternative is to change gerrit core to allow force reindex | 19:09 |
| fungi | jeblair: indeed... https://www.python.org/dev/peps/pep-0503/#id4 | 19:09 |
| dstufft | jeblair: PEP 503 has a copy/paste function, or ``pip install packaging; python -c 'import packaging.utils; print(packaging.utils.cannonclize_name("FlexGet"))'`` | 19:09 |
| dstufft | I think that's the name | 19:10 |
| dstufft | something like that | 19:10 |
| dstufft | probably spelled better than I can type off hand | 19:10 |
| zaro | fungi: I think 2nd option is preferrable but may take a little more time. I'm pretty sure we can backport if it gets accepted on upstream. | 19:10 |
| jeblair | dstufft, fungi: thx, i'll add that in after lunch | 19:11 |
| fungi | zaro: can you elaborate on "change gerrit core to allow force reindex" more? | 19:11 |
| fungi | zaro: is that the review you pushed? | 19:11 |
| dstufft | jeblair: http://packaging.readthedocs.io/en/latest/utils/#packaging.utils.canonicalize_name | 19:11 |
| fungi | zaro: nevermind, i see you pushed a review for the groovy script | 19:12 |
| *** shashank_hegde has quit IRC | 19:12 | |
| fungi | or rather, for the groovy-provider plugin as a prerequisite | 19:12 |
| zaro | fungi: i mean propose change to gerrit core to add a force flag to gerrit index command. | 19:13 |
| fungi | zaro: oh, got it. so online reindexing needs some forcing, and currently the command line tool to initiate a reindex lacks support in gerrit to force it to happen, so it can either be extended directly in an upcoming patch to gerrit _or_ worked around by using a groovy script through a plugin | 19:14 |
| zaro | yes, it's a pre-req for the groovy script. so we would need to add both the groovy-provider + reindex script which would mean update our puppet scripts for that | 19:14 |
| zaro | fungi: yup. | 19:14 |
| fungi | zaro: so is the current lack of workingness for the command-line reindex because it/gerrit has a bug? | 19:15 |
| *** ericksonsantos has quit IRC | 19:15 | |
| fungi | i.e., does it need to be fixed in gerrit eventually anyway, and using groovy to do it through a plugin is just working around the bug for now? | 19:15 |
| zaro | fungi: no, not a bug. it just needs the option to force reindex | 19:15 |
| fungi | zaro: so how is it supposed to work currently without having an option to force reindex? | 19:16 |
| zaro | so we think it's a bug, but i'm not not sure if gerrit guys think the same. | 19:16 |
| fungi | i'm mostly just curious what the actual purpose of that reindex command line is, if it doesn't have the ability to do online reindexing. is it only intended for offline use? | 19:17 |
| zaro | i can propose a change to find out. | 19:17 |
| *** pvaneck has joined #openstack-infra | 19:17 | |
| *** markusry has quit IRC | 19:18 | |
| zaro | purpose is for upgrading gerrit. when you upgrade you have the option to offline or online index | 19:18 |
| fungi | and how is the online index supposed to work in that context? | 19:18 |
| *** pradk has joined #openstack-infra | 19:19 | |
| *** nadya has joined #openstack-infra | 19:19 | |
| zaro | run gerrit init to upgrade gerrit site which upgrades the lucene index version. | 19:19 |
| fungi | trying to figure out if this is an intended feature which doesn't work at all (which by any definition is a bug, i would think?) or whether we're using it in unexpected ways | 19:20 |
| *** julim has joined #openstack-infra | 19:20 | |
| zaro | now you can offline reindex, or online reindex. since the verion changed the online reindex will start | 19:20 |
| fungi | didn't we try to online reindex when we upgraded gerrit last time? | 19:20 |
| zaro | no, i don't believe so | 19:20 |
| fungi | or am i misremembering? | 19:20 |
| fungi | okay | 19:20 |
| zaro | i think you only tried it for renaming projects last time | 19:20 |
| *** ddieterly is now known as ddieterly[away] | 19:20 | |
| fungi | so the difference is that we want to use it to reindex _outside_ the context of a gerrit upgrade, and it's only meant to work _within_ the context of a gerrit upgrade | 19:21 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: New project: js-openstack-lib https://review.openstack.org/314312 | 19:21 |
| therve | clarkb, I am now! | 19:21 |
| zaro | fungi: yeah, basically correct | 19:21 |
| fungi | zaro: thanks, much clearer now. i can see why they might not consider it a bug in that case. off-label use of a tool intended for a slightly different purpose | 19:22 |
| zaro | fungi: there is a 3rd option if it doesn't get accepted in gerrit core | 19:22 |
| zaro | fungi: we can add it to this plugin https://gerrit.googlesource.com/plugins/admin-console/ | 19:23 |
| *** ddieterly[away] is now known as ddieterly | 19:23 | |
| fungi | and now i get what you mean by "force" (normally it would automatically reindex on upgrade, but we want to force it to reindex even though it's not an upgrade) | 19:23 |
| zaro | *it/force reindex command | 19:23 |
| clarkb | therve: I just sat down ti lunch is 10 minutes from now good? | 19:23 |
| *** nadya has quit IRC | 19:24 | |
| therve | clarkb, Yep! | 19:24 |
| zaro | this would allow us to just install one plugin and not have to worry about groovy or scripts. unless groovy is something we want/need | 19:24 |
| zaro | which i don't think we really want. | 19:24 |
| fungi | zaro: is the admin-console plugin usable through rest api methods too? | 19:24 |
| fungi | not that it needs to be for our case, just curious what the options would be to leverage it | 19:25 |
| zaro | yeah, i think these are all ssh commands | 19:25 |
| zaro | they seem to extend the SshCommand class anyway | 19:25 |
| *** yamahata has joined #openstack-infra | 19:26 | |
| fungi | zaro: that seems like a reasonable alternative | 19:26 |
| fungi | zaro: unless we start running into multiple situations where we need to work around missing/incomplete features like this, i don't see there being a lot of motivation to write groovy scripts and so probably not a strong use case for adding the groovy plugin | 19:27 |
| fungi | zaro: if we didn't already have performance issues with gerrit i might not be as concerned, but every plugin is more surface area for added inefficiencies/leaks/bugs | 19:28 |
| zaro | fungi: so just to recap. right now only the groovy script is available. i will attempt to add force reindex feature to core gerrit. if that is rejected then i'll add it to the admin-console plugin. | 19:28 |
| *** ayoung has quit IRC | 19:29 | |
| zaro | fungi: so maybe we should just skip the groovy option? | 19:29 |
| zaro | which means online reindex will probably not be an option for next rename. | 19:30 |
| fungi | zaro: that's where i'm leaning at least. we still have some time before we'd be able to use it either way, and we already know to just plan for lengthy outages regardless since even if we have the solution because there's no guarantee it would work and then we have to be able to do an offline reindex anyway | 19:30 |
| fungi | other infra-root admins who also work on project renames and gerrit administration in general might feel differently though. i'm only one voice | 19:31 |
| *** mikelk has quit IRC | 19:34 | |
| clarkb | therve: ok | 19:34 |
| zaro | fungi: ok, unless you guys screem for groovy option i won't be doing any work to make the groovy script option available. | 19:36 |
| *** oanson has quit IRC | 19:36 | |
| clarkb | therve: I am struggling with a couple things, first how does thsi remove duplication? and we appear to lose tee's stdout with this method so it isn't preserving the old behavior of writing to stdout and a file | 19:36 |
| therve | Hum | 19:37 |
| clarkb | therve: specifically if we had double timestamps before that meant that tsfilter was getting already timestamped lines right? | 19:38 |
| therve | clarkb, That's correct | 19:38 |
| clarkb | therve: I think the proposed code would apply a second timestamp to those lines just like the existing code | 19:38 |
| *** cody-somerville has joined #openstack-infra | 19:38 | |
| clarkb | since it is still running it through tsfilter | 19:38 |
| therve | I have to say I did that a long time ago | 19:38 |
| therve | Let me test it again | 19:38 |
| *** bpokorny has joined #openstack-infra | 19:39 | |
| therve | clarkb, OK so from my test, what it does is send the timestamp to the file, but not to stdout | 19:40 |
| therve | clarkb, http://paste.openstack.org/show/497951/ | 19:40 |
| therve | stdout is then captured by the wrapper script, which calls tsfilter on it and apply it | 19:41 |
| therve | At least that's my understanding | 19:41 |
| *** tonytan4ever has joined #openstack-infra | 19:41 | |
| clarkb | therve: I actually think we are inverted here. The tsfilter that we are modifying there is the outermost tsfilter, so all other timestamps are applied before it runs | 19:41 |
| clarkb | therve: this has two problmes I think, the firstis that the log file will continue to be double timestamped and the second is if there is no inner timestamp attachment then we get no timestamps. This tsfilter was added because of the second issue | 19:42 |
| therve | clarkb, I thought hooks were running inside something else? | 19:42 |
| *** tonytan_brb has quit IRC | 19:42 | |
| *** nwkarsten has quit IRC | 19:42 | |
| clarkb | therve: well the do run inside the jenkins job which does console logs stuff including timestamps | 19:43 |
| clarkb | so maybe that is the real issue | 19:43 |
| clarkb | we want the console log and file on disk to match up | 19:43 |
| clarkb | but they separate attach logs | 19:43 |
| therve | AFAIU tsfilter is called in 2 places | 19:43 |
| *** reed_ has joined #openstack-infra | 19:43 | |
| therve | in run_command | 19:43 |
| therve | And in call_hook_if_defined | 19:44 |
| clarkb | run_command isn't in the hook stack I odn't think | 19:44 |
| clarkb | ya its just setup and cleanup for run_command | 19:45 |
| therve | Hum okay | 19:45 |
| therve | So the outside timestamp come from something else? | 19:46 |
| clarkb | therve: I think in the case of d-g hooks one of two things is happening either a) the hook itself is calling tsfilter in the hook and attaching timestamps there or b) we are bubbling those logs all the way up to the jenkins console log and jenkins is attaching the outer timestamps | 19:46 |
| therve | Hum | 19:47 |
| therve | If I look at http://logs.openstack.org/66/317866/3/check/gate-heat-dsvm-functional-orig-mysql-lbaasv1/03b19f6/console.html | 19:48 |
| therve | You're right that we have some tsfilter calls outside of hooks that don't add the double timestamps | 19:48 |
| therve | eg tsfilter setup_host | 19:48 |
| clarkb | ya there is a single timestamp in http://logs.openstack.org/66/317866/3/check/gate-heat-dsvm-functional-orig-mysql-lbaasv1/03b19f6/logs/devstack-gate-post_test_hook.txt.gz which means that jenkins is adding the outer timestamp | 19:49 |
| therve | So is the tee the issue? I don't see how | 19:49 |
| clarkb | therve: probably the thing to do in this case is to just stop letting stdout get out | 19:49 |
| clarkb | therve: ya I think its bceause we tee and let stdout get to jenkins | 19:49 |
| *** ericksonsantos has joined #openstack-infra | 19:50 | |
| * therve nods | 19:50 | |
| therve | So... | 19:50 |
| clarkb | if people are ok with not using the console log file and instead using the specific hook log files we can just dev null that or stop using tee entirely and just redirect straight to file | 19:50 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: [WIP] Add: lvm element https://review.openstack.org/316529 | 19:51 |
| therve | clarkb, But doesn't removing the timestamp from stdout work then? | 19:51 |
| clarkb | therve: yes but that has the issue of writing the same bits twice and if I get to be selfish thats bad because it uses more disk :) | 19:51 |
| clarkb | therve: I think we should choose where we want to write them, either to the separate lo gfile or to the console log and then fix it to work in that single case | 19:51 |
| clarkb | therve: that should simplify it quite a bit | 19:52 |
| clarkb | to write to the console log we would remove the tsfilter there and stop writing to the special file | 19:52 |
| therve | clarkb, I think I'd prefer using the console log indeed | 19:53 |
| clarkb | either way works for me, my biggest goal is write bits once and make that single place work correctly | 19:53 |
| therve | At least in our case the test run in the hook, so having to go to the hook file is a bit worse | 19:53 |
| therve | +1 | 19:53 |
| therve | except if people are actually relying on the hook output file? | 19:55 |
| therve | I don't personally | 19:55 |
| *** mhickey has quit IRC | 19:55 | |
| clarkb | I am not sure that anyone is. We don't index them in elasticsearch even | 19:55 |
| clarkb | It should be relatively safe given that ^ isn't using it | 19:55 |
| therve | Fair enough | 19:56 |
| therve | I'll update my patch to see if I understood correctly | 19:56 |
| clarkb | great, thank you for talking through that. I definitely understand the situation much better now | 19:57 |
| *** asettle has joined #openstack-infra | 19:57 | |
| *** jsavak has quit IRC | 19:58 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs https://review.openstack.org/286136 | 19:59 |
| *** jsavak has joined #openstack-infra | 19:59 | |
| *** asettle has quit IRC | 19:59 | |
| *** asettle has joined #openstack-infra | 19:59 | |
| *** asettle has quit IRC | 19:59 | |
| clarkb | therve: yup that is about what I expected. If none of the d-g check jobs use a pre or post test hook we can use depends-on that patch in a project that does to double check it too | 20:00 |
| openstackgerrit | Merged openstack-infra/project-config: Fix wheel-build.sh for Python 3 https://review.openstack.org/319236 | 20:01 |
| *** fawadkhaliq has quit IRC | 20:01 | |
| therve | clarkb, Ah depends-on will work on devstack-gate? I was wondering | 20:03 |
| therve | Let me do a quick heat patch | 20:03 |
| *** salv-orlando has joined #openstack-infra | 20:03 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Fix workaround to not install packages during dlrn build https://review.openstack.org/319433 | 20:03 |
| *** Apoorva has quit IRC | 20:04 | |
| *** ihrachys has joined #openstack-infra | 20:04 | |
| therve | clarkb, gate-swift-dsvm-functional-nv seems to use a hook so we're good | 20:04 |
| *** jsavak has quit IRC | 20:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add error logging around FIP delete https://review.openstack.org/319435 | 20:07 |
| *** salv-orlando has quit IRC | 20:08 | |
| Shrews | jeblair: clarkb: mordred: ^^^ | 20:08 |
| *** e0ne has quit IRC | 20:08 | |
| Shrews | would be nice to see nodepool running with that | 20:08 |
| *** _ari_ has quit IRC | 20:09 | |
| *** IlyaG has joined #openstack-infra | 20:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Hard-code deb-openstack-pkg-tools into upload job https://review.openstack.org/319343 | 20:10 |
| fungi | zigo: ^ | 20:10 |
| fungi | that simplifies things a bit for now | 20:11 |
| fungi | mordred: ^ (though i know you're not _really_ around) | 20:11 |
| *** banix has joined #openstack-infra | 20:11 | |
| fungi | while i was at it, i corrected a separate bug with the archive publisher not using the same path for the artifacts as the scp publisher | 20:12 |
| fungi | if reviewers feel strongly, we can update the commit message to mention that | 20:12 |
| *** e0ne has joined #openstack-infra | 20:12 | |
| fungi | but given the job isn't working completely yet anyway, it seems pedantic to enumerate all the ways it was broken | 20:13 |
| *** shashank_hegde has joined #openstack-infra | 20:13 | |
| *** dimtruck is now known as zz_dimtruck | 20:14 | |
| fungi | haypo: 319236 merged, so i suspect that really was a transient network failure and the error message was coming from a fallback behavior, masking the actual network fault | 20:18 |
| openstackgerrit | Merged openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST_ALL_PLUGINS for Ironic grenade https://review.openstack.org/318521 | 20:18 |
| Kaiyan | Hi this is Kaiyan from Monasca team, I'm trying to add monasca tempest tests as non-voting check job for all the monasca repos. But the new patch set failed "gate-project-config-layout". Can any one help me with it please? Thank you! https://review.openstack.org/#/c/316293/ | 20:18 |
| Kaiyan | It failed locally too when I ran to. compare-xml-new: commands failed | 20:18 |
| Kaiyan | *tox | 20:19 |
| *** sdake has quit IRC | 20:22 | |
| *** sdake has joined #openstack-infra | 20:22 | |
| fungi | Kaiyan: i left review comments explaining how to fix that | 20:22 |
| *** esikachev has joined #openstack-infra | 20:23 | |
| haypo | fungi: ok. i will check the job next week | 20:23 |
| *** nadya has joined #openstack-infra | 20:24 | |
| jeblair | Shrews: left a comment and a question on 319435 | 20:25 |
| *** sdake has quit IRC | 20:25 | |
| *** ayoung has joined #openstack-infra | 20:26 | |
| *** fawadkhaliq has joined #openstack-infra | 20:27 | |
| Shrews | jeblair: hrm, i think that's actually a copypasta, but weird we treat the two cases differently right now. i'll fix to keep current behavior | 20:27 |
| jeblair | Shrews: okay. i think that's safe considering the stated goal of just logging. i haven't really dug in to see if the behavior change would matter though. | 20:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add error logging around FIP delete https://review.openstack.org/319435 | 20:29 |
| kgiusti | Hi all - Q: I'm working on a feature branch (oslo.messaging/feature/amqp-dispatch-router) and I'd like to merge master into the feature branch. But I don't have push merge permissions. | 20:31 |
| kgiusti | Is this something a core can do for me? or do I need to get the permissions explicitly set? | 20:31 |
| *** e0ne has quit IRC | 20:32 | |
| clarkb | kgiusti: usually its given to the core group on the project | 20:32 |
| clarkb | if the currebt acl doesnt allow it the acl should be updated | 20:32 |
| kgiusti | kgiusti: interesting, I think the oslo.messaging acl only allows pushMerge for oslo-release, not oslo-messaging-core | 20:33 |
| *** _sarob has quit IRC | 20:33 | |
| kgiusti | clarkb: sorry - I was talking to myself again | 20:33 |
| kgiusti | clarkb: I'll check with the ptl | 20:34 |
| *** yamahata has quit IRC | 20:34 | |
| *** yamahata has joined #openstack-infra | 20:35 | |
| *** fawadkhaliq has quit IRC | 20:35 | |
| *** baoli_ has quit IRC | 20:36 | |
| *** baoli has joined #openstack-infra | 20:37 | |
| *** ddieterly is now known as ddieterly[away] | 20:38 | |
| *** EricGonc_ has quit IRC | 20:40 | |
| *** jsavak has joined #openstack-infra | 20:40 | |
| *** pramchan has joined #openstack-infra | 20:40 | |
| *** nwkarsten has joined #openstack-infra | 20:41 | |
| *** nwkarsten has quit IRC | 20:41 | |
| *** nwkarsten has joined #openstack-infra | 20:42 | |
| *** tonytan4ever has quit IRC | 20:42 | |
| *** cloudtra_ has joined #openstack-infra | 20:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Purge normalized URLs https://review.openstack.org/319449 | 20:43 |
| jeblair | mtreinish, fungi, dstufft: ^ | 20:43 |
| *** cloudtrainme has quit IRC | 20:43 | |
| *** baoli has quit IRC | 20:44 | |
| *** sdake has joined #openstack-infra | 20:45 | |
| JayF | hmm. If someone's address in gerrit is throwing bounces (well, autoresponder saying they aren't employed at $company anymore); do you all typically take action to prevent email to that domain? | 20:46 |
| fungi | JayF: nope | 20:46 |
| JayF | I can give the address/message in pm, just didn't want to say who isn't working where anymore in open chat | 20:46 |
| JayF | so basically I'm going to get bounces until this old spec merges. dray. | 20:46 |
| JayF | *drat | 20:47 |
| jeblair | (gerrit really should have bounce processing) | 20:47 |
| *** Apoorva has joined #openstack-infra | 20:47 | |
| clarkb | therve: looks like that function may have a unittest that is unhappy. Not sure if the test just needs new expectations or if its properly not working | 20:47 |
| Kaiyan | @fungi Thank you! I changed {} to {{}} and seems like its working locally! | 20:47 |
| fungi | JayF: if you as the sysadmin of teh mailserver there would like to "formally" complain that we're spamming your server, i could certainly see nullifying the address for you | 20:47 |
| JayF | jeblair: I've worked on software in that domain before. Pretty rough. | 20:47 |
| jeblair | JayF: verp ftw | 20:48 |
| JayF | jeblair: verb? | 20:48 |
| JayF | jeblair: verp? | 20:48 |
| Kaiyan | so for job-template, do we need the quote'' outside the name? | 20:48 |
| Kaiyan | in my case 'gate-{name}-tempest-dsvm' | 20:48 |
| jeblair | JayF: https://en.wikipedia.org/wiki/Variable_envelope_return_path it makes it much easier -- bypasses all that nasty parsing and stuff :) | 20:49 |
| JayF | jeblair: ah. I haven't worked much in the bounce-parsing business for almost 8 years :) | 20:49 |
| JayF | jeblair: I worked for a $not-a-spammer-we-swear company and we had to parse bounces + abuse reports | 20:49 |
| jeblair | JayF: if you send mail out with VERP return paths, and you later receive a message on that address, you're done -- it's a bounce for that address. you don't have to worry about things like "well, i need to parse an exchange bounce message this way to figure out who it was for" | 20:50 |
| jeblair | JayF: our mailman install uses it | 20:50 |
| jeblair | looks like "openstack-dev-bounces+corvus=inaugust.com@" | 20:51 |
| JayF | ah | 20:52 |
| JayF | yeah, we did similar stuff with return path for bounce and abuse parsing | 20:52 |
| JayF | all we'd do for actual parsing is to check the "bounce||abuse" to see if it matched feedback loop / spam report emails from various I/ESPs | 20:53 |
| *** raildo is now known as raildo-afk | 20:53 | |
| nwkarsten | clarkb: i'm getting started on the debugging we talked about yesterday. how do you turn on shade logging? | 20:54 |
| *** pramchan has quit IRC | 20:55 | |
| clarkb | nwkarsten: Shrews should be able to answer that (I don't know off the top of my head) | 20:56 |
| *** cody-somerville has quit IRC | 20:56 | |
| *** haypo has left #openstack-infra | 20:56 | |
| *** rprakash has joined #openstack-infra | 20:57 | |
| *** IlyaG has quit IRC | 20:57 | |
| *** rprakash has quit IRC | 20:57 | |
| *** dprince has quit IRC | 20:58 | |
| jeblair | nwkarsten: are you using the shade python api, or debugging an existing program with its own logging config? | 20:58 |
| jeblair | nwkarsten: if you're using the shade python api, you can call: shade.simple_logging(debug=True) | 20:58 |
| openstackgerrit | Merged openstack-infra/zuul: Set default_flow_style=False for readability https://review.openstack.org/317149 | 20:59 |
| *** tonytan4ever has joined #openstack-infra | 20:59 | |
| jeblair | nwkarsten: if you're working with some existing program that uses the python logging api, you can configure it something like how we do for nodepool: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/nodepool.logging.conf.erb#n19 | 20:59 |
| nwkarsten | jeblair: the shade python api. thanks | 20:59 |
| jeblair | nwkarsten: (but set level=DEBUG instead of WARNING) | 20:59 |
| jeblair | nwkarsten: cool, so that function is an easy start point. shade uses the python logging api internally, so if you want more control, or later want to better incorporate it into your app, i recommend using that. (i don't know what you're actually doing, so i'm covering all bases) | 21:00 |
| *** IlyaG has joined #openstack-infra | 21:01 | |
| *** claudiub has joined #openstack-infra | 21:01 | |
| nwkarsten | jeblair: i was planning on injecting it into the nodepool code to figure out why i can't connect to my cloud | 21:02 |
| jeblair | nwkarsten: ah, then actually just changing the logging config that nodepool uses is probably easiest | 21:03 |
| *** ddieterly[away] is now known as ddieterly | 21:03 | |
| *** gomarivera has quit IRC | 21:03 | |
| *** nadya has quit IRC | 21:03 | |
| jeblair | i'm going to divert some jobs to zuulv2.5 | 21:04 |
| *** claudiub|2 has quit IRC | 21:04 | |
| fungi | sounds good | 21:04 |
| *** inc0_ has quit IRC | 21:04 | |
| fungi | infra jobs again, presumably? | 21:04 |
| jeblair | fungi: no, random. nodepool isn't really set up for much else :( | 21:05 |
| *** ddieterly is now known as ddieterly[away] | 21:05 | |
| openstackgerrit | Ken Giusti proposed openstack-infra/project-config: Add pushMerge for oslo.messaging core for feature branches https://review.openstack.org/319452 | 21:05 |
| *** inc0_ has joined #openstack-infra | 21:05 | |
| *** kgiusti has left #openstack-infra | 21:05 | |
| fungi | oh, right-o | 21:05 |
| *** cody-somerville has joined #openstack-infra | 21:05 | |
| *** cody-somerville has joined #openstack-infra | 21:05 | |
| fungi | i guess you could maybe divert by node label | 21:05 |
| fungi | (infra is the only consumer of ubuntu-precise!) | 21:06 |
| *** akshai has quit IRC | 21:06 | |
| fungi | but friday afternoons are yolo territory anyway as far as i'm concerned | 21:07 |
| jeblair | ++ | 21:07 |
| clarkb | jeblair: I hav reviewed the zk for nodepool and zuul interaction spec | 21:07 |
| jeblair | clarkb: thanks! | 21:08 |
| fungi | also worth noting, our fip quotas in bluebox and osic are completely full of leaked addresses | 21:08 |
| fungi | i think they have been all day | 21:08 |
| openstackgerrit | Brian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin https://review.openstack.org/319368 | 21:08 |
| *** tonytan4ever has quit IRC | 21:08 | |
| fungi | so we've really not been using those ~150 nodes worth of quota | 21:08 |
| jeblair | fungi: :( and we reverted the fip cleanup, right? | 21:08 |
| fungi | yeah, pending the shade-side fix | 21:08 |
| clarkb | because it was crashy | 21:09 |
| fungi | used a previously unexercised and untested method in shade | 21:09 |
| fungi | which, as you'd expect, therefore did not actually work | 21:09 |
| fungi | pabelanger worked on writing a fix and also some tests to exercise that code from now on | 21:10 |
| clarkb | hrm I thought that the claim was test wasn't needed in nodepool because it was tested in sahde :( | 21:10 |
| *** lucas-afk has quit IRC | 21:11 | |
| *** thorst has quit IRC | 21:11 | |
| *** pradk has quit IRC | 21:11 | |
| fungi | maybe nobody realized the testing for it in shade didn't exercise the same code paths/arguments nodepool used | 21:11 |
| *** thorst has joined #openstack-infra | 21:11 | |
| Kaiyan | @fungi, {{}} fixed the compare-xml-new problem. But when I run tox locally it is still failing with other errors. Here is the summary I got with running to: compare-xml-new: commands succeeded congratulations :) find: -printf: unknown primary or operator ERROR: InvocationError: '/Users/kaiyansheng/python_repos/project-config/tools/run-layout.sh' | 21:11 |
| Kaiyan | linters: commands succeeded ERROR: gerrit: commands failed grafyaml: commands succeeded ERROR: projects: commands failed jjb: commands succeeded jenkins-project: commands succeeded ERROR: zuul: commands failed | 21:12 |
| Kaiyan | But if I only run ./tools/run-layout.sh, it passed... | 21:12 |
| pabelanger | fungi: clarkb: jeblair: The tests in shade are currently wrong, they test (and validate) the broken code path. See: https://review.openstack.org/#/c/318369/ | 21:12 |
| Kaiyan | why it complained about a InvocationError when I ran tox then? | 21:13 |
| *** trown is now known as trown|outtypewww | 21:13 | |
| fungi | Kaiyan: hard to say without seeing the full log. you could push your modified change up for a new patchset and we can just look at what the ci reports | 21:14 |
| *** alexey_weyl has joined #openstack-infra | 21:14 | |
| *** jsavak has quit IRC | 21:14 | |
| alexey_weyl | Hi | 21:14 |
| Kaiyan | Sounds good i can do that! Thanks! | 21:14 |
| fungi | Kaiyan: looks like it's possible you're running that on a platform which doesn't support the same options to the find command, for example | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| fungi | Kaiyan: what platform did you try it on? we run that particular test on ubuntu 12.04 lts, i'm pretty sure | 21:15 |
| fungi | mainly because the service which consumes that configuration is currently running on an ubuntu 12.04 lts server | 21:15 |
| *** alexey_weyl has quit IRC | 21:15 | |
| clarkb | pabelanger: a case of mocks are badness | 21:16 |
| *** thorst has quit IRC | 21:16 | |
| fungi | Kaiyan: the "find: -printf: unknown primary or operator" bit is what makes me wonder | 21:16 |
| pabelanger | clarkb: indeed | 21:16 |
| pabelanger | clarkb: also, I can monitor stackviz build if needed | 21:16 |
| *** cody-somerville has quit IRC | 21:17 | |
| clarkb | pabelanger: ok you have the WIP on the change and said you would so was going to let you approve when ready | 21:17 |
| *** lucasagomes has joined #openstack-infra | 21:18 | |
| pabelanger | clarkb: WIP removed, I can approve this weekend to test the build | 21:18 |
| *** jerryz has quit IRC | 21:19 | |
| *** rcernin has joined #openstack-infra | 21:20 | |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add check-requirements check job for monasca projects https://review.openstack.org/317735 | 21:20 |
| *** markvoelker has joined #openstack-infra | 21:21 | |
| *** daemontool has quit IRC | 21:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:24 | |
| mordred | pabelanger, fungi: did we change the OSIC rate back to a higher rate? | 21:24 |
| mordred | or are we leaking still at the lower rate? | 21:24 |
| *** markvoelker has quit IRC | 21:24 | |
| fungi | mordred: i do not know whether the change to raise the delay for osic merged | 21:24 |
| fungi | i'll check | 21:24 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca]Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 21:24 |
| *** markvoelker has joined #openstack-infra | 21:24 | |
| openstackgerrit | David Lyle proposed openstack-infra/devstack-gate: Test patch https://review.openstack.org/319409 | 21:25 |
| fungi | doesn't look like it's gotten approved yet. hunting it down now | 21:25 |
| pabelanger | mordred: fungi: I manually tested that yesterday, we still leaked FIPs at 0.5 | 21:25 |
| *** flepied1 has quit IRC | 21:25 | |
| fungi | oh, poo | 21:25 |
| pabelanger | mordred: fungi: it wasn't as fast, but we still leaked | 21:25 |
| *** markvoelker_ has joined #openstack-infra | 21:25 | |
| *** flepied has joined #openstack-infra | 21:26 | |
| fungi | mordred: if you're not around, you might want to not review the 318369 fix to shade's floating ip handling | 21:26 |
| fungi | Shrews has a comment raising specific questions about your intent with it | 21:27 |
| openstackgerrit | Brian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin https://review.openstack.org/319368 | 21:27 |
| *** IlyaG has quit IRC | 21:27 | |
| *** ddieterly[away] is now known as ddieterly | 21:28 | |
| openstackgerrit | David Lyle proposed openstack-infra/devstack-gate: Test patch https://review.openstack.org/319409 | 21:28 |
| *** markvoelker_ has quit IRC | 21:28 | |
| *** markvoelker_ has joined #openstack-infra | 21:28 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 21:29 |
| *** markvoelker has quit IRC | 21:30 | |
| *** matt-borland has quit IRC | 21:32 | |
| *** thorst has joined #openstack-infra | 21:34 | |
| asselin__ | Any puppet/Gem folks know what the correct format for this is supposed to be? There was an error parsing `Gemfile`: Illformed requirement ["'3.8.1'"]. Bundler cannot continue. There was an error parsing `Gemfile`: Illformed requirement ["'~> 3.8.1'"]. Bundler cannot continue. | 21:35 |
| asselin__ | nibalizer, ^^ ?? | 21:35 |
| clarkb | asselin__: maybe no space? | 21:35 |
| *** lakshmiS has quit IRC | 21:35 | |
| asselin__ | will try it | 21:35 |
| clarkb | no apparently the space is normal | 21:36 |
| asselin__ | here shows a space: https://github.com/garethr/puppet-module-skeleton/blob/master/skeleton/.travis.yml | 21:36 |
| clarkb | asselin__: I have found examples of both so assuming that either is fine | 21:36 |
| *** aysyd has quit IRC | 21:37 | |
| clarkb | asselin__: ~> means >= 3.8.1 and < 4.0 | 21:37 |
| clarkb | so the ~> should be valid too | 21:37 |
| *** xyang1 has quit IRC | 21:37 | |
| *** thorst has quit IRC | 21:38 | |
| asselin__ | this is using bundler-1.12.4 | 21:38 |
| *** IlyaG has joined #openstack-infra | 21:40 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Print installed tempest-plugins after tempest run https://review.openstack.org/319465 | 21:40 |
| clarkb | mtreinish: ^ reminded me, any progress using a common venv for tempest? | 21:41 |
| mtreinish | clarkb: it's all done | 21:41 |
| mtreinish | clarkb: david-lyle is testing using it for the horizon plugin right now: https://review.openstack.org/#/c/319409/3 | 21:42 |
| clarkb | woot, did that speed up jobs by a couple minutes? | 21:42 |
| mtreinish | the real patch is: https://review.openstack.org/314795 | 21:42 |
| mtreinish | clarkb: you know I never checked, let me look at openstack-health | 21:42 |
| clarkb | I have a d-g change up that should give us back a minute or two as well | 21:43 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Prevent corruption of cache by using atomic update https://review.openstack.org/314789 | 21:43 |
| clarkb | (related to making swapfiles) | 21:43 |
| *** cody-somerville has joined #openstack-infra | 21:45 | |
| openstackgerrit | Prakash Ramchandran proposed openstack-infra/project-config: add new project meghdwar https://review.openstack.org/319466 | 21:45 |
| *** kzaitsev_mb has quit IRC | 21:45 | |
| mtreinish | clarkb: hmm no noticeable change in devstack: http://status.openstack.org/openstack-health/#/test/devstack?groupKey=project&resolutionKey=hour&end=2016-05-20T21:44:54.691Z | 21:46 |
| mtreinish | the patch landed on may 9th | 21:46 |
| mtreinish | https://review.openstack.org/#/c/312615/ | 21:46 |
| * mtreinish just noticed we don't graph run execute time anywhere on o-h | 21:47 | |
| mtreinish | I should fix that | 21:47 |
| mtreinish | (for the whole job) | 21:47 |
| clarkb | graphite has the data | 21:47 |
| mtreinish | clarkb: in this case devstack was always making to venvs, and d-g an occasional 3rd | 21:48 |
| *** savihou has quit IRC | 21:48 | |
| clarkb | and now we are down to one right? | 21:48 |
| mtreinish | yep | 21:48 |
| *** kzaitsev_mb has joined #openstack-infra | 21:48 | |
| mtreinish | except for the all-plugin job, then it's 2 (that 1 has to be seperate because it uses system site-packages) | 21:48 |
| *** ffrank has joined #openstack-infra | 21:49 | |
| *** lascii is now known as alaski | 21:50 | |
| *** cody-somerville has quit IRC | 21:50 | |
| mtreinish | clarkb: unrelated can you check if elastic-recheck got cloned and installed in the openstack-health venv on health.o.o? | 21:51 |
| mtreinish | I don't see any evidence the recent puppet changes took effect on puppetboard | 21:51 |
| *** kzaitsev_mb has quit IRC | 21:53 | |
| clarkb | mtreinish: there is a /opt/elastic-recheck | 21:53 |
| mtreinish | ok cool | 21:53 |
| mtreinish | then I guess it ran and just didn't say anything on puppetboard | 21:54 |
| *** eventingmonkey has quit IRC | 21:54 | |
| mtreinish | I guess I'll have to wait ~30min for the cache to go stale to see if it picks up anything on e-r on the next refresh | 21:55 |
| *** Swami_ has joined #openstack-infra | 21:56 | |
| clarkb | mtreinish: we don't report failures properly to puppetboard anymore | 21:56 |
| *** jsavak has joined #openstack-infra | 21:57 | |
| *** dmk0202 has quit IRC | 21:57 | |
| clarkb | but I see no errors in syslog | 21:57 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Make reproduce function quiet https://review.openstack.org/296171 | 21:57 |
| *** cody-somerville has joined #openstack-infra | 21:58 | |
| mtreinish | clarkb: yeah I think it worked fine it just didn't show the exec or the new VCS repo on puppetboard | 21:58 |
| mtreinish | clarkb: it's definitely applied the latest patch: http://health.openstack.org/tests/recent/fail has the bug field now | 21:59 |
| mtreinish | it's just empty right now | 21:59 |
| Kaiyan | @fungi I ran it on macbook hmmm | 21:59 |
| Kaiyan | @fungi now it passed the check job! | 21:59 |
| *** Swami has quit IRC | 21:59 | |
| *** sdake has quit IRC | 21:59 | |
| fungi | Kaiyan: that explains it. -printf is probably not covered by posix, likely a gnu-ism | 22:00 |
| *** Swami_ has quit IRC | 22:00 | |
| fungi | (or perhaps osx's bsd find is non-posix compliant?) | 22:00 |
| fungi | regardless, i'm unsurprised that script doesn't work as intended on non-gnu-linux | 22:00 |
| Kaiyan | ohhh gotcha! Thank you!!! | 22:01 |
| Kaiyan | yeah that make sense! | 22:01 |
| *** kus has quit IRC | 22:01 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:01 | |
| mtreinish | fungi: http://www.opengroup.org/openbrand/register/brand3612.htm | 22:01 |
| clarkb | I almost missed http://www.openstack.org/blog/2016/05/faq-evolving-the-openstack-design-summit/ so putting it here now for others that may have missed it | 22:02 |
| mtreinish | fungi: so probably a gnuism | 22:03 |
| *** burgerk_ has quit IRC | 22:04 | |
| *** sarob has joined #openstack-infra | 22:04 | |
| *** _sarob has joined #openstack-infra | 22:05 | |
| *** ddieterly has quit IRC | 22:06 | |
| mtreinish | fungi: although I don't see find on the commands tested: http://www.opengroup.org/csq/repository/norationale=1&noreferences=1&RID=apple%252FCX1%25252F8.html | 22:07 |
| mtreinish | I do like how they check for a fortran 77 compiler though | 22:08 |
| fungi | mtreinish: a sub-possibility of my "-printf is probably not covered by posix" guess. find on the whole is perhaps just not covered | 22:09 |
| *** sarob has quit IRC | 22:09 | |
| *** sdake has joined #openstack-infra | 22:09 | |
| *** inc0_ has quit IRC | 22:09 | |
| fungi | yeah, posix is an interesting tour of things people thought at one time would be important to standardize but which often had little relevance in hindsight | 22:10 |
| pleia2 | aw "Your proposal for Jenkins World 2016 has not been accepted" (nibalizer submitted a refreshed version of how we use Jenkins) | 22:10 |
| *** cloudtra_ has quit IRC | 22:11 | |
| pleia2 | but it's probably for the best, given zv3 and all | 22:11 |
| timrc | :P | 22:11 |
| timrc | "How to systematically review Jenkins from your CI" | 22:11 |
| timrc | remove* | 22:11 |
| timrc | lol @ review | 22:11 |
| pleia2 | hehe | 22:11 |
| timrc | I think my brain and fingers are working together to tell me it's time for a beer. | 22:12 |
| fungi | just now? | 22:12 |
| *** vhosakot has quit IRC | 22:12 | |
| *** vhosakot_ has joined #openstack-infra | 22:12 | |
| fungi | i guess texas is an "after 5" crowd | 22:12 |
| *** regXboi has quit IRC | 22:13 | |
| *** sridhar_ram has joined #openstack-infra | 22:16 | |
| jeblair | i just wrote a 3 line function because it needs 18 lines of comments. thanks rsync! | 22:16 |
| *** esikachev has quit IRC | 22:16 | |
| *** amitgandhinz has quit IRC | 22:18 | |
| *** HeOS has quit IRC | 22:20 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Install the Gerrit delete-project plugin https://review.openstack.org/319400 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH https://review.openstack.org/318975 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command https://review.openstack.org/318982 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP https://review.openstack.org/318979 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers https://review.openstack.org/318411 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME https://review.openstack.org/318987 | 22:21 |
| *** flepied has quit IRC | 22:21 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Install the Gerrit delete-project plugin https://review.openstack.org/319400 | 22:24 |
| *** stevemar is now known as notstevemar | 22:24 | |
| *** IlyaG has quit IRC | 22:24 | |
| *** jsavak has quit IRC | 22:24 | |
| *** zz_dimtruck is now known as dimtruck | 22:26 | |
| *** tpsilva has quit IRC | 22:27 | |
| timrc | fungi: It could be sooner than 5... desire > effort is usually the tipping point ;-) | 22:28 |
| *** bhunter71 has quit IRC | 22:30 | |
| *** |-paul-| has joined #openstack-infra | 22:30 | |
| *** bauwser is now known as bauzas | 22:35 | |
| *** camunoz_ has joined #openstack-infra | 22:36 | |
| nibalizer | pleia2: haha | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command https://review.openstack.org/318982 | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP https://review.openstack.org/318979 | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME https://review.openstack.org/318987 | 22:37 |
| clarkb | mtreinish: small bug in https://review.openstack.org/#/c/319465/1/devstack-vm-gate.sh | 22:38 |
| mtreinish | clarkb: oops, good catch | 22:40 |
| mtreinish | I'll respin it right now | 22:40 |
| jeblair | okay, one more zuulv2.5 test with those ^ | 22:40 |
| *** Kaiyan has quit IRC | 22:42 | |
| pleia2 | nibalizer: mind reader | 22:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Print installed tempest-plugins after tempest run https://review.openstack.org/319465 | 22:42 |
| mtreinish | clarkb: ^^^ | 22:42 |
| *** ddieterly has joined #openstack-infra | 22:43 | |
| *** ddieterly is now known as ddieterly[away] | 22:43 | |
| *** ddieterly[away] is now known as ddieterly | 22:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH https://review.openstack.org/318975 | 22:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command https://review.openstack.org/318982 | 22:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP https://review.openstack.org/318979 | 22:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers https://review.openstack.org/318411 | 22:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME https://review.openstack.org/318987 | 22:45 |
| *** jamielennox|away is now known as jamielennox | 22:48 | |
| nibalizer | pleia2: o/ | 22:49 |
| pleia2 | nibalizer: one of my close friends was just hired to maintain the infra for gluster <3 | 22:50 |
| *** Swami has joined #openstack-infra | 22:50 | |
| nibalizer | w00t | 22:50 |
| *** chem`` has quit IRC | 22:50 | |
| *** chem`` has joined #openstack-infra | 22:51 | |
| jeblair | OH! we could have zuulv2.5 *not* upload logs when zuul aborts a running job! | 22:52 |
| clarkb | ooh fancy features | 22:53 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Give example #startmeeting command https://review.openstack.org/318894 | 22:53 |
| notmorgan | jeblair: oooh | 22:53 |
| jeblair | (zuul aborts a lot of jobs, that should save us some space) | 22:53 |
| notmorgan | hmm | 22:54 |
| asselin__ | clarkb, just to follow-up. The version syntax is correct ~>, just that jenkins environment injection doesn't work with that. Exporting it directly works in the jenkins job solves the issue. | 22:55 |
| fungi | heads up that http://lists.openstack.org/pipermail/openstack-dev/2016-May/095598.html might be good to include some infra peeps with a vested interest in glance consistency/interoperability (note it's also during our precise-to-trusty upgrades week) | 22:56 |
| *** krotscheck is now known as krotscheck_dcm | 22:57 | |
| *** xarses has quit IRC | 22:57 | |
| notmorgan | fungi: out of curiosity, we're planning to keey trusty around for a while? I noticed xenial ships with py35 not py34 (though the differences are relatively minor) | 22:59 |
| *** salv-orlando has joined #openstack-infra | 22:59 | |
| notmorgan | fungi: (i ran into this when doing some py34 fixes for keystone) | 22:59 |
| fungi | notmorgan: yeah, the idea is to test >=newton on xenial (so probably py35 jobs will get added) | 23:00 |
| *** bpokorny has quit IRC | 23:00 | |
| notmorgan | fungi: ah okie. | 23:00 |
| notmorgan | cool. that was my thought but just confirming | 23:00 |
| fungi | notmorgan: <newton sticks to trusty, and we can continue to test py34 there | 23:00 |
| * notmorgan is babysitting the final py34 fixes for keystone though the gate. | 23:00 | |
| *** csomerville has joined #openstack-infra | 23:01 | |
| notmorgan | had to compile-from-source py34 so i didn't need to run a VM on my laptop | 23:01 |
| notmorgan | i mena... sure that *sounds* insane when i say it outloud | 23:01 |
| fungi | compiling software from source sounds insane? | 23:02 |
| notmorgan | no compiling software from source to avoid running a VM or docker container... | 23:02 |
| notmorgan | that sounds kindof insane | 23:02 |
| *** hongbin has quit IRC | 23:02 | |
| fungi | meh | 23:03 |
| *** sdague has quit IRC | 23:03 | |
| * fungi has done that plenty | 23:03 | |
| notmorgan | i haven't figured out where i misplaced _curses though in my py34 build :P | 23:03 |
| *** |-paul-| has quit IRC | 23:03 | |
| *** ams__ has quit IRC | 23:03 | |
| notmorgan | so i can't use bpython | 23:03 |
| jeblair | http://logs.openstack.org/37/319437/2/check/gate-python-neutronclient-python34/d5ad213/ | 23:04 |
| jeblair | this is starting to look really close... | 23:04 |
| *** cody-somerville has quit IRC | 23:05 | |
| *** csomerville has quit IRC | 23:05 | |
| clarkb | notmorgan: fwiw everyone I have talked to says py34 just works under py35 | 23:05 |
| clarkb | the other direction not so much | 23:05 |
| notmorgan | clarkb: that is my experience | 23:05 |
| notmorgan | clarkb: which is great. | 23:06 |
| clarkb | since py35 introduces type annotations and other stuff | 23:06 |
| notmorgan | yeah i kindof dig that | 23:06 |
| notmorgan | fwiw | 23:06 |
| notmorgan | but unfortuantely we wont get to use that for a nice long time :P | 23:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Append to console log, rather than truncating https://review.openstack.org/319488 | 23:07 |
| notmorgan | clarkb: fwiw, keystone should be python3 compat today :) | 23:07 |
| clarkb | notmorgan: are you gatibg on it? | 23:07 |
| jeblair | that will address the surprisingly short console logs :) | 23:07 |
| notmorgan | clarkb: yes. will need to re-check 2 patches | 23:08 |
| notmorgan | clarkb: and we'll have no more blacklisted tests | 23:08 |
| notmorgan | clarkb: the biggest barrier was pyldap vs python-ldap | 23:08 |
| notmorgan | and ldappool. | 23:08 |
| clarkb | notmorgan: I also spoke to pypy people yesterday and they seem keen to do get pypy testing going again | 23:08 |
| jeblair | clarkb: i think this is the major zuulv2.5 mystery remaining: http://logs.openstack.org/65/319465/2/check/gate-tempest-dsvm-neutron-full/070394f/console.txt | 23:08 |
| clarkb | notmorgan: they also said keystone + eventlet + pypi is quicker than with wsgi server | 23:08 |
| notmorgan | well eventlet will not happen in keystone | 23:09 |
| notmorgan | clarkb: it's been dropped from our code base 100% | 23:09 |
| jeblair | i'm guessing wsgi is pypy incompatible | 23:10 |
| clarkb | ya I told tgen that they shouldve said somethinguch sooner | 23:10 |
| notmorgan | jeblair: as is uwsgi, but gunicorn should be compat | 23:10 |
| clarkb | jeblair: it runs but I think it has trouble with jit warmup | 23:10 |
| clarkb | among other things | 23:10 |
| jeblair | ah | 23:10 |
| notmorgan | jeblair: the issue we have with gunicorn is oslo_config stomps all over gunicorn's config | 23:10 |
| *** asselin has joined #openstack-infra | 23:11 | |
| notmorgan | but i bet gunicorn + pypy + keystone will be good once we solve oslo_config issues | 23:11 |
| * notmorgan prefers uwsgi. | 23:11 | |
| clarkb | jeblair: file perms maybe? | 23:11 |
| clarkb | at least with apache mod wsgi I am not sure processes live long enough to achieve max jit goodness | 23:12 |
| clarkb | not aure if gunicorn hamdles that better | 23:12 |
| jeblair | clarkb: i guess there are 2 mysteries -- 1) why no logs? and 2) why exit? | 23:12 |
| notmorgan | clarkb they can w/ mod_wsgi | 23:12 |
| notmorgan | clarkb: itr's just config driven | 23:12 |
| notmorgan | but... meh | 23:12 |
| jeblair | clarkb: i was able to hold that node, so that should help | 23:12 |
| jeblair | clarkb: it's at 149.202.165.169 | 23:12 |
| notmorgan | clarkb, jeblair: what is the rule about dropping things from g-r | 23:13 |
| notmorgan | code search doesn't seem to indicate anyone in newton depends on python-ldap | 23:13 |
| jeblair | clarkb: i suspect the symlink at logs/ may be part of the cause of (1) | 23:13 |
| clarkb | jeblair: no logs because d-g isnt there | 23:13 |
| * notmorgan is checking before propsing it. | 23:13 | |
| *** marcusvrn_ has quit IRC | 23:13 | |
| jeblair | clarkb: there is a logs directory | 23:13 |
| jeblair | 2016-05-20 22:55:06.449 | rsync: mkdir "/opt/stack/new/devstack" failed: Permission denied (13) | 23:14 |
| *** asselin__ has quit IRC | 23:14 | |
| *** gyee has quit IRC | 23:15 | |
| jeblair | nope, that's too late, sorry; actual cause is earlier i think | 23:15 |
| jeblair | 2016-05-20 22:55:06.336 | /home/jenkins/workspace/gate-tempest-dsvm-neutron-full/devstack-gate/functions.sh: line 354: blkid: command not found | 23:15 |
| jeblair | clarkb: i think we assume /sbin is in the path :) | 23:16 |
| clarkb | ah | 23:16 |
| fungi | path yeah | 23:16 |
| jeblair | so we could add it, or perhaps this is another thing that should nudge us to just running this stuff in a login shell :) | 23:16 |
| fungi | blkid is in util-linux after all | 23:16 |
| jeblair | yeah, which says it's in sbin | 23:17 |
| clarkb | ++ to using login shell | 23:17 |
| *** xarses has joined #openstack-infra | 23:17 | |
| jeblair | okay, i'll do that and drop our PATH and HOME stuff; then figure out what the deal is with rsync and symlinks | 23:17 |
| fungi | may need to take extra care with child processes inheriting file descriptors in that case? | 23:18 |
| fungi | i know running devstack-gate/tempest under an interactive shell has often left me unable to disconnect after without hunting down child processes to kill because they had refcount on stdin/stdout/stderr | 23:19 |
| *** ihrachys has quit IRC | 23:19 | |
| fungi | that's at least one up-side to the jenkins slave agent model | 23:20 |
| jeblair | i will probably "/bin/bash -l -c /tmp/abcdef.sh" | 23:21 |
| fungi | should help, yeah | 23:21 |
| fungi | i think it was mostly stuff forking and disassociating from child processes but not closing inherited file descriptors | 23:21 |
| fungi | so they weren't proper children any longer | 23:22 |
| * fungi starts to worry he's giving questionable parenting advice now | 23:22 | |
| *** nwkarsten has quit IRC | 23:23 | |
| *** roxanagh_ has quit IRC | 23:25 | |
| *** vhosakot_ has quit IRC | 23:25 | |
| * fungi believes it was dr. spock who said, "fork once, close your inherited file descriptors, reset your cwd to /, unset the foreground process group, then fork again!" | 23:27 | |
| *** nwkarsten has joined #openstack-infra | 23:27 | |
| *** Swami_ has joined #openstack-infra | 23:27 | |
| *** charz_ has quit IRC | 23:27 | |
| *** timburke has quit IRC | 23:27 | |
| *** roxanagh_ has joined #openstack-infra | 23:30 | |
| *** Swami has quit IRC | 23:31 | |
| *** charz has joined #openstack-infra | 23:32 | |
| *** nwkarsten has quit IRC | 23:32 | |
| *** salv-orlando has quit IRC | 23:33 | |
| *** jpr has quit IRC | 23:35 | |
| notmorgan | fungi: lol | 23:36 |
| *** timburke has joined #openstack-infra | 23:38 | |
| *** jerryz has joined #openstack-infra | 23:40 | |
| mnaser | does nodepool build raw images at the moment? | 23:45 |
| fungi | mnaser: it builds raw/vhd and qcow2 | 23:48 |
| fungi | mnaser: we prefer to upload qcow2 to providers who support it since that has native compression applied | 23:48 |
| *** bpokorny has joined #openstack-infra | 23:48 | |
| *** Swami__ has joined #openstack-infra | 23:48 | |
| mnaser | fungi: hmm, this has positives and negatives, if you upload qcow2 it'll be quite slow to deploy (it has to download the image, convert to raw and upload it back to ceph) | 23:49 |
| mnaser | vs if it's raw, it'll just make a new cow volume | 23:49 |
| *** markvoelker_ has quit IRC | 23:49 | |
| openstackgerrit | David Lyle proposed openstack-infra/devstack-gate: Test patch https://review.openstack.org/319409 | 23:50 |
| fungi | mnaser: the flip side is we frequently fail to upload raw versions of our images to providers because the significantly larger size makes them more prone to network disruption, timeouts, et cetera | 23:50 |
| fungi | and in theory the network inside the provider is faster and more stable than the internet at large | 23:50 |
| mnaser | fungi: i see where you're coming from, that's perfectly valid | 23:51 |
| mnaser | fungi: wonder if this could be worked around by having glance tasks that import qcow and do the conversion on the glance host | 23:52 |
| *** Swami_ has quit IRC | 23:52 | |
| mnaser | sorta like how some other provider do conversions to vhd | 23:52 |
| *** pvaneck has quit IRC | 23:52 | |
| openstackgerrit | Brian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin https://review.openstack.org/319368 | 23:52 |
| fungi | mnaser: i thought more recent glance releases had implemented that, but i may have been misinformed | 23:52 |
| mnaser | i think this was introduced in kilo (tasks api + conversion) | 23:54 |
| mnaser | https://specs.openstack.org/openstack/glance-specs/specs/kilo/conversion-of-images.html | 23:55 |
| *** kus has joined #openstack-infra | 23:59 | |
| *** kus has quit IRC | 23:59 | |
| *** roxanagh_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!