| *** spzala has quit IRC | 00:00 | |
| sc68cal | clarkb: mordred: anteaya: I'm OK with a patch that switches accept_ra = 2, | 00:01 |
|---|---|---|
| *** david-lyle has quit IRC | 00:01 | |
| clarkb | sc68cal: why not stop setting forwarding across the baord? | 00:01 |
| sc68cal | because then your instances wouldn't be able to reach the world. | 00:02 |
| clarkb | it seems like the default should be the leave the physical interface alone and let people munge that directly if they desire | 00:02 |
| clarkb | sc68cal: that is not necessary for development | 00:02 |
| clarkb | and for the people that want it they can munge their interface that way | 00:02 |
| mordred | clarkb: I believe people want to be able to run apt-get install in guests | 00:02 |
| mordred | in the gate | 00:02 |
| sc68cal | ^ this | 00:02 |
| clarkb | mordred: really? | 00:02 |
| mordred | yes | 00:02 |
| sc68cal | sdague ran into this and was annoyed | 00:02 |
| clarkb | but why | 00:02 |
| anteaya | I do | 00:02 |
| mordred | because lots of projects have things htat run in guests | 00:02 |
| anteaya | well not so much in the gate myself | 00:02 |
| anteaya | but I do like apt get install | 00:03 |
| sc68cal | OK - anyway I'm good with accept_ra = 2 - sorry I should have caught that years ago but nobody had real IPv6 infra to test this on for a while so it slipped through | 00:03 |
| clarkb | mordred: fwiw I don't think any of the gate tests have that working today | 00:03 |
| sc68cal | I gotta leave for dinner | 00:03 |
| clarkb | mordred: since we don't plumb that through by default iirc | 00:03 |
| mordred | sc68cal: thanks! | 00:03 |
| anteaya | sc68cal: thank you | 00:03 |
| anteaya | sc68cal: enjoy dinner | 00:03 |
| clarkb | mordred: guests can talk to each other and the hypervisors can talk to the guests thats it | 00:03 |
| clarkb | mordred: since br-ex by default is not plumbed through to the "physical" nic | 00:04 |
| mordred | clarkb: well, it may be worth finding a time in barcelona to sit down with sdague and sc68cal and talk through needs/desires/interfaces there | 00:04 |
| *** tonytan4ever has joined #openstack-infra | 00:04 | |
| mordred | clarkb: perhaps accept_ra = 2 gets us breathing room until we can circle up at a table in spain? | 00:04 |
| clarkb | mordred: sure I think that will work | 00:04 |
| clarkb | are we wanting to change that on our images or in devstack? | 00:05 |
| mordred | I think devstack | 00:05 |
| *** amotoki has joined #openstack-infra | 00:05 | |
| clarkb | ok. we should tell the people not using devstack that they will need to do similar | 00:05 |
| mordred | probably at the same places that does the sysctl for forwarding I'd guess | 00:05 |
| clarkb | kolla osa etc | 00:05 |
| mordred | yah | 00:05 |
| *** zhurong has quit IRC | 00:05 | |
| mordred | I mean- we could also do it in image setup | 00:05 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 00:05 |
| jeblair | --image setup | 00:05 |
| jeblair | i don't really want to be responsible for this :) | 00:05 |
| mordred | yha | 00:05 |
| mordred | clarkb: what's the full key? I've got devstack open to the place | 00:06 |
| clarkb | ya I don't think we should be responsible for it | 00:06 |
| pabelanger | ++ | 00:06 |
| *** jerryz has quit IRC | 00:06 | |
| *** tonytan4ever has quit IRC | 00:06 | |
| clarkb | uh let me find it | 00:07 |
| *** tonytan4ever has joined #openstack-infra | 00:07 | |
| mordred | clarkb: is it net.ipv6.conf.all.accept_ra = 2 ? | 00:07 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 00:07 |
| *** david-lyle has joined #openstack-infra | 00:07 | |
| clarkb | mordred: net.ipv6.conf.all.accept_ra = 2 | 00:07 |
| clarkb | mordred: please write a nice comment for why too | 00:08 |
| clarkb | mordred: that way in 6 months when someone thinks we were silyl for breaking the rfcs we have a reason | 00:08 |
| *** ddieterly has quit IRC | 00:09 | |
| mordred | clarkb: https://review.openstack.org/359490 | 00:09 |
| mordred | lemme make a longer comment | 00:09 |
| clarkb | mordred: and lets maybe recheck it a lot to make sure it runs happily on osic | 00:10 |
| *** amotoki has quit IRC | 00:10 | |
| mordred | clarkb: k. patchset 2 has a "please don't remove wihtout talking to infra" note | 00:10 |
| *** sarob has quit IRC | 00:12 | |
| clarkb | mordred: you added tabs that people may or may not like (I have no opinions) | 00:12 |
| *** Jeffrey4l_ has joined #openstack-infra | 00:12 | |
| mordred | I did? | 00:12 |
| anteaya | yes | 00:12 |
| mordred | how did I add _tabs_ | 00:12 |
| anteaya | >> | 00:12 |
| anteaya | someone added tabs | 00:12 |
| mordred | fixed | 00:12 |
| mordred | it was me | 00:13 |
| mordred | I think my editor settings only have force-spaces set for python :) | 00:13 |
| * mordred should fix that | 00:13 | |
| clarkb | mordred: let me paste you the magic for that | 00:13 |
| mordred | clarkb: I have some really excellent python-specific indentation rules now | 00:14 |
| clarkb | http://paste.openstack.org/show/562778/ there you go | 00:14 |
| mordred | like, vim now understands that I do not like visual indentation even | 00:14 |
| mordred | thanks! | 00:14 |
| mordred | clarkb: yah - that I have - I think I need to do it for not-python thought | 00:15 |
| clarkb | softtabstop and expandtab I think are the important ones | 00:15 |
| mordred | yah | 00:15 |
| clarkb | oh right | 00:15 |
| clarkb | this was bash | 00:15 |
| mordred | yah. that got it - thanks! | 00:16 |
| openstackgerrit | Merged openstack-infra/system-config: Add root:puppet permissions to hieradata https://review.openstack.org/326649 | 00:16 |
| jeblair | it's funny -- the main argument against tabs is that doing it right requires configuring your editor. :) | 00:17 |
| mordred | :) | 00:17 |
| clarkb | mordred: ps3 caught a bunch of osic nodes too so this should tell us stuff | 00:17 |
| mordred | clarkb: woot! | 00:17 |
| *** ddieterly has joined #openstack-infra | 00:17 | |
| *** docaedo4 has quit IRC | 00:25 | |
| *** david-lyle has quit IRC | 00:26 | |
| *** vhosakot has quit IRC | 00:27 | |
| pabelanger | clarkb: I am seeing a fair bit of SSHTimeoutExceptions for rax-iad subnodes: http://paste.openstack.org/show/562779/ | 00:28 |
| *** tpsilva has quit IRC | 00:28 | |
| clarkb | pabelanger: huh, I wonder if that timeout value aggregates for both hosts and we run over in that case | 00:28 |
| pabelanger | ya, I can check that now | 00:29 |
| clarkb | pabelanger: since I believe it does try to ssh to them serially | 00:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Ensure per-resource caches work without global cache https://review.openstack.org/358776 | 00:30 |
| mordred | clarkb: I'm having un-fun getting this stuff landed. otoh - the nodepool-shade job has caught two legit issues today - so yay for testing! | 00:30 |
| *** docaedo4 has joined #openstack-infra | 00:31 | |
| Shrews | mordred: back now. what are the things? | 00:31 |
| clarkb | mordred: I think we may have lost the host running your ps3 :/ but you might also need to flip the order of your accept_ra and forwading settings | 00:32 |
| clarkb | mordred: possible that maybe things break immediately when you set forwarding to 1 then we either don't continue and fix it or it does but we don't reconnect? | 00:32 |
| mordred | clarkb: nod | 00:33 |
| mordred | clarkb: updated | 00:33 |
| mordred | Shrews: all sorts of fun | 00:33 |
| clarkb | in any case I think this is headed down the right track, so hopefully we make progress | 00:33 |
| mordred | Shrews: battling a few gate jobs - but although some of dealing with them have been a struggle, they've also caught some real errors, so it's worthwhile struggle | 00:33 |
| *** sdake has quit IRC | 00:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Install nodepool and shade into a virtualenv https://review.openstack.org/359425 | 00:36 |
| *** sdake has joined #openstack-infra | 00:37 | |
| mordred | Shrews: like that^^ | 00:37 |
| mordred | but I think this one is a winner | 00:37 |
| Shrews | i +2'd the shade things you need | 00:38 |
| mordred | thanks! | 00:38 |
| mordred | Shrews: also https://review.openstack.org/#/c/357517/ | 00:38 |
| mordred | Shrews: I mean, assuming it passes tests | 00:38 |
| *** claudiub has quit IRC | 00:38 | |
| *** caowei has quit IRC | 00:41 | |
| *** docaedo4 has quit IRC | 00:41 | |
| *** gildub has joined #openstack-infra | 00:42 | |
| pabelanger | clarkb: it actually looks like each (sub)node gets the full timeout value. For some reason, rax-ord 120 doesn't appear to be long enough. We could double it to 240 and see how that performs for tomorrow | 00:43 |
| clarkb | pabelanger: huh ok | 00:43 |
| *** pvaneck has quit IRC | 00:46 | |
| *** Julien-zte has joined #openstack-infra | 00:50 | |
| pabelanger | clarkb: http://paste.openstack.org/show/562780/ of the failure to launch subnode | 00:52 |
| clarkb | mordred: arg it died again | 00:52 |
| *** esikachev has joined #openstack-infra | 00:52 | |
| clarkb | mordred: I think what we want to do is ssh in, tail the devstacklog locally then whatever was last printed is likely to be either the thing that breaks us or the thing just prior | 00:52 |
| clarkb | I need to go family time now so won't do that myself but I think doing ^ should et us really close | 00:53 |
| *** kaisers_ has joined #openstack-infra | 00:53 | |
| *** kzaitsev_mb has quit IRC | 00:53 | |
| * clarkb finds dinner | 00:56 | |
| *** esikachev has quit IRC | 00:57 | |
| *** kaisers_ has quit IRC | 00:58 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Provide unit test coverage for process_results https://review.openstack.org/355385 | 01:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Double rax-ord boot-timeout value https://review.openstack.org/359496 | 01:03 |
| *** raunak has quit IRC | 01:03 | |
| *** amotoki has joined #openstack-infra | 01:06 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: launch-node.py: save key when failing early https://review.openstack.org/359417 | 01:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: launch-node.py: set ansible log path https://review.openstack.org/359499 | 01:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: launch-node.py: More verbose logging https://review.openstack.org/359500 | 01:07 |
| ianw | that's one nicely shaved yak i think | 01:07 |
| *** docaedo4 has joined #openstack-infra | 01:08 | |
| ianw | pabelanger: puppet's changed that permission ... trying again | 01:08 |
| pabelanger | clarkb: ianw: ^ 359496 doubles our boot-timeout in rax-ord | 01:09 |
| pabelanger | ianw: ya, did some yak shaving today too | 01:09 |
| *** amotoki has quit IRC | 01:10 | |
| *** tonytan_brb has joined #openstack-infra | 01:11 | |
| *** mtanin___ has quit IRC | 01:12 | |
| *** Sukhdev has quit IRC | 01:12 | |
| *** zhurong has joined #openstack-infra | 01:13 | |
| *** esikachev has joined #openstack-infra | 01:13 | |
| *** tonytan4ever has quit IRC | 01:14 | |
| *** shashank_hegde has quit IRC | 01:15 | |
| prometheanfire | so, how's gate? | 01:16 |
| * prometheanfire stops pouring gas on the fire | 01:16 | |
| *** pt_15 has quit IRC | 01:16 | |
| *** yanyanhu has joined #openstack-infra | 01:19 | |
| *** tqtran has quit IRC | 01:19 | |
| *** esikachev has quit IRC | 01:19 | |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: update base_email_ext to adapt Email-ext plugin https://review.openstack.org/355139 | 01:20 |
| *** chlong has joined #openstack-infra | 01:20 | |
| *** itisha has quit IRC | 01:20 | |
| *** gyee has quit IRC | 01:22 | |
| *** baoli has joined #openstack-infra | 01:23 | |
| *** baoli_ has joined #openstack-infra | 01:24 | |
| anteaya | I keep seeing that as chaching Intel | 01:28 |
| *** baoli has quit IRC | 01:28 | |
| *** fguillot has quit IRC | 01:30 | |
| *** fguillot has joined #openstack-infra | 01:31 | |
| ianw | Aug 24 01:23:44 review-dev puppet-user[21995]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]) Could not find command '/usr/local/bin/manage-projects' <- i guess we are missing jeepyb dependency? | 01:33 |
| *** nwkarsten has joined #openstack-infra | 01:33 | |
| *** Apoorva_ has joined #openstack-infra | 01:33 | |
| *** Apoorva has quit IRC | 01:37 | |
| *** nwkarsten has quit IRC | 01:37 | |
| *** Apoorva_ has quit IRC | 01:37 | |
| ianw | no ... just another yak ... missing dependencies looks like | 01:39 |
| *** Julien-zte has quit IRC | 01:39 | |
| *** salv-orlando has joined #openstack-infra | 01:40 | |
| *** tonytan_brb has quit IRC | 01:41 | |
| cloudnull | evenings | 01:48 |
| cloudnull | clarkb: idk if you're still about, | 01:48 |
| * cloudnull reading scrollback | 01:48 | |
| clarkb | cloudnull: I can tell you its ok you don't need to debug cloud | 01:48 |
| clarkb | cloudnull: so you can mostly ignore the scrollback | 01:48 |
| clarkb | mordred: pabelanger sc68cal http://paste.openstack.org/show/562798/ is where we seem to be breaking with ipv6 | 01:48 |
| clarkb | cloudnull: ^ basically neutron + devstack creating an ipv6 subnet on osic causes things to bork | 01:49 |
| clarkb | we also need to set accept_ra to 2 because neutron wants to set forwarding | 01:49 |
| clarkb | mordred: I think we still want your change because its more correct but we also need to figure out why we break when creating that subnet | 01:49 |
| pabelanger | looking | 01:49 |
| *** kzaitsev_mb has joined #openstack-infra | 01:49 | |
| cloudnull | clarkb: is it something we're doing that's causing devstack to die in a fire ? | 01:50 |
| clarkb | IMO we should hand this off to the neutron team... This is reliably reproduceable and should be something that works | 01:50 |
| clarkb | cloudnull: no I think neutron just doesn't support being nested with ipv6 only because it borks routing (at least when devstack runs it) | 01:50 |
| clarkb | cloudnull: basically this should work, it doesn't. Its up to the project teams to figure out why I Think | 01:50 |
| cloudnull | ok | 01:51 |
| *** salv-orlando has quit IRC | 01:51 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/puppet-openstack_infra_spec_helper: Revert "Pin puppetlabs-spec-helper" https://review.openstack.org/359539 | 01:51 |
| pabelanger | clarkb: Agreed, I would say this is a test failure and neutron team should dive more into it | 01:51 |
| clarkb | cloudnull: what we found was that the instances we available from within osic both over private ipv4 and via global ipv6 | 01:51 |
| *** thorst_ has joined #openstack-infra | 01:52 | |
| clarkb | cloudnull: but are not accessible over global ipv6 outside of osic. route -6n shows we have no working global route when we get into that state but the route for the local nets are still there which is why it worsk from osic | 01:52 |
| clarkb | cloudnull: at first I thought it may ahve been something in osic but once I saw our routes were going wonky like that I was pretty sure it was on our end | 01:52 |
| pabelanger | clarkb: did the patch mordred propose work? | 01:53 |
| pabelanger | I haven't checked | 01:53 |
| clarkb | pabelanger: no, I think we need it though | 01:53 |
| pabelanger | ack | 01:53 |
| clarkb | pabelanger: basically I think there are layers of broken here and mordreds patch fixes layer one | 01:53 |
| pabelanger | right | 01:53 |
| *** yamahata has quit IRC | 01:57 | |
| *** raunak has joined #openstack-infra | 01:57 | |
| *** dkehn_ has quit IRC | 01:57 | |
| *** dkehn has quit IRC | 01:57 | |
| cloudnull | ok so finally done reading... :) | 02:00 |
| cloudnull | clarkb: so this is an issue w/ devstack creating a v6 network w/in osic nodes. has this always been a problem in the devstack gate or is this new? | 02:00 |
| * cloudnull have not used devstack in FOREVER... | 02:01 | |
| clarkb | cloudnull: I think it has always been an issue we just never noticed because we have primarily been v4 | 02:01 |
| cloudnull | ok. | 02:01 |
| clarkb | cloudnull: zuul/jenkins have almost always used v4 for us until now | 02:01 |
| clarkb | cloudnull: so the problem is in devstack/neutron because this *should* work | 02:01 |
| *** caowei has joined #openstack-infra | 02:02 | |
| cloudnull | ok | 02:02 |
| cloudnull | is this a gate blocker for them? | 02:03 |
| clarkb | cloudnull: sort of? basically zuul will reschedule the job over and over until it gets on not osic | 02:03 |
| clarkb | so things aren't broken broken just slow | 02:03 |
| clarkb | I really think we should use that as the carrot for getting it fixed though >_> | 02:03 |
| cloudnull | sounds reasonable to me | 02:04 |
| jeblair | we have been too lazybusy to put a cap on the number of re-launches zuul will perform :) | 02:04 |
| *** tonytan4ever has joined #openstack-infra | 02:04 | |
| clarkb | because really you should be able to run neutron on an instance that relies on ipv6 networking | 02:05 |
| jeblair | join us in the future | 02:05 |
| jeblair | by future, i mean 1994. | 02:05 |
| cloudnull | ha! | 02:05 |
| *** vhosakot has joined #openstack-infra | 02:06 | |
| *** amotoki has joined #openstack-infra | 02:06 | |
| *** dkehn_ has joined #openstack-infra | 02:10 | |
| *** kzaitsev_mb has quit IRC | 02:11 | |
| *** amotoki has quit IRC | 02:11 | |
| *** dkehn has joined #openstack-infra | 02:11 | |
| *** sdague has joined #openstack-infra | 02:12 | |
| *** ddieterly has quit IRC | 02:15 | |
| *** esikachev has joined #openstack-infra | 02:16 | |
| clarkb | also I think ipv6 by default in devsatck runs is relatively new so this may just be perfect storm of new things not yet tested | 02:19 |
| *** zz_dimtruck is now known as dimtruck | 02:19 | |
| *** fguillot has quit IRC | 02:19 | |
| dougwig | do you have a bug number yet? | 02:20 |
| *** esikachev has quit IRC | 02:20 | |
| clarkb | no I am juggling beer and dinner | 02:20 |
| clarkb | the paste is above where it dies | 02:20 |
| clarkb | if someone else wants to file it otherwise I can file one in the morning | 02:20 |
| dougwig | clarkb: looking, though i am fairly ipv6 illiterate. i've rattled cages, but folks seem to be in bed. | 02:21 |
| clarkb | dougwig: basically as soon as devstack makes an ipv6 subnet we lose connectivity | 02:21 |
| clarkb | dougwig: and we appear to lose connectivity because the ipv6 default route goes away | 02:21 |
| dougwig | clarkb: can i call it a feature? | 02:21 |
| dougwig | :) | 02:22 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/yaml2ical: Remove discover from test-requirements https://review.openstack.org/345809 | 02:23 |
| dougwig | clarkb: confused, the end of that paste is creating the external v4 subnet, not v6 | 02:23 |
| *** gouthamr_ has joined #openstack-infra | 02:25 | |
| *** spzala has joined #openstack-infra | 02:25 | |
| *** raunak has quit IRC | 02:26 | |
| clarkb | hrm maybe it cut off short | 02:27 |
| *** raunak has joined #openstack-infra | 02:27 | |
| clarkb | ya i think paste failed me | 02:27 |
| clarkb | I will repaste ina bit | 02:27 |
| dougwig | clarkb: ok, thanks. | 02:28 |
| *** gouthamr has quit IRC | 02:28 | |
| cloudnull | mordred sc68cal pabelanger clarkb -- should we also set "net.ipv6.conf.default.accept_ra" in https://review.openstack.org/#/c/359490/4/lib/neutron_plugins/services/l3 ? | 02:30 |
| *** sdague has quit IRC | 02:33 | |
| dougwig | clarkb: bug 1616282 | 02:34 |
| openstack | bug 1616282 in neutron "creating ipv6 subnet on ipv6 vm will cause loss of connectivity" [Critical,Confirmed] https://launchpad.net/bugs/1616282 | 02:34 |
| clarkb | dougwig: http://paste.openstack.org/show/562831/ | 02:34 |
| clarkb | cloudnull: maybe? | 02:35 |
| clarkb | dougwig: I trimmed context so I could get the whole thing, hopefulyl enough there though | 02:35 |
| *** kzaitsev_mb has joined #openstack-infra | 02:37 | |
| *** caowei has quit IRC | 02:40 | |
| *** caowei has joined #openstack-infra | 02:41 | |
| *** kaisers_ has joined #openstack-infra | 02:42 | |
| *** kaisers_ has quit IRC | 02:46 | |
| *** sdake has quit IRC | 02:46 | |
| *** sdake has joined #openstack-infra | 02:49 | |
| *** vhosakot has quit IRC | 02:50 | |
| *** salv-orlando has joined #openstack-infra | 02:50 | |
| *** yuanying_ has joined #openstack-infra | 02:51 | |
| *** yuanying has quit IRC | 02:52 | |
| *** yuanying has joined #openstack-infra | 02:53 | |
| *** kzaitsev_mb has quit IRC | 02:53 | |
| cloudnull | we're seeing a package installation problems in a few of our gates. http://logs.openstack.org/25/359225/7/check/gate-openstack-ansible-os_watcher-ansible-func-ubuntu-xenial/3518959/console.html#_2016-08-24_02_48_16_344532 | 02:53 |
| *** salv-orlando has quit IRC | 02:53 | |
| cloudnull | its using the osic repo servers at "http://mirror.regionone.osic-cloud1.openstack.org/ubuntu xenial" | 02:54 |
| cloudnull | have others seen something similar recently? | 02:54 |
| clarkb | cloudnull: that looks like you may need to apt get update? | 02:55 |
| *** yuanying has quit IRC | 02:55 | |
| *** yuanying has joined #openstack-infra | 02:56 | |
| cloudnull | yea. im looking into that now, the ansible module **should** be doing an update, but maybe its not for some reason | 02:56 |
| *** tphummel has quit IRC | 02:56 | |
| *** yuanying_ has quit IRC | 02:57 | |
| *** Jeffrey4l_ has quit IRC | 03:03 | |
| *** gongysh has joined #openstack-infra | 03:03 | |
| *** gouthamr_ is now known as gouthamr | 03:05 | |
| *** amotoki has joined #openstack-infra | 03:07 | |
| *** Jeffrey4l_ has joined #openstack-infra | 03:08 | |
| *** mtanino has joined #openstack-infra | 03:08 | |
| *** amotoki has quit IRC | 03:11 | |
| dougwig | clarkb: what version of openstack is osic running? | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-jeepyb: Ensure development files installed https://review.openstack.org/359562 | 03:14 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas functional job not experimental https://review.openstack.org/359320 | 03:17 |
| ianw | zaro / rcarrillocruz : ^ review-dev doesn't want to deploy due to issues with dependencies here ... but getting there! | 03:17 |
| *** Wei_Liu has joined #openstack-infra | 03:18 | |
| amrith | any infra cores around, could I get a second +2 and an A+1 on https://review.openstack.org/#/c/354881/ please. AJaeger has already put one +2 on it. It adds a couple of non voting jobs to the Trove gate. Thanks! | 03:19 |
| Wei_Liu | hi, I have one question about jjb, does it support docker-plugin in jenkins? | 03:20 |
| clarkb | dougwig: I think cloudnull said liberty | 03:21 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 03:22 |
| *** amotoki has joined #openstack-infra | 03:24 | |
| *** rajinir has quit IRC | 03:25 | |
| *** dimtruck is now known as zz_dimtruck | 03:28 | |
| cloudnull | dougwig clarkb: whats that ? | 03:29 |
| cloudnull | may be missing a message there. | 03:29 |
| dougwig | cloudnull: i asked what version of openstack that the osic cloud was running? | 03:30 |
| cloudnull | ah. it liberty | 03:30 |
| dougwig | which core plugin? ovs? | 03:30 |
| cloudnull | linux bridge | 03:30 |
| dougwig | lb, ok. | 03:30 |
| cloudnull | ML2 + lxb | 03:30 |
| zaro | Wei_Liu: there's something about docker build publish plugin: http://docs.openstack.org/infra/jenkins-job-builder/builders.html?highlight=docker#builders.docker-build-publish | 03:32 |
| zaro | ianw: which dependencies? Gerrit? let me know if i can help | 03:33 |
| *** zz_dimtruck is now known as dimtruck | 03:34 | |
| ianw | zaro: for jeepyb, see log in 359562 ... i'm just bumbling with the puppet now | 03:34 |
| zaro | that seems odd. doesn't review.o.o and review-dev share that? | 03:34 |
| zaro | and i think we've recently built a new review.0.0 to do the same thing (make it bigger). | 03:36 |
| *** gouthamr has quit IRC | 03:36 | |
| *** gouthamr has joined #openstack-infra | 03:37 | |
| *** amotoki has quit IRC | 03:38 | |
| *** thorst_ has quit IRC | 03:38 | |
| dougwig | if you're up, i need one more infra core to peek at this devstack-gate change, so i don't break the world when the lbaas v1 removal happens: https://review.openstack.org/#/c/358258/ | 03:39 |
| dougwig | (trying to get that done at this week's midcycle) | 03:39 |
| *** thorst_ has joined #openstack-infra | 03:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-jeepyb: Ensure development files installed https://review.openstack.org/359562 | 03:39 |
| ianw | zaro: well, it's dependencies ... so if something else just happened to come first it would be satisfied and you wouldn't know | 03:41 |
| *** vikrant has joined #openstack-infra | 03:41 | |
| *** Sukhdev has joined #openstack-infra | 03:42 | |
| *** bhunter71 has quit IRC | 03:42 | |
| *** gouthamr has quit IRC | 03:45 | |
| *** AnarchyAo has joined #openstack-infra | 03:46 | |
| Wei_Liu | zaro, no, I mean docker plugin, which is able to use a docker host to dynamically provision a slave and run build | 03:47 |
| *** senk has joined #openstack-infra | 03:47 | |
| *** thorst_ has quit IRC | 03:47 | |
| *** yuanying has quit IRC | 03:48 | |
| *** spzala has quit IRC | 03:50 | |
| *** bauzas has left #openstack-infra | 03:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:50 | |
| *** aeng has quit IRC | 03:50 | |
| *** tonytan4ever has quit IRC | 03:50 | |
| *** yuanying has joined #openstack-infra | 03:51 | |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Enable zaqar for senlin integration test https://review.openstack.org/354566 | 03:55 |
| *** raunak has quit IRC | 03:57 | |
| *** salv-orlando has joined #openstack-infra | 03:58 | |
| *** gongysh has quit IRC | 03:59 | |
| *** amotoki has joined #openstack-infra | 04:00 | |
| *** baoli_ has quit IRC | 04:01 | |
| *** DmZDsfZoQv has quit IRC | 04:03 | |
| *** dimtruck is now known as zz_dimtruck | 04:05 | |
| *** aeng has joined #openstack-infra | 04:07 | |
| *** raunak has joined #openstack-infra | 04:08 | |
| *** salv-orlando has quit IRC | 04:09 | |
| *** amotoki has quit IRC | 04:09 | |
| *** raunak has quit IRC | 04:09 | |
| *** VmtcXzLmiz has joined #openstack-infra | 04:10 | |
| *** amotoki has joined #openstack-infra | 04:10 | |
| *** VmtcXzLmiz has quit IRC | 04:10 | |
| *** raunak has joined #openstack-infra | 04:13 | |
| *** roxanagh_ has joined #openstack-infra | 04:13 | |
| *** jraju has joined #openstack-infra | 04:14 | |
| *** jraju has quit IRC | 04:15 | |
| *** shashank_hegde has joined #openstack-infra | 04:18 | |
| *** gyx has joined #openstack-infra | 04:18 | |
| *** esikachev has joined #openstack-infra | 04:18 | |
| *** roxanagh_ has quit IRC | 04:19 | |
| *** esikachev has quit IRC | 04:23 | |
| *** kdas__ has joined #openstack-infra | 04:24 | |
| *** mdrabe has quit IRC | 04:28 | |
| *** kaisers_ has joined #openstack-infra | 04:31 | |
| *** amotoki has quit IRC | 04:31 | |
| *** kdas__ is now known as kushal | 04:31 | |
| *** kushal has joined #openstack-infra | 04:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep: Give some examples https://review.openstack.org/358811 | 04:33 |
| *** kaisers_ has quit IRC | 04:35 | |
| *** amotoki has joined #openstack-infra | 04:37 | |
| *** senk has quit IRC | 04:38 | |
| *** bhunter71 has joined #openstack-infra | 04:38 | |
| *** esikachev has joined #openstack-infra | 04:39 | |
| *** yamamoto has quit IRC | 04:39 | |
| *** yamamoto has joined #openstack-infra | 04:39 | |
| *** raunak has quit IRC | 04:40 | |
| *** yamamoto has quit IRC | 04:41 | |
| *** mtanino has quit IRC | 04:42 | |
| *** raunak has joined #openstack-infra | 04:42 | |
| *** yamamoto has joined #openstack-infra | 04:42 | |
| *** yamamoto has quit IRC | 04:42 | |
| *** esikachev has quit IRC | 04:45 | |
| *** thorst_ has joined #openstack-infra | 04:45 | |
| *** asettle has joined #openstack-infra | 04:51 | |
| *** tonytan4ever has joined #openstack-infra | 04:51 | |
| *** thorst_ has quit IRC | 04:52 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config: Switch all heat jobs to use devstack plugin https://review.openstack.org/317817 | 04:53 |
| *** salv-orlando has joined #openstack-infra | 04:54 | |
| *** tonytan4ever has quit IRC | 04:56 | |
| *** asettle has quit IRC | 04:56 | |
| *** Daisy has joined #openstack-infra | 05:02 | |
| *** Daisy has quit IRC | 05:02 | |
| *** andymaier_ has joined #openstack-infra | 05:05 | |
| *** AnarchyAo has quit IRC | 05:05 | |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Require packages before installing jeepyb https://review.openstack.org/327707 | 05:07 |
| *** jaosorior has joined #openstack-infra | 05:07 | |
| *** jaosorior has quit IRC | 05:09 | |
| *** jaosorior has joined #openstack-infra | 05:10 | |
| *** kzaitsev_mb has quit IRC | 05:10 | |
| *** psachin has joined #openstack-infra | 05:17 | |
| *** javeriak has joined #openstack-infra | 05:18 | |
| *** senk has joined #openstack-infra | 05:27 | |
| *** Sukhdev has quit IRC | 05:27 | |
| *** rkukura_ has joined #openstack-infra | 05:27 | |
| *** rkukura has quit IRC | 05:29 | |
| *** rkukura_ is now known as rkukura | 05:29 | |
| *** javeriak has quit IRC | 05:32 | |
| *** raunak has quit IRC | 05:38 | |
| *** andymaier_ has quit IRC | 05:39 | |
| *** roxanagh_ has joined #openstack-infra | 05:39 | |
| *** harlowja_at_home has joined #openstack-infra | 05:39 | |
| *** yamamoto has joined #openstack-infra | 05:39 | |
| *** roxanagh_ has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Move other-requirements.txt to bindep.txt https://review.openstack.org/354861 | 05:40 |
| *** esikachev has joined #openstack-infra | 05:41 | |
| *** armax has quit IRC | 05:41 | |
| *** esikachev has quit IRC | 05:46 | |
| *** ilyashakhat has joined #openstack-infra | 05:47 | |
| ianw | zaro: ok, so i have to wait for https://review.openstack.org/327707 to get pushed to puppetmaster because new hosts get their /etc/puppet/modules copied from there by ansible. once that's done, i should be able to start the new review-dev, detach the cinder volume with the old one's ~gerrit2 and re-attach it, and update dns. so hopefully finish this tomorrow my time | 05:48 |
| *** coolsvap has joined #openstack-infra | 05:49 | |
| ianw | fungi: you might like to review the patches at https://review.openstack.org/#/q/status:open+project:openstack-infra/system-config+branch:master+topic:launch-node before trying to launch your wiki nodes. at least the interface restart is required, the logging is nice to have & has helped me debug | 05:50 |
| *** thorst_ has joined #openstack-infra | 05:50 | |
| *** rbuzatu has quit IRC | 05:51 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 05:52 |
| *** adriant has quit IRC | 05:52 | |
| *** tonytan4ever has joined #openstack-infra | 05:52 | |
| *** tonytan4ever has quit IRC | 05:57 | |
| *** thorst_ has quit IRC | 05:58 | |
| *** javeriak has joined #openstack-infra | 05:59 | |
| *** oanson has joined #openstack-infra | 06:00 | |
| *** sandanar has joined #openstack-infra | 06:02 | |
| *** esikachev has joined #openstack-infra | 06:02 | |
| *** _nadya_ has joined #openstack-infra | 06:03 | |
| *** tqtran has joined #openstack-infra | 06:03 | |
| *** mikelk has joined #openstack-infra | 06:03 | |
| *** _nadya_ has quit IRC | 06:04 | |
| *** esikachev has quit IRC | 06:06 | |
| *** ilyashakhat has quit IRC | 06:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:07 | |
| AJaeger | ianw: could you review pabelanger's change https://review.openstack.org/359496 , please? | 06:12 |
| *** r-mibu has quit IRC | 06:12 | |
| *** _nadya_ has joined #openstack-infra | 06:12 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Set the ssl_key_file_contents to mandatory https://review.openstack.org/359294 | 06:13 |
| *** brad_behle_ has joined #openstack-infra | 06:19 | |
| *** kaisers_ has joined #openstack-infra | 06:19 | |
| *** AnarchyAo has joined #openstack-infra | 06:22 | |
| *** rcernin has joined #openstack-infra | 06:23 | |
| *** brad_behle has quit IRC | 06:23 | |
| *** kaisers_ has quit IRC | 06:24 | |
| *** aeng has quit IRC | 06:28 | |
| *** kzaitsev_mb has quit IRC | 06:30 | |
| *** ccamacho has joined #openstack-infra | 06:32 | |
| *** r-mibu has joined #openstack-infra | 06:32 | |
| *** ccamacho has quit IRC | 06:32 | |
| *** javeriak has quit IRC | 06:33 | |
| yolanda | good morning | 06:34 |
| *** ccamacho has joined #openstack-infra | 06:36 | |
| *** senk has quit IRC | 06:36 | |
| *** kushal has quit IRC | 06:37 | |
| *** kushal has joined #openstack-infra | 06:38 | |
| *** indistylo has joined #openstack-infra | 06:38 | |
| *** andreas_s has joined #openstack-infra | 06:38 | |
| *** aeng has joined #openstack-infra | 06:42 | |
| *** harlowja_at_home has quit IRC | 06:44 | |
| AJaeger | good morning, yolanda ! | 06:44 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 06:47 |
| *** kaisers_ has joined #openstack-infra | 06:47 | |
| *** DrifterZA has joined #openstack-infra | 06:50 | |
| *** _nadya_ has quit IRC | 06:52 | |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config: Rename Smaug to Karbor https://review.openstack.org/353304 | 06:52 |
| *** yuval has joined #openstack-infra | 06:55 | |
| *** thorst_ has joined #openstack-infra | 06:56 | |
| *** tesseract- has joined #openstack-infra | 06:56 | |
| *** javeriak has joined #openstack-infra | 06:57 | |
| *** AnarchyAo has quit IRC | 06:58 | |
| *** _nadya_ has joined #openstack-infra | 07:02 | |
| *** thorst_ has quit IRC | 07:03 | |
| *** esikachev has joined #openstack-infra | 07:03 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: WIP: Don't ignore data-files https://review.openstack.org/345210 | 07:07 |
| *** esikachev has quit IRC | 07:07 | |
| *** coolsvap is now known as coolsvap_ | 07:09 | |
| *** flepied has joined #openstack-infra | 07:11 | |
| *** florianf has joined #openstack-infra | 07:11 | |
| *** indistylo has quit IRC | 07:12 | |
| *** javeriak has quit IRC | 07:12 | |
| *** sputnik13 has quit IRC | 07:15 | |
| *** pgadiya has joined #openstack-infra | 07:15 | |
| *** javeriak has joined #openstack-infra | 07:17 | |
| *** javeriak has joined #openstack-infra | 07:17 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 07:18 |
| *** jordanP has joined #openstack-infra | 07:19 | |
| *** jordanP has quit IRC | 07:20 | |
| *** claudiub has joined #openstack-infra | 07:24 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:27 | |
| *** tphummel has joined #openstack-infra | 07:28 | |
| _nadya_ | dear openstack-infra team! Could you please add nprivalova@mirantis.com to the group https://review.openstack.org/#/admin/groups/1535,members ? It is a new project created by https://review.openstack.org/#/c/355406/ . Thanks in advance! | 07:29 |
| *** jpich has joined #openstack-infra | 07:32 | |
| yuval | AJaeger: hey :) | 07:33 |
| yuval | AJaeger: regarding https://review.openstack.org/#/c/353304/ and smaug => karbor. Is there a preference in renaming pypi packages or creating new ones? | 07:34 |
| *** ganesan has joined #openstack-infra | 07:37 | |
| AJaeger | yuval: no idea ;( | 07:38 |
| AJaeger | yuval: I think that depends on how many already use your packages. | 07:38 |
| *** vincentll has joined #openstack-infra | 07:39 | |
| *** e0ne has joined #openstack-infra | 07:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Double rax-ord boot-timeout value https://review.openstack.org/359496 | 07:40 |
| *** andymaier_ has joined #openstack-infra | 07:44 | |
| *** mikelk has quit IRC | 07:46 | |
| *** ifarkas_afk is now known as ifarkas | 07:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Removed directory changes in npm-dsvm-macro https://review.openstack.org/359428 | 07:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Update api-site zuul config https://review.openstack.org/359656 | 07:49 |
| *** yaume has joined #openstack-infra | 07:50 | |
| *** bethwhite_ has joined #openstack-infra | 07:50 | |
| *** abregman has joined #openstack-infra | 07:51 | |
| *** martinkopec has joined #openstack-infra | 07:51 | |
| ganesan | I am getting the error on nodepool.log "JenkinsException: Error in request. Possibly authentication failed [500]: Server Error" | 07:52 |
| *** e0ne has quit IRC | 07:53 | |
| ganesan | anyone help me to find the problem with my CI setup | 07:53 |
| yuval | Would appreciate if any infra core could help merge https://review.openstack.org/#/c/359019/ (Karbor Fullstack Path Fix) | 07:53 |
| *** tonytan4ever has joined #openstack-infra | 07:54 | |
| ganesan | nodes are created by nodepool and it is able to ssh into that | 07:54 |
| yuval | thanks ianw :) | 07:55 |
| *** matthewbodkin has joined #openstack-infra | 07:55 | |
| *** tonytan4ever has quit IRC | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** thorst_ has joined #openstack-infra | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** amotoki has quit IRC | 08:01 | |
| *** yaume_ has joined #openstack-infra | 08:02 | |
| *** kushal has quit IRC | 08:02 | |
| *** abregman has quit IRC | 08:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Karbor (Smaug) Fullstack Path Fix https://review.openstack.org/359019 | 08:03 |
| *** esikachev has joined #openstack-infra | 08:04 | |
| *** matrohon has joined #openstack-infra | 08:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Add python34-jobs and python35-jobs to Almanach https://review.openstack.org/359345 | 08:05 |
| *** yaume has quit IRC | 08:05 | |
| *** matrohon has quit IRC | 08:06 | |
| *** derekh has joined #openstack-infra | 08:08 | |
| *** esikachev has quit IRC | 08:08 | |
| *** thorst_ has quit IRC | 08:09 | |
| *** matrohon has joined #openstack-infra | 08:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Use aliasByNode for Node Launches panel https://review.openstack.org/359299 | 08:11 |
| openstackgerrit | Merged openstack-infra/project-config: Remove stale octavia job that never gets run, and is broken https://review.openstack.org/359367 | 08:11 |
| *** lucas-afk is now known as lucasagomes | 08:12 | |
| ianw | yolanda : you know anything about having to run "$ java -jar ./bin/gerrit.war reindex" as gerrit2 before "/etc/init.d/gerrit start" works? because it seems our current puppet doesn't so the startup of gerrit fails | 08:12 |
| yolanda | ianw, i remember doing that manually yes. But not on a clean install, i've done it on migrations | 08:13 |
| *** abregman has joined #openstack-infra | 08:13 | |
| ianw | yolanda: yeah, i'm wondering if https://review.openstack.org/#/c/355194/2/manifests/init.pp actually broke it for clean installs? | 08:13 |
| *** abregman has quit IRC | 08:14 | |
| yolanda | ianw, so the parameter value has been changed? i see defaults for secondary-index to false as well, the behaviour shall be the same? | 08:15 |
| ianw | yolanda: do we set secondary_index though? if we are, then we've stopped doing the index first i think? | 08:16 |
| yolanda | so... looks like | 08:16 |
| yolanda | i see system-config setting secondary_index to true | 08:16 |
| ianw | yeah modules/openstack_project/manifests/gerrit.pp | 08:16 |
| ianw | so basically i guess we're not doing it any more? | 08:16 |
| yolanda | yes, per that change, we have stopped doing that reindex | 08:17 |
| ianw | and yeah, on a new install, gerrit refuses to start saying you haven't indexed or something | 08:17 |
| yolanda | can you try a clean install with that offline_reindex set to true? | 08:17 |
| openstackgerrit | Merged openstack-infra/project-config: Add new project "os-failures" https://review.openstack.org/355819 | 08:18 |
| ianw | yolanda: umm, not really ... this is from launch-node.py trying to create a new review-dev host | 08:18 |
| ianw | well maybe i can, but too hard for right now :) | 08:18 |
| yolanda | i don't have the background, but why has secondary_index parameter disappeared in favour of that offline reindex? | 08:18 |
| *** dizquierdo has joined #openstack-infra | 08:19 | |
| ianw | yolanda: for posterity, the error is http://paste.openstack.org/show/562898/ | 08:19 |
| *** woodster_ has quit IRC | 08:19 | |
| *** abregman has joined #openstack-infra | 08:19 | |
| yolanda | so yes, that change looks as the problem | 08:20 |
| yolanda | do you want to revert and retry? | 08:20 |
| ianw | yolanda: i'm about done for the day. i might just put a change in, and we can let zaro have a look | 08:21 |
| ianw | this server is for him anyway, so no rush :) | 08:21 |
| yolanda | ok sounds good | 08:21 |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config: Rename Smaug to Karbor https://review.openstack.org/353304 | 08:21 |
| ianw | bringing this up has been a bit of a nightmare :( bitrot sets in on this stuff when it's not used that frequently | 08:21 |
| openstackgerrit | Henry Gessau proposed openstack-infra/project-config: Fix neutron failure rates dashboard integrated jobs list https://review.openstack.org/358462 | 08:21 |
| *** tphummel has quit IRC | 08:22 | |
| *** jordanP has joined #openstack-infra | 08:22 | |
| *** Na3iL has joined #openstack-infra | 08:23 | |
| *** esikachev has joined #openstack-infra | 08:25 | |
| *** hashar has joined #openstack-infra | 08:25 | |
| *** yamahata has joined #openstack-infra | 08:27 | |
| *** asettle has joined #openstack-infra | 08:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-gerrit: Add secondary_index check https://review.openstack.org/359683 | 08:28 |
| ianw | zaro: ^ if you can look at that, it will help getting review-dev working :) | 08:28 |
| *** abregman has quit IRC | 08:29 | |
| *** abregman has joined #openstack-infra | 08:29 | |
| *** pcaruana has joined #openstack-infra | 08:29 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/359687 | 08:30 |
| *** _nadya_ has quit IRC | 08:31 | |
| *** esikachev has quit IRC | 08:31 | |
| *** ganesan has quit IRC | 08:32 | |
| *** jed56 has joined #openstack-infra | 08:33 | |
| *** tqtran has quit IRC | 08:34 | |
| *** ilyashakhat has joined #openstack-infra | 08:37 | |
| *** flepied has quit IRC | 08:37 | |
| *** ilyashakhat has quit IRC | 08:44 | |
| *** electrofelix has joined #openstack-infra | 08:45 | |
| *** kzaitsev_mb has quit IRC | 08:49 | |
| *** yamahata has quit IRC | 08:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:50 | |
| *** dingyichen has quit IRC | 08:53 | |
| *** tonytan4ever has joined #openstack-infra | 08:55 | |
| vrovachev | Hi around, please take a look https://review.openstack.org/#/c/359704/1 | 08:56 |
| *** tonytan4ever has quit IRC | 08:59 | |
| *** kzaitsev_mb has quit IRC | 09:01 | |
| *** asettle has quit IRC | 09:01 | |
| AJaeger | vrovachev: why is that needed at all? | 09:02 |
| AJaeger | vrovachev: your refs/heads/ rule should be just fine. | 09:02 |
| *** senk has joined #openstack-infra | 09:05 | |
| *** mikelk has joined #openstack-infra | 09:05 | |
| *** asettle has joined #openstack-infra | 09:06 | |
| *** thorst_ has joined #openstack-infra | 09:06 | |
| *** esikachev has joined #openstack-infra | 09:08 | |
| *** asettle has quit IRC | 09:08 | |
| *** amotoki has joined #openstack-infra | 09:09 | |
| *** abregman has quit IRC | 09:12 | |
| *** flepied has joined #openstack-infra | 09:12 | |
| *** ggnel_t has joined #openstack-infra | 09:12 | |
| *** thorst_ has quit IRC | 09:13 | |
| *** esikachev has quit IRC | 09:15 | |
| *** Hal has quit IRC | 09:15 | |
| *** Hal has joined #openstack-infra | 09:15 | |
| *** yolanda has quit IRC | 09:16 | |
| *** amotoki has quit IRC | 09:17 | |
| *** sambetts|afk is now known as sambetts | 09:18 | |
| *** shashank_hegde has quit IRC | 09:19 | |
| *** yolanda has joined #openstack-infra | 09:19 | |
| *** javeriak has quit IRC | 09:21 | |
| *** jaosorior is now known as jaosorior_lunch | 09:21 | |
| *** kushal has joined #openstack-infra | 09:26 | |
| vrovachev | Jaeger: It is need for right workflow according to product development by different teams and for right CI works | 09:26 |
| *** abregman has joined #openstack-infra | 09:26 | |
| *** amotoki has joined #openstack-infra | 09:26 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:27 | |
| *** _nadya_ has joined #openstack-infra | 09:27 | |
| *** ihrachys has joined #openstack-infra | 09:28 | |
| *** yolanda has quit IRC | 09:28 | |
| *** eset has joined #openstack-infra | 09:29 | |
| *** yolanda has joined #openstack-infra | 09:29 | |
| *** eset has quit IRC | 09:29 | |
| *** eset has joined #openstack-infra | 09:29 | |
| *** _nadya_ has quit IRC | 09:30 | |
| *** _nadya_ has joined #openstack-infra | 09:30 | |
| *** vinaypotluri has quit IRC | 09:31 | |
| *** _degorenko|afk is now known as degorenko | 09:32 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Create /etc/nova/ssl folder on compute https://review.openstack.org/359737 | 09:33 |
| *** jlibosva has joined #openstack-infra | 09:35 | |
| zigo | AJaeger: Hi there! Can you +2 adding deb-python-fixtures ? https://review.openstack.org/#/c/358819/ | 09:36 |
| zigo | AJaeger: Or should I add the Needed-By: thing? | 09:36 |
| AJaeger | zigo: I wait for PTL +1 before I +2 any new repos - see the comment I left in there | 09:36 |
| zigo | AJaeger: PTL of what? | 09:37 |
| zigo | packaging-deb ? | 09:37 |
| AJaeger | yes, sure | 09:37 |
| zigo | That's mordred, but he's not really an active PTL ... | 09:37 |
| zigo | I'll try to ping him when he wakes up. | 09:38 |
| *** salv-orlando has quit IRC | 09:40 | |
| jlibosva | hi, how are we gonna support Newton Open Stack in Ubuntu. Will be 16.04 the minimal required version or it's gonna be available in older versions too. Is there any document with the plan? | 09:40 |
| *** salv-orlando has joined #openstack-infra | 09:41 | |
| *** berendt has joined #openstack-infra | 09:42 | |
| *** dizquierdo has quit IRC | 09:42 | |
| jlibosva | s/Open Stack/OpenStack/ :) | 09:42 |
| AJaeger | yolanda: could you check https://github.com/openstack/os-failures , please? - this is created empty but the imported content is not synced over. Or should we wait a bit longer? | 09:43 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 09:43 |
| AJaeger | jlibosva: better ask the Ubuntu team that packages it. jamespage might give you a pointer | 09:44 |
| jamespage | jlibosva, 16.04 will be the minimum required ubuntu version | 09:44 |
| jamespage | no backport to 14.04 | 09:45 |
| jlibosva | jamespage: AJaeger great, thanks a lot | 09:45 |
| jamespage | jlibosva, https://wiki.ubuntu.com/OpenStack/CloudArchive for reference | 09:45 |
| jlibosva | so this https://wiki.ubuntu.com/OpenStack/CloudArchive?action=AttachFile&do=get&target=plan.png shows the plan also for basic ubuntu, not just cloud archive | 09:45 |
| jamespage | yeah | 09:45 |
| jlibosva | cool, thanks | 09:46 |
| jamespage | jlibosva, newton will be in Ubuntu 16.10, and in the UCA for 16.04 | 09:46 |
| jamespage | UCA = Ubuntu Cloud Archive | 09:46 |
| jlibosva | AJaeger: jamespage so is there a document/plan to switch all gate jobs to Xenial and when is this gonna happen? | 09:46 |
| jamespage | that I don't know the complete answer to, but I see alot of activity switching things to xenial | 09:47 |
| jlibosva | I found "Running CI jobs on Ubuntu Xenial by default" ML thread, sorry for asking before searching :) | 09:48 |
| AJaeger | jlibosva: we're working on that - help to move jobs over is welcome | 09:48 |
| AJaeger | Note that we will not switch all jobs over - we want master on Xenial, older branches on Trusty. | 09:48 |
| jlibosva | AJaeger: I can help with fullstack and functional tests for Neutron. I don't have that much experience with project-config though | 09:49 |
| jlibosva | AJaeger: if I send a patch to project-config, how can I trigger changed job to see it passed/failed? | 09:49 |
| AJaeger | jlibosva: you cannot ;( That's not really possible | 09:50 |
| jlibosva | eh | 09:50 |
| *** tosky has joined #openstack-infra | 09:50 | |
| jlibosva | AJaeger: so we send a patch and hope for the best or we introduce a xenial nv job in parallel? | 09:50 |
| AJaeger | we test syntax of changes but since those jobs run in public virtual machines as root, we have security concerns for full self-service without review | 09:51 |
| *** yanyanhu has quit IRC | 09:51 | |
| AJaeger | jlibosva: check the archives how we did it for the first jobs. | 09:51 |
| jlibosva | AJaeger: ok | 09:52 |
| *** pilgrimstack1 has quit IRC | 09:52 | |
| AJaeger | jlibosva: example https://review.openstack.org/348078 | 09:52 |
| AJaeger | So, those were "hope for the best" | 09:53 |
| AJaeger | and if it breaks, we would have reverted quickly | 09:53 |
| *** hashar has quit IRC | 09:54 | |
| *** hashar_ has joined #openstack-infra | 09:54 | |
| AJaeger | jlibosva: best talk with clarkb later - he's US based - and coordinate | 09:54 |
| *** tonytan4ever has joined #openstack-infra | 09:56 | |
| *** pilgrimstack has joined #openstack-infra | 09:56 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud https://review.openstack.org/359757 | 09:56 |
| yolanda | hi AJaeger , taking a look | 09:56 |
| *** zhurong has quit IRC | 09:59 | |
| AJaeger | thanks, yolanda ! | 09:59 |
| yolanda | i just fixed, so please take a look in short | 09:59 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - test https://review.openstack.org/359759 | 09:59 |
| *** hashar_ has quit IRC | 10:00 | |
| *** tonytan4ever has quit IRC | 10:00 | |
| *** hashar has joined #openstack-infra | 10:00 | |
| AJaeger | will do, thanks | 10:01 |
| AJaeger | yolanda: already fixed - wow! | 10:02 |
| yolanda | easy mechanical thing :) | 10:02 |
| AJaeger | ;) | 10:04 |
| *** dtantsur has joined #openstack-infra | 10:07 | |
| *** kushal has quit IRC | 10:09 | |
| *** kushal has joined #openstack-infra | 10:10 | |
| *** thorst_ has joined #openstack-infra | 10:11 | |
| *** Na3iL has quit IRC | 10:13 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Re-enable use of cached images https://review.openstack.org/359765 | 10:14 |
| amrith | yolanda, any other infra cores, would someone please approve https://review.openstack.org/#/c/354881/. There's a comment from Ian Weinand, which isn't a -1 but is it something that you want fixed? AJaeger already has a +2 on this, looking for the second. thanks! | 10:14 |
| *** Na3iL has joined #openstack-infra | 10:17 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud https://review.openstack.org/359757 | 10:17 |
| yolanda | amrith, i saw that, so i prefered to wait for the thread with Ian to finish | 10:17 |
| *** asettle has joined #openstack-infra | 10:18 | |
| *** thorst_ has quit IRC | 10:18 | |
| yolanda | amrith, if that's not a blocker, i'd prefer if the question is answered and Ian is ok with that before approving | 10:19 |
| amrith | yolanda, g'morning. I'm not sure that there's going to be anything more on that thread unless someone either says 'yes, we don't want the snowflake' or someone approves the change. i.e. as written the filter seems fine, but I agree, it is unique. | 10:19 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Create /etc/nova/ssl folder on compute https://review.openstack.org/359737 | 10:19 |
| amrith | yolanda, thanks. I will pose the question to Ian in that case. | 10:20 |
| yolanda | ++ | 10:20 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - test https://review.openstack.org/359759 | 10:21 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud https://review.openstack.org/359757 | 10:25 |
| *** sdake has quit IRC | 10:28 | |
| *** kaisers_ has quit IRC | 10:30 | |
| *** jaosorior_lunch is now known as jaosorior | 10:30 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: [trove] Add more nv scenario tests https://review.openstack.org/354881 | 10:32 |
| amrith | sorry yolanda I took plan (b). made the change ian suggested and brought the line into compliance with the rest. AJaeger please take a look again. | 10:33 |
| amrith | and NO, this is NOT a blocker | 10:33 |
| amrith | so no requirement to do this urgently, let the CI do it's thing etc. | 10:33 |
| amrith | thanks! | 10:34 |
| *** rodrigods has quit IRC | 10:34 | |
| *** rodrigods has joined #openstack-infra | 10:34 | |
| yolanda | amrith, makes sense | 10:36 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Rename nodepool infracloud cacert from west to vanilla https://review.openstack.org/359781 | 10:39 |
| *** javeriak has joined #openstack-infra | 10:39 | |
| AJaeger | amrith: I'm sure it passes - +2A. | 10:41 |
| amrith | AJaeger, rcarrillocruz thank you | 10:42 |
| rcarrillocruz | np | 10:42 |
| *** florianf has quit IRC | 10:42 | |
| AJaeger | you're welcome, amrith | 10:42 |
| amrith | AJaeger, I'd also like to chat with you and annegentle about getting rid of the trove checkbuild job. | 10:45 |
| amrith | I think its time is now past | 10:46 |
| amrith | <link follows shortly> | 10:46 |
| AJaeger | amrith: I've done it already ;) | 10:46 |
| amrith | YOU DID! | 10:47 |
| amrith | nice | 10:47 |
| amrith | thanks AJaeger | 10:47 |
| AJaeger | amrith: https://review.openstack.org/358446 | 10:47 |
| AJaeger | amrith: and there's https://review.openstack.org/358738 to cleanup in trove | 10:47 |
| AJaeger | So, everything taken care of ;) | 10:47 |
| amrith | ah well, almost. | 10:48 |
| amrith | I saw this https://review.openstack.org/#/c/358943/ | 10:48 |
| amrith | and realized that I should've just taken care of it when I pushed the change for api-ref | 10:48 |
| amrith | but, maybe leave it there and every now and then, someone will try and fix it and we can -1 it :) | 10:48 |
| AJaeger | amrith: https://review.openstack.org/#/c/358738/3/tox.ini removes the lines that you commented on. | 10:49 |
| AJaeger | but it has not merged yet, is currently in gate. | 10:49 |
| amrith | Why do you think of everything? | 10:49 |
| amrith | :) | 10:50 |
| amrith | thanks AJaeger | 10:50 |
| AJaeger | amrith: I was on vacation and pushed a new openstack-doc-tools version out - and then checked for the fallout ;) | 10:50 |
| amrith | I guess I've missed the action these past few days at openstack-east | 10:50 |
| amrith | aha | 10:50 |
| amrith | cool | 10:50 |
| AJaeger | So, noticed that trove has no more XML content (great!) and old code lying around... | 10:50 |
| amrith | yup | 10:51 |
| amrith | now we've got to get the api-ref document maintained | 10:51 |
| amrith | I was speaking with sdague about how other projects handle keeping it up to date | 10:51 |
| amrith | and he had some good ideas for me | 10:51 |
| *** kzaitsev_mb has joined #openstack-infra | 10:52 | |
| AJaeger | might be worth sharing with other projects, keeping those updated was always a challenge ;( | 10:52 |
| AJaeger | amrith: mugsie has been working on theming for api-ref - see https://review.openstack.org/357926. I expect some followup on that. But that's presentation not content. Just a FYI | 10:53 |
| amrith | we were talking about something simple like just -1'ing changes if they don't update api-ref but make changes to the api | 10:53 |
| amrith | yes, saw that one | 10:53 |
| AJaeger | amrith: yes, good idea | 10:53 |
| amrith | when the thing was wadl and xml it was harder to enforce | 10:53 |
| *** hrubi_ has quit IRC | 10:53 | |
| AJaeger | like you -1 a change without release-notes if it needs one | 10:53 |
| amrith | now we can actually try and enforce it | 10:53 |
| amrith | yup | 10:53 |
| amrith | we do that now | 10:53 |
| AJaeger | RST is much easier in this regard | 10:54 |
| amrith | that it is | 10:54 |
| amrith | thanks AJaeger | 10:54 |
| amrith | will get ready for another fine day at @openstackeast now ... | 10:54 |
| * AJaeger pulled the plug on DocBook XML building yesterday - after all the great work annegentle, sdague, and others have been driving. | 10:54 | |
| AJaeger | amrith: enjoy! | 10:54 |
| amrith | yup, saw that email. Great work! | 10:54 |
| amrith | take care, and thanks again yolanda rcarrillocruz AJaeger ... now for some coffee! | 10:55 |
| rcarrillocruz | ++ | 10:55 |
| *** florianf has joined #openstack-infra | 10:56 | |
| *** jlibosva has quit IRC | 10:57 | |
| *** d0ugal has quit IRC | 10:59 | |
| *** d0ugal has joined #openstack-infra | 10:59 | |
| openstackgerrit | Merged openstack-infra/project-config: [trove] Add more nv scenario tests https://review.openstack.org/354881 | 11:00 |
| *** hrubi has joined #openstack-infra | 11:00 | |
| *** dizquierdo has joined #openstack-infra | 11:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove Neutron Postgres job from the Neutron check queue https://review.openstack.org/357519 | 11:10 |
| *** caowei has quit IRC | 11:12 | |
| *** jtomasek_ is now known as jtomasek | 11:13 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add admin infracloud connection details to Puppetmaster all-clouds https://review.openstack.org/359797 | 11:13 |
| *** rhallisey has joined #openstack-infra | 11:14 | |
| *** kushal has quit IRC | 11:15 | |
| *** thorst_ has joined #openstack-infra | 11:16 | |
| *** sdague has joined #openstack-infra | 11:17 | |
| *** Na3iL has quit IRC | 11:18 | |
| *** jkilpatr has joined #openstack-infra | 11:20 | |
| *** ramishra has quit IRC | 11:20 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/project-config: Few changes to the kolla-kubernetes job https://review.openstack.org/355199 | 11:21 |
| *** ramishra has joined #openstack-infra | 11:22 | |
| *** kushal has joined #openstack-infra | 11:23 | |
| *** thorst_ has quit IRC | 11:23 | |
| *** salv-orlando has quit IRC | 11:25 | |
| *** salv-orlando has joined #openstack-infra | 11:25 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add admin-infracloud cloud to launcher layouts https://review.openstack.org/359807 | 11:31 |
| *** sdague has quit IRC | 11:32 | |
| *** tpsilva has joined #openstack-infra | 11:34 | |
| *** thorst_ has joined #openstack-infra | 11:37 | |
| openstackgerrit | Merged openstack-infra/system-config: Rename nodepool infracloud cacert from west to vanilla https://review.openstack.org/359781 | 11:45 |
| *** asettle has quit IRC | 11:45 | |
| *** dtardivel has joined #openstack-infra | 11:47 | |
| *** Wei_Liu has quit IRC | 11:47 | |
| openstackgerrit | Merged openstack-infra/system-config: Add admin infracloud connection details to Puppetmaster all-clouds https://review.openstack.org/359797 | 11:49 |
| zigo | AJaeger: Where may I find the list of mirrors where the AFS is hosting the Debian packages? | 11:49 |
| *** hashar has quit IRC | 11:50 | |
| *** sigmavirus|away is now known as sigmavirus | 11:51 | |
| *** jaosorior has quit IRC | 11:51 | |
| *** jaosorior has joined #openstack-infra | 11:52 | |
| *** jlibosva has joined #openstack-infra | 11:53 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:54 | |
| *** ldnunes has joined #openstack-infra | 11:54 | |
| *** tonytan4ever has joined #openstack-infra | 11:57 | |
| *** rfolco has joined #openstack-infra | 11:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Install nodepool and shade into a virtualenv https://review.openstack.org/359425 | 11:58 |
| *** Wei_Liu has joined #openstack-infra | 11:59 | |
| *** vincentll has quit IRC | 11:59 | |
| *** lucasagomes is now known as lucas-hungry | 12:01 | |
| *** tonytan4ever has quit IRC | 12:01 | |
| *** andymaier has joined #openstack-infra | 12:03 | |
| *** andymaier has quit IRC | 12:03 | |
| AJaeger | zigo, for what do you need those? Those are for usage by our test jobs only and setup in the images itself | 12:04 |
| *** ggnel_t has quit IRC | 12:05 | |
| zigo | AJaeger: To answer Lana's question about where to find the repos for testing the doc. | 12:06 |
| zigo | AJaeger: I believe I've find a list, but I'm not sure if that's authoritative. | 12:06 |
| *** kushal has quit IRC | 12:07 | |
| zigo | AJaeger: Of course, the "official" repo list will be a Debian one after the release, but for testing the install-guide, it's IMO ok to use stuff hosted on the AFS. | 12:07 |
| *** psilvad has joined #openstack-infra | 12:07 | |
| *** ansmith has joined #openstack-infra | 12:08 | |
| *** kushal has joined #openstack-infra | 12:09 | |
| mordred | AJaeger: ++ for zigo's patch | 12:10 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: added: timmy https://review.openstack.org/359831 | 12:10 |
| *** dprince has joined #openstack-infra | 12:12 | |
| *** ccamacho is now known as ccamacho|lunch | 12:12 | |
| *** psilvad has quit IRC | 12:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move list_server cache to dogpile https://review.openstack.org/358871 | 12:14 |
| *** xyang1 has joined #openstack-infra | 12:14 | |
| *** bethwhite_ has quit IRC | 12:15 | |
| AJaeger | zigo: best discuss with pabelanger what location to hand out | 12:15 |
| zigo | AJaeger: Ok. | 12:15 |
| *** bethwhite_ has joined #openstack-infra | 12:15 | |
| zigo | AJaeger: Since Monty did a +2, maybe you can +2 workflow now? https://review.openstack.org/#/c/358819/ | 12:15 |
| zigo | :) | 12:15 |
| zigo | Sorry to insist, but fixtures is a build-dependency of almost everything ... | 12:16 |
| *** kushal has quit IRC | 12:16 | |
| *** hrybacki is now known as hrybacki|appt | 12:17 | |
| *** pradk has joined #openstack-infra | 12:18 | |
| *** vincentll has joined #openstack-infra | 12:18 | |
| *** kushal has joined #openstack-infra | 12:18 | |
| *** kaisers_ has joined #openstack-infra | 12:18 | |
| *** kdas__ has joined #openstack-infra | 12:21 | |
| *** coolsvap_ has quit IRC | 12:21 | |
| *** hieulq_ has joined #openstack-infra | 12:22 | |
| *** javeriak_ has joined #openstack-infra | 12:23 | |
| *** kaisers_ has quit IRC | 12:23 | |
| *** javeriak_ has quit IRC | 12:24 | |
| *** baoli has joined #openstack-infra | 12:24 | |
| *** Julien-zte has joined #openstack-infra | 12:24 | |
| *** kushal has quit IRC | 12:25 | |
| *** markvoelker has joined #openstack-infra | 12:25 | |
| *** gouthamr has joined #openstack-infra | 12:26 | |
| *** javeriak has quit IRC | 12:26 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config: Non-voting jobs for Neutron Xenial fullstack and functional https://review.openstack.org/359843 | 12:26 |
| *** yolanda has quit IRC | 12:27 | |
| AJaeger | zigo: will do in a bit... | 12:27 |
| *** hashar has joined #openstack-infra | 12:28 | |
| *** julim has joined #openstack-infra | 12:28 | |
| *** kdas__ is now known as kushal | 12:28 | |
| *** kushal has quit IRC | 12:28 | |
| *** kushal has joined #openstack-infra | 12:28 | |
| *** yolanda has joined #openstack-infra | 12:29 | |
| AJaeger | zigo +2A | 12:29 |
| zigo | Awesome ! | 12:29 |
| *** Na3iL has joined #openstack-infra | 12:31 | |
| *** vincentll has quit IRC | 12:31 | |
| *** vincentll has joined #openstack-infra | 12:34 | |
| *** xyang1 has quit IRC | 12:34 | |
| *** xyang1 has joined #openstack-infra | 12:36 | |
| jlibosva | AJaeger: thanks for review. I pinged clarkb in neutron channel to ping me back once he's online :) | 12:38 |
| AJaeger | jlibosva: great. He'll be here as well - but might take two more hours | 12:38 |
| *** asettle has joined #openstack-infra | 12:40 | |
| *** mdrabe has joined #openstack-infra | 12:41 | |
| *** markusry has joined #openstack-infra | 12:41 | |
| *** caowei has joined #openstack-infra | 12:42 | |
| *** yamamoto has quit IRC | 12:50 | |
| *** edmondsw has joined #openstack-infra | 12:50 | |
| *** zul has quit IRC | 12:51 | |
| *** pradk has quit IRC | 12:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:52 | |
| pabelanger | morning | 12:52 |
| mordred | morning pabelanger | 12:53 |
| mordred | pabelanger: https://review.openstack.org/#/c/359425/ passed gate jobs | 12:53 |
| *** markvoelker has quit IRC | 12:53 | |
| mordred | rcarrillocruz: ^^ you too | 12:53 |
| AJaeger | morning, pabelanger ! | 12:53 |
| *** vincentll has quit IRC | 12:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Add deb-python-fixtures to packaging-deb https://review.openstack.org/358819 | 12:54 |
| *** Julien-zte has quit IRC | 12:55 | |
| *** rlandy has joined #openstack-infra | 12:56 | |
| *** kushal has quit IRC | 12:56 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat https://review.openstack.org/358868 | 12:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for registering a watch. https://review.openstack.org/358837 | 12:58 |
| *** tonytan4ever has joined #openstack-infra | 12:58 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 12:58 |
| EmilienM | nibalizer: fyi https://review.openstack.org/#/c/359539/ | 12:59 |
| *** lucas-hungry is now known as lucasagomes | 12:59 | |
| AJaeger | mordred, pabelanger: Here're a few changes to move other-requirements to bindep.txt - could you help to land these, please? https://review.openstack.org/#/q/status:open+projects:openstack-infra+topic:bindep-mv | 13:00 |
| *** gyx has quit IRC | 13:00 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 13:01 |
| pabelanger | So, still seeing a large amount of failures in rax-iad when launching nodes. We are timing out SSHing into the ipv4 address. We have a contact at rackspace that can help debug the issue or should I just create a ticket? | 13:01 |
| *** tonytan4ever has quit IRC | 13:02 | |
| rcarrillocruz | pabelanger, mordred : fyi, just came across https://github.com/ansible/ansible-modules-extras/pull/2709/files | 13:03 |
| rcarrillocruz | it's good, although we rather put the logic in shade, then call it from the module | 13:03 |
| pabelanger | AJaeger: looking | 13:03 |
| *** ddieterly has joined #openstack-infra | 13:03 | |
| *** baoli_ has joined #openstack-infra | 13:04 | |
| mordred | rcarrillocruz: looking | 13:05 |
| rcarrillocruz | do we have documentation on what the quotas should look like on our clouds, for the openstackjenkins/openstackci projects | 13:05 |
| rcarrillocruz | ? | 13:05 |
| *** vikrant has quit IRC | 13:05 | |
| mordred | rcarrillocruz: I agree - that is good | 13:05 |
| mordred | rcarrillocruz: maybe let's land it, but before we do ask if it's ok with the author if we move the logic he wrote there into shade | 13:06 |
| pabelanger | rcarrillocruz: http://docs.openstack.org/infra/system-config/contribute-cloud.html | 13:06 |
| mordred | rcarrillocruz: we need his permission because the ansible module is GPL and shade isn't | 13:06 |
| rcarrillocruz | mordred: ack | 13:06 |
| *** kgiusti has joined #openstack-infra | 13:06 | |
| pabelanger | 8vCPUs, 8GB RAM, 80GB HDD our the flavors we want, so quota should reflect that | 13:06 |
| *** baoli has quit IRC | 13:06 | |
| mordred | then we can land a followup patch to remove logic from the module | 13:07 |
| rcarrillocruz | pabelanger: k, can you look https://review.openstack.org/#/c/359807/1 ? | 13:07 |
| cloudnull | mornings | 13:07 |
| *** salv-orlando has quit IRC | 13:07 | |
| *** ddieterly has quit IRC | 13:07 | |
| rcarrillocruz | i'll run the launcher, then create the needed oscc clouds on puppetmaster | 13:07 |
| *** eharney has joined #openstack-infra | 13:07 | |
| *** psachin has quit IRC | 13:08 | |
| pabelanger | rcarrillocruz: looks good | 13:08 |
| *** dprince has quit IRC | 13:08 | |
| rcarrillocruz | thx | 13:08 |
| mordred | cloudnull: morning! did you see all the fun scrollback from last night related to osic ipv6? | 13:09 |
| *** pgadiya has quit IRC | 13:09 | |
| mordred | cloudnull: oh - I see thta you did! | 13:09 |
| *** vincentll has joined #openstack-infra | 13:09 | |
| *** dprince has joined #openstack-infra | 13:09 | |
| *** asettle has quit IRC | 13:09 | |
| *** kushal has joined #openstack-infra | 13:10 | |
| *** asettle has joined #openstack-infra | 13:10 | |
| cloudnull | re : the sysctl bits and devstack? | 13:10 |
| openstackgerrit | Merged openstack-infra/groups: Move other-requirements.txt to bindep.txt https://review.openstack.org/354859 | 13:10 |
| mordred | cloudnull: yah. fun stuff right? | 13:11 |
| *** esberglu has joined #openstack-infra | 13:12 | |
| cloudnull | it looked like a blast :) | 13:12 |
| rcarrillocruz | so folks, do we keep the images built by nodepool somewhere? http://nodepool.openstack.org just show logs . Any 'cache' folder where dib keeps those images ? | 13:13 |
| cloudnull | but good stuff to find IMO. I'm sure all of this work will be greatly appreciated as more clouds spin up and go the v6 route. | 13:13 |
| * rcarrillocruz would love his ISP would follow the v6 route :/ | 13:14 | |
| mordred | rcarrillocruz: /opt/nodepool_dib | 13:14 |
| cloudnull | we've also got another OSIC cloud planned for the santa clara DC which will have similar properties so once thats online (I've no idea when that'll be) we'll have yet another region for everyone to play with. | 13:15 |
| rcarrillocruz | k, those numbers correlate to nodepool image build logs i assume | 13:15 |
| mordred | yah | 13:15 |
| mordred | cloudnull: woot! | 13:15 |
| openstackgerrit | Merged openstack-infra/reviewday: Move other-requirements.txt to bindep.txt https://review.openstack.org/354864 | 13:15 |
| openstackgerrit | Merged openstack-infra/gear: Move other-requirements.txt to bindep.txt https://review.openstack.org/354858 | 13:16 |
| openstackgerrit | Merged openstack-infra/openstackid: Move other-requirements.txt to bindep.txt https://review.openstack.org/354860 | 13:16 |
| openstackgerrit | Merged openstack-infra/nodepool: Add floating-ip batching settings to clouds.yaml https://review.openstack.org/359327 | 13:18 |
| mordred | rcarrillocruz: hey - while I'm bothering you ... | 13:18 |
| openstackgerrit | Merged openstack-infra/nodepool: Install nodepool and shade into a virtualenv https://review.openstack.org/359425 | 13:18 |
| mordred | rcarrillocruz: https://review.openstack.org/#/c/359378/ ... this is in support of launch-node - but also is likely something we can carry over into os_server and cloud-launcher | 13:18 |
| *** raunak has joined #openstack-infra | 13:18 | |
| rcarrillocruz | oh yeah | 13:18 |
| rcarrillocruz | that's neat | 13:18 |
| rcarrillocruz | let me review | 13:18 |
| rcarrillocruz | cloudnull: woot, moar clouds pls! | 13:19 |
| rcarrillocruz | lgtm mordred | 13:20 |
| mordred | thx! | 13:20 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Move other-requirements.txt to bindep.txt https://review.openstack.org/354862 | 13:22 |
| pabelanger | jeblair: we have an example of nodepool not getting data from gearman: http://paste.openstack.org/show/562933/ | 13:22 |
| pabelanger | nodepool: http://paste.openstack.org/show/562934/ | 13:22 |
| pabelanger | zuul-launcher: http://paste.openstack.org/show/562935/ | 13:22 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: Ansible-like tool https://review.openstack.org/359831 | 13:22 |
| *** woodster_ has joined #openstack-infra | 13:23 | |
| *** senk has quit IRC | 13:24 | |
| *** pgadiya has joined #openstack-infra | 13:25 | |
| *** openstackgerrit has quit IRC | 13:26 | |
| *** openstackgerrit has joined #openstack-infra | 13:26 | |
| *** jlibosva has quit IRC | 13:28 | |
| *** amotoki has quit IRC | 13:29 | |
| AJaeger | thanks, mordred and pabelanger ! | 13:29 |
| *** haleyb has joined #openstack-infra | 13:30 | |
| *** tqtran has joined #openstack-infra | 13:32 | |
| *** yamamoto has joined #openstack-infra | 13:32 | |
| *** sdake has joined #openstack-infra | 13:34 | |
| *** pcrews has quit IRC | 13:34 | |
| *** _ari_ has joined #openstack-infra | 13:35 | |
| *** tonytan4ever has joined #openstack-infra | 13:35 | |
| *** tqtran has quit IRC | 13:37 | |
| *** sdake_ has joined #openstack-infra | 13:37 | |
| *** zhurong has joined #openstack-infra | 13:37 | |
| AJaeger | pabelanger: could you also review https://review.openstack.org/355038 - for windmill, please? | 13:37 |
| *** eharney has quit IRC | 13:38 | |
| *** AJaeger has quit IRC | 13:39 | |
| *** yamamoto has quit IRC | 13:40 | |
| *** thiagop has joined #openstack-infra | 13:40 | |
| *** sdake has quit IRC | 13:40 | |
| *** ddieterly has joined #openstack-infra | 13:41 | |
| *** zhurong has quit IRC | 13:41 | |
| _nadya_ | dear openstack-infra team! Could you please add nprivalova@mirantis.com to the group https://review.openstack.org/#/admin/groups/1535,members ? It is a new project created by https://review.openstack.org/#/c/355406/ . Thanks in advance! | 13:42 |
| *** zhurong has joined #openstack-infra | 13:42 | |
| rcarrillocruz | _nadya_: done | 13:42 |
| *** eharney has joined #openstack-infra | 13:43 | |
| haleyb | So does the accept_ra change seem to help with ipv6 at OSIC? i'm just waking up but was pinged and see the patch in review | 13:43 |
| mordred | haleyb: it's half of the issue | 13:43 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr: Add fuxi subproject to gerritbot https://review.openstack.org/359899 | 13:43 |
| *** amotoki has joined #openstack-infra | 13:44 | |
| mordred | haleyb: the other half we don't have a solution for ... but it seems that as soon as an ipv6 subnet is created in the devstack neutron, the default ipv6 route on the host gets set to use the loopback interface | 13:44 |
| _nadya_ | rcarrillocruz: thanks! | 13:44 |
| openstackgerrit | Merged openstack-infra/system-config: Add admin-infracloud cloud to launcher layouts https://review.openstack.org/359807 | 13:44 |
| mordred | which is then why we lose connectivity to the host | 13:44 |
| *** sandanar has quit IRC | 13:45 | |
| haleyb | mordred: yuck, easy to reproduce? | 13:45 |
| mordred | haleyb: yup. it happens consistently every time | 13:45 |
| mordred | haleyb: clarkb may be more useful than me as soon as he wakes up - I'm mostly just caught up from reading scrollback | 13:45 |
| *** AJaeger has joined #openstack-infra | 13:46 | |
| *** ddieterly has quit IRC | 13:46 | |
| *** ddieterly has joined #openstack-infra | 13:46 | |
| mordred | haleyb: here's a paste http://paste.openstack.org/show/562831/ of the relevant devstack log leading up to the route going away | 13:46 |
| haleyb | mordred: ok, i'm just wondering if i can reproduce it locally to debug it | 13:46 |
| *** VnrycepuxO has joined #openstack-infra | 13:46 | |
| dougwig | haleyb: crazy easy to reproduce. i expect we can do so by getting an ipv6 node on any cloud and installing devstack. | 13:46 |
| dougwig | haleyb: or watch zuul, it fails every time. | 13:47 |
| mordred | haleyb: and this: http://paste.openstack.org/show/562777/ has the nics and routes on a host that this has happened to | 13:47 |
| haleyb | just on OSIC thought, right? | 13:47 |
| mordred | ::/0 :: !n -1 1 9429 lo | 13:47 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 13:47 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 13:47 |
| dougwig | haleyb: correct, but that's the only ipv6 addressed cloud infra uses. | 13:47 |
| mordred | well, it happens everywhere with ipv6 - but osic is the only place that ipv6 is the only thing we have | 13:47 |
| mordred | so we can connect to other things via ipv4 | 13:47 |
| haleyb | mordred: so are you on that system now from the paste? | 13:48 |
| mordred | actually ... I'm kind of confused now that I say that outloud ... rackspace has ipv6 and we talk to it over ipv6 ... but I wonder if the specific route for the cloud itself saves us there, since nodepool is in rackspace as well | 13:48 |
| mordred | (we can connect to the osic nodes from osic, because there is a specific route for osic) | 13:49 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr-libnetwork-core addition https://review.openstack.org/357945 | 13:49 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr-kubernetes-core addition https://review.openstack.org/357946 | 13:49 |
| mordred | haleyb: I'm not anymore, no | 13:49 |
| dougwig | mordred: rax nodes have ipv4 addresses listed in zuul. | 13:49 |
| mordred | dougwig: good point | 13:49 |
| mordred | I guess I'm not sure why they're using ipv4 and not 6 - but let's consider that a happy accident right now | 13:49 |
| *** matt-borland has joined #openstack-infra | 13:49 | |
| dougwig | haleyb: clarkb said that after the subnet create in that log, the ipv6 default route is gone. | 13:49 |
| mordred | haleyb: I do have that machine still | 13:50 |
| mordred | and am on it again | 13:50 |
| mordred | dougwig, haleyb: would access to the held node be useful? | 13:50 |
| pabelanger | mordred: rackspace has ipv6 and we talk to it over ipv6; from where nodepool? | 13:51 |
| mordred | pabelanger: yah | 13:51 |
| haleyb | mordred: can you show the routing table using 'ip -6 r' instead of route? my eyes are just more used to it, and yeah, access might help | 13:51 |
| dougwig | mordred: i'm not versed enough in ipv6. haleyb ? | 13:51 |
| mordred | haleyb: sure | 13:51 |
| cloudnull | haleyb: I'm guessing that it has something to do with bindings w/in "/etc/radvd.conf" | 13:51 |
| *** sdague has joined #openstack-infra | 13:51 | |
| *** pt_15 has joined #openstack-infra | 13:51 | |
| *** AJaeger has quit IRC | 13:51 | |
| mordred | haleyb: http://paste.openstack.org/show/563004/ | 13:51 |
| *** xarses has quit IRC | 13:52 | |
| *** AJaeger has joined #openstack-infra | 13:52 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| dougwig | mordred: is that node using ovs or lb? | 13:53 |
| haleyb | cloudnull: so this devstack node is running radvd too? | 13:53 |
| mordred | dougwig: the cloud the vm is running on? or the devstack installed in the node? | 13:53 |
| pabelanger | mordred: I am not sure we are actually doing that. At least in rax-iad we use ipv4, but not sure why that is | 13:53 |
| dougwig | mordred: the latter | 13:53 |
| sdague | mordred et al. - can we promote - 359721 which is the cinder fix for the gate? | 13:54 |
| *** raildo has joined #openstack-infra | 13:54 | |
| pabelanger | mordred: we get ipv6 but nodepool is still using ipv4 | 13:54 |
| *** DrifterZA has quit IRC | 13:54 | |
| mordred | pabelanger: weird. we should be using ipv6 since it's there ... I'll look in to that once we fix this :) | 13:54 |
| mordred | dougwig: it's running ovs | 13:54 |
| openstackgerrit | Volodymyr Stoiko proposed openstack-infra/project-config: Add fuel-plugin-rally project https://review.openstack.org/359076 | 13:54 |
| dougwig | mordred: can you get us "ovs-vsctl show" ? | 13:54 |
| pabelanger | mordred: Ya, I can check no problem. Just wanted to confirm we want to use ipv6 for rackspace from nodepool | 13:54 |
| *** florianf has quit IRC | 13:55 | |
| *** DrifterZA has joined #openstack-infra | 13:55 | |
| dougwig | pabelanger: not until we fix this, we don't. | 13:55 |
| mordred | pabelanger: in theory, we want to use it everywhere | 13:55 |
| mordred | but yeah, what dougwig said | 13:55 |
| pabelanger | right | 13:55 |
| mordred | pabelanger: can you get sdague's promote? | 13:55 |
| pabelanger | looking | 13:55 |
| mordred | dougwig: http://paste.openstack.org/show/563005/ | 13:56 |
| haleyb | mordred: that paste wasn't as useful as i thought, ' ip -6 r s table all' might be better, but a login might be best if the logs show anything | 13:56 |
| mordred | haleyb: so - I realized after I offered that that you need a vm in osic to be able to bounce through | 13:56 |
| mordred | since the node only has working routing to other osic nodes :) | 13:56 |
| *** kzaitsev_mb has quit IRC | 13:56 | |
| mordred | haleyb: http://paste.openstack.org/show/563006/ | 13:57 |
| dougwig | haleyb, mordred: so, the default route in the first paste was through br-ex, and looking at the ovs dump, eth0 is not in br-ex. | 13:57 |
| *** hongbin has joined #openstack-infra | 13:57 | |
| mordred | dougwig: right. which is what we're expecting - in theory the neutron on the node should not be modifying eth0 | 13:57 |
| mordred | so that's good | 13:58 |
| cloudnull | haleyb i thought so. | 13:58 |
| dougwig | right, but how do the non-neutron packets get out anymore? they'd go into br-ex and stop. unless going out to the local net (i.e. osic). | 13:58 |
| cloudnull | i thought that's what folks were saying last night | 13:58 |
| cloudnull | but I could be completely wrong | 13:59 |
| dougwig | mordred: i was expecting to see eth0 in br-ex, not eth0 being modified. | 13:59 |
| *** pradk has joined #openstack-infra | 13:59 | |
| pabelanger | mordred: sdague: promoted | 14:00 |
| *** zz_dimtruck is now known as dimtruck | 14:00 | |
| fungi | mordred: to answer your earlier question about ipv6 on job nodes in rackspace... we lost that when we switched to glean. i suspect v6 address info is not being included in the configdrive metadata but haven't had time to dig into it yet | 14:00 |
| fungi | or at least that was the case for a while | 14:01 |
| mordred | fungi: ah. gotcha | 14:01 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add fingerprint for cinder backup test fails https://review.openstack.org/359904 | 14:01 |
| sdague | prometheanfire: thank you | 14:01 |
| * rcarrillocruz thinks the cloud launcher output could be GREATLY improved, to know what's going on at any given time | 14:01 | |
| mordred | rcarrillocruz: :) | 14:01 |
| prometheanfire | sdague: ? | 14:01 |
| *** kushal has quit IRC | 14:01 | |
| sdague | oh, sorry, wrong tab complete | 14:01 |
| sdague | pabelanger: thank you | 14:02 |
| prometheanfire | :D | 14:02 |
| *** amotoki has quit IRC | 14:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 14:02 |
| fungi | mordred: just jumped on a trusty node in rax-ord and indeed, /mnt/config/openstack/content/0000 only has v4 addresses in it | 14:03 |
| fungi | mordred: same for /mnt/config/openstack/latest/network_data.json | 14:03 |
| rcarrillocruz | agh | 14:04 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add a margin to the bottom of all pages https://review.openstack.org/359119 | 14:04 |
| *** kzaitsev_mb has joined #openstack-infra | 14:04 | |
| rcarrillocruz | puppet-infracloud doesn't put the cert of the controller on the puppetmaster, does it? | 14:04 |
| rcarrillocruz | yolanda: do you remember? ^ | 14:04 |
| rcarrillocruz | i think my launcher runs fail due to that | 14:05 |
| rcarrillocruz | pretty sure, the oscc yaml does not have verify: no | 14:05 |
| rcarrillocruz | and haven't messes any update ca certificates manually | 14:05 |
| yolanda | no, it doesn't | 14:05 |
| dougwig | mordred, haleyb: so, what happened to this: | 14:05 |
| dougwig | https://www.irccloud.com/pastebin/7GTbZcGw/ | 14:05 |
| rcarrillocruz | booooh | 14:05 |
| *** asettle has quit IRC | 14:05 | |
| mordred | fungi: ip? I'd like to look | 14:05 |
| fungi | mordred: also /mnt/config/openstack/latest/vendor_data.json seems to be v4-only | 14:05 |
| yolanda | puppet-infracloud is just managing controllers and computes | 14:05 |
| fungi | mordred: 104.130.216.250 but it's not held so may go away at any time | 14:06 |
| haleyb | mordred: so i have a local VM i ran devstack on yesterday and it's just as broken, v4 too, but could be a red herring for me | 14:06 |
| mordred | haleyb: woot! | 14:06 |
| haleyb | i just never noticed because it's running a window manager | 14:06 |
| mordred | dougwig: hrm. are we maybe running that too late? | 14:07 |
| rcarrillocruz | hmm, we should puppetize that | 14:07 |
| rcarrillocruz | because we do that on nodepool for example, but not on puppetmaster | 14:07 |
| *** kaisers_ has joined #openstack-infra | 14:07 | |
| *** sbezverk_ has quit IRC | 14:07 | |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr-libnetwork-core addition https://review.openstack.org/357945 | 14:07 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr-kubernetes-core addition https://review.openstack.org/357946 | 14:07 |
| *** yamahata has joined #openstack-infra | 14:07 | |
| mordred | pabelanger: what was it you did to get the ipv6 address to show up on the osic mirror? | 14:08 |
| *** sbezverk has joined #openstack-infra | 14:08 | |
| apuimedo | thanks for the review AJaeger | 14:08 |
| apuimedo | I fixed the redundant line | 14:08 |
| pabelanger | mordred: you mean when I added eth1.conf and ifup? | 14:09 |
| *** florianf has joined #openstack-infra | 14:09 | |
| mordred | yah | 14:09 |
| mordred | that was all? | 14:10 |
| pabelanger | http://paste.openstack.org/show/563008/ | 14:10 |
| pabelanger | yup | 14:10 |
| AJaeger | apuimedo: LGTM | 14:10 |
| pabelanger | dropped that into /etc/network/interfaces.d | 14:10 |
| pabelanger | sudo ifup | 14:10 |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: update base_email_ext to adapt Email-ext plugin https://review.openstack.org/355139 | 14:10 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Refactor infra-cloud-bridge element to support CentOS/RH https://review.openstack.org/359909 | 14:10 |
| apuimedo | thanks AJaeger | 14:10 |
| *** kushal has joined #openstack-infra | 14:11 | |
| dougwig | mordred: can i see /etc/neutron/plugins/ml2/ml2_conf.ini ? | 14:11 |
| *** kaisers_ has quit IRC | 14:11 | |
| mordred | dougwig: http://paste.openstack.org/show/563009/ | 14:13 |
| *** rbrndt has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack-infra/zuul: Improve debug output from tests https://review.openstack.org/358008 | 14:16 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove support for legacy rh1 https://review.openstack.org/347918 | 14:18 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use delorean's exit code to decide on retry https://review.openstack.org/348790 | 14:19 |
| *** yamamoto has quit IRC | 14:20 | |
| *** Swami has joined #openstack-infra | 14:20 | |
| *** Swami_ has joined #openstack-infra | 14:20 | |
| mtreinish | fungi, pleia2: it doesn't look the e-r cron jobs are updating things | 14:21 |
| pleia2 | let's see | 14:21 |
| mtreinish | I thought we got that sorted before, is there anything in the logs | 14:21 |
| mtreinish | pleia2: thanks | 14:21 |
| *** amotoki has joined #openstack-infra | 14:22 | |
| fungi | pleia2: i'm happy to look too if you get stuck, but am even more happy to leave it to you ;) | 14:22 |
| pleia2 | mtreinish: the new/ directory is back, and I'm positive that a failure that didn't allow the cron command to complete and delete that is what caused it to remain | 14:23 |
| *** michauds has joined #openstack-infra | 14:24 | |
| pleia2 | mtreinish: I manually removed it just now, but maybe start the cron command with rmdir /var/lib/elastic-recheck/new/ ? | 14:24 |
| mtreinish | pleia2: ok, I can do that. I'll push a patch to add that in front of the command | 14:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move os-api-ref to release team https://review.openstack.org/359916 | 14:25 |
| pleia2 | or have a conditional around creating it, so it doesn't fail if it exists | 14:25 |
| * pleia2 looks at mkdir man page | 14:25 | |
| pleia2 | mtreinish: ok, so just change "mkdir new" to "mkdir -p new" | 14:26 |
| *** hpe-hj has joined #openstack-infra | 14:26 | |
| pleia2 | no need to add rmdir | 14:26 |
| mtreinish | pleia2: ok, that's an easier patch | 14:27 |
| * pleia2 nods | 14:27 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: sahara/tempest: run also client tests and a pre script https://review.openstack.org/359920 | 14:27 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-elastic_recheck: Add missing -p to mkdir on uncat cron job https://review.openstack.org/359921 | 14:28 |
| mtreinish | pleia2, fungi: ^^^ | 14:28 |
| *** michauds has quit IRC | 14:28 | |
| dougwig | mordred: can you get /opt/stack/logs tarred up and put somewhere? | 14:29 |
| mtreinish | fungi: you can fast +A that too :) | 14:29 |
| rcarrillocruz | oh, voucher from ovh | 14:29 |
| rcarrillocruz | that's nice :-) | 14:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move os-api-ref to release team https://review.openstack.org/359916 | 14:30 |
| *** javeriak has joined #openstack-infra | 14:30 | |
| fungi | mtreinish: i'm curious why it had the cleanup rm at the end. does it not overwrite the directory contents it finds anyway? | 14:30 |
| *** ccamacho|lunch is now known as ccamacho | 14:30 | |
| fungi | rcarrillocruz: no idea how far that voucher will stretch, though i gather they're one of the less expensive providers so maybe pretty far | 14:32 |
| rcarrillocruz | ++ | 14:32 |
| *** kaisers_ has joined #openstack-infra | 14:32 | |
| mtreinish | fungi: the e-r command should overwrite the contents. I'm not sure why the rm isn't doing it's thing though | 14:32 |
| mordred | dougwig: http://mirror.regionone.osic-cloud1.openstack.org/logs.tgz | 14:32 |
| mtreinish | but yeah the rm is not needed anymore if we mkdir -p it | 14:32 |
| pleia2 | mtreinish: I think the issue is that sometimes the command fails, so it doesn't *get* to the rm part | 14:32 |
| mtreinish | tbh I think I just left the rm there because the old single file version used it | 14:32 |
| pleia2 | because computers are terrible | 14:32 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Refactor infra-cloud-bridge element to support CentOS/RH https://review.openstack.org/359909 | 14:33 |
| fungi | mtreinish: yeah, just seems unnecessary (also, post-run cleanup with anything other than a trap is failure-prone) | 14:34 |
| *** piet has joined #openstack-infra | 14:34 | |
| *** piet has quit IRC | 14:34 | |
| *** mdrabe has quit IRC | 14:34 | |
| mtreinish | fungi: I can push another patch to remove it if you'd like | 14:35 |
| mtreinish | pleia2: in the meantime can we manually run the thing to update stuff while waiting for the gate | 14:35 |
| fungi | mtreinish: if you don't mind--it makes the cronjob a bit cleaner | 14:35 |
| mtreinish | fungi: sure np, one sec | 14:35 |
| pleia2 | mtreinish: yeah, sec | 14:35 |
| *** amotoki has quit IRC | 14:36 | |
| *** eharney has quit IRC | 14:36 | |
| *** berendt has quit IRC | 14:37 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-elastic_recheck: Remove unnecessary rm from uncat cron job https://review.openstack.org/359927 | 14:37 |
| mtreinish | fungi: ^^^ | 14:37 |
| pleia2 | recheck user is running it now | 14:37 |
| mtreinish | pleia2: thanks | 14:37 |
| *** mdrabe has joined #openstack-infra | 14:39 | |
| *** Swami_ has quit IRC | 14:40 | |
| *** Swami has quit IRC | 14:40 | |
| *** pcrews has joined #openstack-infra | 14:40 | |
| *** Swami has joined #openstack-infra | 14:40 | |
| pleia2 | mtreinish: should be updated now | 14:40 |
| mtreinish | pleia2: yep, thanks | 14:40 |
| *** psachin has joined #openstack-infra | 14:41 | |
| *** nwkarsten has joined #openstack-infra | 14:41 | |
| *** piet has joined #openstack-infra | 14:42 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Deploy minimal services in multinode job https://review.openstack.org/355097 | 14:42 |
| *** oanson has quit IRC | 14:43 | |
| *** yamamoto has joined #openstack-infra | 14:43 | |
| *** esikachev has joined #openstack-infra | 14:43 | |
| *** hongbin has quit IRC | 14:43 | |
| *** xarses has joined #openstack-infra | 14:44 | |
| *** raunak has quit IRC | 14:44 | |
| *** raunak has joined #openstack-infra | 14:44 | |
| *** hieulq_ has quit IRC | 14:45 | |
| d0ugal | What/where is the best place for me to report an invalid check-osc-plugin failure? | 14:46 |
| openstackgerrit | Merged openstack-infra/nodepool: Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 14:47 |
| AJaeger | d0ugal: I suggest to ask on #openstack-sdks - that's where the osc folks hang out. stevemar wrote the tests | 14:48 |
| d0ugal | AJaeger: Will do, thanks. | 14:48 |
| rcarrillocruz | hmm, i added the controller certificate to the ca-certificates on the puppetmaster. However I get errors, due to SSL when runninc osc. Passing explicitly OS_CACERT=/etc/ssl/certs/ca-certificates work. I was expecting the osc would default to that? | 14:48 |
| rcarrillocruz | mordred: ^ | 14:48 |
| stevemar | d0ugal: link? | 14:48 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: Ansible-like tool https://review.openstack.org/359831 | 14:48 |
| d0ugal | stevemar: https://review.openstack.org/#/c/359784 | 14:48 |
| rcarrillocruz | should I put an cacert param on the infracloud cloud on clouds.yaml ? | 14:48 |
| mordred | rcarrillocruz: are you using clouds.yaml for othe rthings? | 14:48 |
| mordred | rcarrillocruz: yes | 14:48 |
| *** rajinir has joined #openstack-infra | 14:48 | |
| rcarrillocruz | k thx | 14:48 |
| mordred | env vars do not overlay over top of clouds.yaml settings | 14:49 |
| mordred | because there are multiple clouds in a clouds.yaml, it turns out to behave unexpectedly more often than not | 14:49 |
| rcarrillocruz | ah gotcha | 14:49 |
| mordred | env vars (other than OS_CLOUD and OS_REGION_NAME) go into a cloud named "envvars" | 14:49 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: highlight when integrated_gate data is out of date https://review.openstack.org/359932 | 14:49 |
| *** amotoki has joined #openstack-infra | 14:50 | |
| d0ugal | stevemar: I'd like to add the exception to the output here: https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/check_osc_commands.py#L86 - do you think that would make sense? | 14:50 |
| *** javeriak has quit IRC | 14:52 | |
| *** kaisers_ has quit IRC | 14:52 | |
| haleyb | mordred: would https://review.openstack.org/#/c/359490/ be part of the undercloud deployment? just wondering how to test any patches, not including getting them to run in OSIC | 14:52 |
| stevemar | d0ugal: sure, i'm good with that | 14:53 |
| stevemar | d0ugal: i haven't changed it since i created it | 14:53 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Pass cacert param for admin-infracloud https://review.openstack.org/359934 | 14:54 |
| AJaeger | d0ugal: why does it fail? | 14:54 |
| d0ugal | AJaeger: I have no idea, the exception is swalled :) | 14:54 |
| *** matt-borland has quit IRC | 14:54 | |
| rcarrillocruz | mordred: ^ , i put in place in the puppetmaster all-clouds.yaml and seems to work | 14:54 |
| *** abregman has quit IRC | 14:54 | |
| *** eharney has joined #openstack-infra | 14:55 | |
| openstackgerrit | Merged openstack-infra/nodepool: Remove unnecessary NodePoolBuilder thread https://review.openstack.org/356676 | 14:55 |
| d0ugal | AJaeger: From my reading of the code I must be hitting this path: https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/check_osc_commands.py#L86 | 14:55 |
| *** edtubill has joined #openstack-infra | 14:55 | |
| AJaeger | d0ugal, stevemar: I'm wondering whether you have a bug in your change - or whether it's valid. So, producing a good exception is something I happyily +2 - just ignoring it sounds wrong. Not sure what you want to do... | 14:58 |
| d0ugal | AJaeger: Adding a patch to output the traceback. | 14:58 |
| stevemar | AJaeger: every time that gate has failed has been for legimate reasons :P | 14:58 |
| *** Julien-zte has joined #openstack-infra | 14:58 | |
| d0ugal | stevemar: heh, that's fair. I may have done something but I have tested the code loads :( | 14:59 |
| stevemar | normally it's a typo, but not seeing one in the patch | 14:59 |
| AJaeger | then we're in agreement... | 14:59 |
| *** yuval has quit IRC | 14:59 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Add the exception string when a osc command fails to load https://review.openstack.org/359937 | 14:59 |
| *** jistr is now known as jistr|mtg | 15:00 | |
| *** rbrndt has quit IRC | 15:00 | |
| scottda | AJaeger: could you have a look at "Add experimental Cinder job for multibackend" when you've a chance? I think I've addressed your comments and it just needs a post-rebase re-approval . https://review.openstack.org/#/c/330678/ | 15:00 |
| d0ugal | AJaeger, stevemar: ^ it's a bit rough, but might be enough? | 15:00 |
| *** hrybacki|appt is now known as hrybacki | 15:00 | |
| AJaeger | scottda: LGTM. | 15:01 |
| AJaeger | d0ugal: will review later, thanks. | 15:01 |
| *** raunak has quit IRC | 15:01 | |
| * AJaeger needs to step out for a bit now | 15:01 | |
| scottda | AJaeger: Thanks. yolanda had +A'd, would either of you please re-approve ? | 15:02 |
| sdague | ok, new patch, for direct dump into gate - https://review.openstack.org/#/c/359939/ | 15:05 |
| sdague | now that uncategorized failures are up to date, I think that's our major issue | 15:05 |
| *** rbrndt has joined #openstack-infra | 15:05 | |
| sdague | so if someone could push to gate and promote, it will probably make a lot of things better | 15:05 |
| *** raunak has joined #openstack-infra | 15:06 | |
| sdague | pabelanger / mordred / fungi can I tag one of you for promote on that? | 15:06 |
| *** raunak has quit IRC | 15:06 | |
| rcarrillocruz | scottda: approved | 15:07 |
| scottda | rcarrillocruz: Thanks for that. | 15:07 |
| *** zhurong has quit IRC | 15:08 | |
| *** kaisers_ has joined #openstack-infra | 15:08 | |
| fungi | sdague: sure, working on that now | 15:09 |
| *** abregman has joined #openstack-infra | 15:09 | |
| hashar | hello ! | 15:10 |
| hashar | does anyone have details about the OVH quota accounting being off back in July ? I have seen the Nodepool max-server got lowered to accommodate with false "Quota exceeds for instances" in https://review.openstack.org/#/c/347075/ | 15:10 |
| hashar | turns out I am hitting the same issue with an OpenStack Liberty cloud provider and I am looking for any potential hints :} | 15:10 |
| hashar | maybe pabelanger clarkb AJaeger would know | 15:11 |
| fungi | sdague: okay, 359939,2 is at the top of the gate pipeline now | 15:12 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental Cinder job for multibackend https://review.openstack.org/330678 | 15:13 |
| *** vinaypotluri has joined #openstack-infra | 15:14 | |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: Ansible-like tool https://review.openstack.org/359831 | 15:14 |
| *** thcipriani has joined #openstack-infra | 15:15 | |
| *** caowei has quit IRC | 15:15 | |
| sdague | fungi: thank you | 15:15 |
| *** andreas_s has quit IRC | 15:15 | |
| zigo | pabelanger: I'd be nice if you could reconsider the change for min-ready of the jessie nodes. Look at this: http://grafana.openstack.org/dashboard/db/nodepool in the jessie "Ready nodes" graph. More than 50% of the time when I'm working on the packaging, there's no node ready, and therefore, I need to wait. | 15:15 |
| fungi | sdague: ant time. let me know if you turn up others | 15:15 |
| pleia2 | ant time \o/ | 15:16 |
| zigo | pabelanger: It shows well on the last hour graph right now. | 15:16 |
| *** hockeynut has joined #openstack-infra | 15:16 | |
| sdague | pleia2: ++ especially with infra logo | 15:17 |
| pleia2 | :D | 15:17 |
| *** greg-g has joined #openstack-infra | 15:17 | |
| fungi | zigo: rephrased, you're saying we shouldn't build as many ubuntu nodes even though there are a massive backlog of jobs waiting for them | 15:17 |
| *** david-lyle has joined #openstack-infra | 15:17 | |
| *** salv-orlando has joined #openstack-infra | 15:18 | |
| hashar | thcipriani: so here is the place of people maintain OpenStack infra namely Zuul / Nodepool and other things :d | 15:19 |
| mordred | zigo: for context, I've been trying to land 5 shade patches for 2 solid days | 15:19 |
| fungi | zigo: we have capacity to build around 1000 nodes currently, and the number of debian jobs run is almost certainly less than 0.1% | 15:19 |
| *** pgadiya has quit IRC | 15:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove unecessecary builder scheduler thread https://review.openstack.org/359949 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove 'running' as a public method from builder https://review.openstack.org/359950 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Simplify builder start/stop methods https://review.openstack.org/359951 | 15:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Rename BuilderScheduler to NodePoolBuilder https://review.openstack.org/359952 | 15:20 |
| thcipriani | hashar: ooh I see :) | 15:21 |
| *** tonytan4ever has quit IRC | 15:21 | |
| *** matt-borland has joined #openstack-infra | 15:21 | |
| *** asettle has joined #openstack-infra | 15:21 | |
| *** armax has joined #openstack-infra | 15:22 | |
| zigo | fungi: Not what I've wrote at all. | 15:22 |
| sc68cal | sdague: https://review.openstack.org/#/c/359490/4 | 15:22 |
| *** piet has quit IRC | 15:22 | |
| *** yamahata has quit IRC | 15:23 | |
| dougwig | devstack cores, spoke with sc68cal, and he recommended the approach here for fixing the osic resets: https://review.openstack.org/#/c/359490/ | 15:23 |
| eantyshev | Hello! I think https://review.openstack.org/238988 needs a final push, it's a rather important Zuul patch which many people already use in 3rd party CIs | 15:24 |
| *** tonytan4ever has joined #openstack-infra | 15:24 | |
| fungi | zigo: the ready nodes counts in that graph are misleading. it would make more sense to scale those by the amount of time spend in that state | 15:24 |
| *** eantyshev has left #openstack-infra | 15:24 | |
| *** salv-orlando has quit IRC | 15:25 | |
| *** david-lyle_ has joined #openstack-infra | 15:25 | |
| *** sdake_ has quit IRC | 15:25 | |
| *** yamahata has joined #openstack-infra | 15:25 | |
| fungi | we do spike up over 100 xenial or trusty nodes at times, but they sit ready for only a few seconds. they're basically already spoken for by waiting jobs | 15:26 |
| *** david-lyle_ has quit IRC | 15:26 | |
| *** vhosakot has joined #openstack-infra | 15:26 | |
| fungi | what might be more useful for this particular issue is if we could graph the average time jobs spend waiting in a particular pipeline, arranged by node type | 15:27 |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Improve the support for checking vlan interfaces https://review.openstack.org/359961 | 15:27 |
| *** hockeynu_ has joined #openstack-infra | 15:28 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add fingerprint for cinder backup test fails https://review.openstack.org/359904 | 15:28 |
| fungi | i suspect that the average time spent waiting for jessie nodes is actually lower than for more popular node types, if only because we force nodepool to build them even when the demand ratio would indicate they're accounting for less than one node of demand across the entirety of our backlog | 15:28 |
| *** hockeynut has quit IRC | 15:29 | |
| fungi | (expressed as an overall percentage of our quota) | 15:29 |
| *** piet has joined #openstack-infra | 15:32 | |
| *** psachin has quit IRC | 15:32 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: remove vitrage-release tag rights for vitrage repos https://review.openstack.org/359965 | 15:33 |
| mordred | sc68cal: clarkb seemed to think that there is also a second issue | 15:33 |
| mordred | sc68cal: in addition to that one | 15:34 |
| openstackgerrit | Merged openstack-infra/shade: Support more than one network in create_server https://review.openstack.org/359378 | 15:34 |
| dougwig | mordred, sc68cal: i believe that was about the default route disappearing. though mordred's change seems to have jobs not resetting at the 15 minute mark. | 15:35 |
| *** yaume_ has quit IRC | 15:35 | |
| *** krtaylor has quit IRC | 15:35 | |
| *** hockeynu_ has quit IRC | 15:36 | |
| *** sdake has joined #openstack-infra | 15:36 | |
| mordred | dougwig: oh good | 15:37 |
| *** dizquierdo has quit IRC | 15:38 | |
| rcarrillocruz | asselin: https://review.openstack.org/#/c/359918/ | 15:39 |
| openstackgerrit | Dmitry Ilyin proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 15:39 |
| lucasagomes | 5 min tops! | 15:40 |
| fungi | hah | 15:40 |
| lucasagomes | finishing addressing all comments | 15:40 |
| rcarrillocruz | i'll self approve to get past the looping failures | 15:40 |
| *** hockeynut has joined #openstack-infra | 15:40 | |
| *** sdague has quit IRC | 15:41 | |
| sc68cal | dougwig: the default v6 route disappears as a side effect of setting v6 forwarding to 1 | 15:42 |
| sc68cal | since the default route was discovered via RA | 15:42 |
| mordred | sc68cal: ah - fascinating | 15:42 |
| sc68cal | hence, accept_ra = 2 | 15:42 |
| sc68cal | sorry, I had a whole gist about this stuff from my home networking setup | 15:42 |
| mordred | and if we do that before setting forwarding to 1, it's all good | 15:42 |
| sc68cal | just thought I had more time before I had to tell people about it | 15:42 |
| fungi | oh, right-o. i forgot linux routing gets picky about accepting router announcements when it is itself configured as a router | 15:43 |
| *** andymaier_ has quit IRC | 15:43 | |
| mordred | fungi: so - on the rax nodes | 15:43 |
| fungi | er, router advertisements | 15:44 |
| dougwig | so we're likely about 6 hours from 359490,4 merging, assuming no gate restarts. that one might be worth bumping ahead in line. | 15:44 |
| fungi | mordred: yeah, figured anything useful out there? | 15:44 |
| mordred | my hunch is that because we're configuring eth0 with ipv4 static addresses, it's ignoring the RAs | 15:44 |
| fungi | mordred: entirely possible, and easy to test if you hold a node | 15:44 |
| *** tonytan_brb has joined #openstack-infra | 15:44 | |
| mordred | and that we need to iface eth0 inet6 auto | 15:45 |
| mordred | to interfaces | 15:45 |
| fungi | yep | 15:45 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: Ansible-like tool https://review.openstack.org/359831 | 15:45 |
| fungi | that's what i do on my dual-stack systems using ifupdown | 15:45 |
| fungi | i guess it would be a patch to glean | 15:46 |
| clarkb | wait mordreds change is working? I thought it was failing last night buy I may have gotten co fused it was late | 15:46 |
| clarkb | excellent news if mordreds change is sufficient | 15:47 |
| mordred | right? | 15:47 |
| mordred | clarkb: also, sc68cal's explanation of the why makes sense | 15:47 |
| clarkb | mordred: as long as we confirm it works on osic amd doesnt crash I am happy | 15:47 |
| *** tonytan4ever has quit IRC | 15:48 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Zuul has a channel, let's tell people about it https://review.openstack.org/359976 | 15:48 |
| fungi | in fact, most routers have a similar behavior (you need to jump through hoops to configure a router to accept an ra since they usually assume other routing protocols fulfill that role) | 15:48 |
| dougwig | mordred: clarkb: i did a recheck on his change to watch, noted all the v6 nodes assigned, and have been watching for resets. several have finished with success. given the previous 100% fail rate, i'd call that a win. | 15:48 |
| clarkb | maybe I was looking at the wrong change last night when I thought I saw it crash again | 15:48 |
| *** salv-orlando has joined #openstack-infra | 15:48 | |
| clarkb | dougwig: were those neutron jobs though? | 15:48 |
| *** awayne has joined #openstack-infra | 15:49 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Add check to skip bridge interfaces https://review.openstack.org/359982 | 15:49 |
| clarkb | dougwig: we need to make syre that the devstack tests that run neutron run on osic and dont lose connectivity. if that is the case then yay | 15:49 |
| *** vhosakot has quit IRC | 15:49 | |
| *** vhosakot has joined #openstack-infra | 15:49 | |
| dougwig | let me double-check | 15:49 |
| mordred | clarkb: http://logs.openstack.org/90/359490/4/check/gate-devstack-dsvm-updown/6df96ef/console.html | 15:51 |
| clarkb | also we can get that merged and it will be easy to see if the problem persists after | 15:52 |
| mordred | rcarrillocruz: awesome. the os_quota guy said we can put the code into shade | 15:53 |
| *** abregman has quit IRC | 15:53 | |
| rcarrillocruz | \o/ | 15:53 |
| rcarrillocruz | there may be some overlap, i remember ghe putting some stuff about quotas already | 15:53 |
| rcarrillocruz | but i believe it was just for a given resource, network quotas or the likes | 15:54 |
| *** Sukhdev has joined #openstack-infra | 15:54 | |
| mordred | yah | 15:54 |
| mordred | I like his approach to comprehensive quotas on a single thing | 15:54 |
| clarkb | mordred: that test doesn't appear to run the neutron command to create the ipv6-public-subnet | 15:54 |
| clarkb | which is where things were crashing for me in my paste from last night | 15:55 |
| clarkb | (which is why I thought there may be a second thing) | 15:55 |
| clarkb | http://paste.openstack.org/show/562831/ | 15:55 |
| *** mikelk has quit IRC | 15:55 | |
| *** adrian_otto has joined #openstack-infra | 15:55 | |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck: Add missing -p to mkdir on uncat cron job https://review.openstack.org/359921 | 15:56 |
| clarkb | but it doesn't appear to cause any failures itself so merging then reviewing results is likely fine. (I do think it is part of the puzzle at the very least) | 15:57 |
| *** berendt has joined #openstack-infra | 15:58 | |
| dougwig | clarkb: you're right, i was looking at an n-net job. made a bad assumption about the default in devstack changing. watching further. | 15:59 |
| Zara | rcarrillocruz: aw, thanks for the nice email reply :) | 16:00 |
| rcarrillocruz | ;-) nicely done! | 16:01 |
| *** krtaylor has joined #openstack-infra | 16:02 | |
| *** piet has quit IRC | 16:03 | |
| *** tphummel has joined #openstack-infra | 16:03 | |
| *** gyee has joined #openstack-infra | 16:03 | |
| *** vincentll has quit IRC | 16:05 | |
| clarkb | I need caffeine and foods but then I can help dig in more on the ipv6 stuff | 16:08 |
| *** ifarkas is now known as ifarkas_afk | 16:08 | |
| *** jistr|mtg is now known as jistr | 16:08 | |
| haleyb | sc68cal: i was trying to find the kernel code that would make the v6 router disappear setting forwarding=1, but agree that, at least eventually, things would break | 16:11 |
| *** yamahata has quit IRC | 16:11 | |
| *** raunak has joined #openstack-infra | 16:11 | |
| dougwig | clarkb: i added 359996 to fire mordred's patch against more neutron jobs. | 16:12 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Update zuul-env on job nodes https://review.openstack.org/359352 | 16:12 |
| fungi | jeblair: ^ reworked per your suggestion | 16:12 |
| fungi | AJaeger: ^ | 16:13 |
| jeblair | yay i made a good suggestion! (?) | 16:13 |
| fungi | i certainly thought so | 16:13 |
| haleyb | sc68cal: found the code - rt6_purge_dflt_routers() - and it removes default routes learned from RA's when forwarding is enabled and accept_ra is not 2 | 16:15 |
| pabelanger | hashar: There was a quota mismatch with our project in OVH. While I don't know the fix, they did "re-sync our tenant". Not that it helps with the fix. | 16:15 |
| *** _nadya_ has quit IRC | 16:15 | |
| *** Julien-zte has quit IRC | 16:15 | |
| *** matthewbodkin has quit IRC | 16:15 | |
| haleyb | dougwig, clarkb ^^ see my comment there, think the accept_ra change will do it | 16:15 |
| haleyb | not that it wasn't obvious already | 16:15 |
| hashar | pabelanger: I guess there is a glitch in how the quota is tracked and somehow it is not always updated on instance deletion / spawn error | 16:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Update zuul-env on job nodes https://review.openstack.org/359352 | 16:15 |
| fungi | correcting small typo | 16:16 |
| *** shashank_hegde has joined #openstack-infra | 16:16 | |
| hashar | pabelanger: so with time the quota keeps pilling up. I guess they manually updated the quota in whatever data backend has the quotas. | 16:16 |
| jeblair | fungi: something something zuul something | 16:16 |
| fungi | indeed | 16:16 |
| clarkb | haleyb: I think it is part of the puzzle but not convinced it solves it yet | 16:16 |
| hashar | thcipriani: got the answer. OVH did "re-sync our tenant" which I guess is manually adjusting the bad quota | 16:16 |
| fungi | lots of brainfog between /usr/zuul-env and /opt/zuul | 16:16 |
| hashar | pabelanger: thank you ! | 16:16 |
| clarkb | need to see a non crashed run on osic for a neutron job first | 16:16 |
| *** Apoorva has joined #openstack-infra | 16:17 | |
| *** Apoorva has quit IRC | 16:17 | |
| pabelanger | zigo: Yes, you do need to wait, however it still is shorter then jobs requiring ubuntu-trusty / ubuntu-xenial. For every min-ready node we bring online, we remove that node from the pool of jobs. And since debian-jessie is only used between openstack-infra and deb-packaging team, I still believe 3 is a good number. | 16:17 |
| *** Apoorva has joined #openstack-infra | 16:17 | |
| pabelanger | zigo: even if we bump it to 5, you are going to have to wait for nodes to come online. | 16:17 |
| haleyb | clarkb: agreed. i'm assuming it will take a while to get that patch in place to then run another check on-top in the osic cloud if my infra 101 learning is correct | 16:19 |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config: Move cinder multinode grenade job to check https://review.openstack.org/359275 | 16:19 |
| pabelanger | hashar: yes, I don't know why that happens, just that it does. Since we increased OVH back to the original, we haven't had the problem yet. From what I understand, it was / is a common enough problem that people knew that was the issue before we asked OVH to look into it | 16:19 |
| hashar | pabelanger: yeah I got the same issue with OpenStack Liberty. Whenever some instances deletion / spawn screw up for some reason | 16:20 |
| clarkb | haleyb: no its self testing so its tested immediately. we just need a run on osic | 16:20 |
| hashar | pabelanger: I suspect something in nova or whatever glitches and does not necessarily properly lower the quota when an instance is disposed. Or at least it does not periodically refresh it with actual values | 16:21 |
| hashar | pabelanger: manually editing the database would fix the dispredancy for sure and I guess that is what they ended up doing | 16:21 |
| *** matrohon has quit IRC | 16:22 | |
| dougwig | clarkb: the dependent review will likely hit osic at least once. | 16:23 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories https://review.openstack.org/359895 | 16:23 |
| pabelanger | clarkb: fungi: mordred: just catching up on 359490, are we planning on promoting? or waiting until we know it works in check? | 16:25 |
| clarkb | dougwig: haleyb telnet://2001:4800:1ae1:18:f816:3eff:feee:1908:19885 is a run against 359490 | 16:25 |
| dougwig | clarkb: oh, sweet. recorded, and watching. | 16:26 |
| *** Na3iL has quit IRC | 16:27 | |
| jeblair | pabelanger, clarkb, dougwig: i think we should promote it once the run clarkb is looking at passes the mark | 16:27 |
| clarkb | I am tailing its devstack log now too since that was more useful last night | 16:27 |
| *** vhosakot has quit IRC | 16:27 | |
| *** florianf has quit IRC | 16:27 | |
| dougwig | jeblair: +1 | 16:27 |
| clarkb | jeblair: pabelanger dougwig ya if this job gets past the neutron setup in devstack I think we can promote | 16:27 |
| *** vhosakot has joined #openstack-infra | 16:28 | |
| pabelanger | thanks, I figured that was the reason | 16:28 |
| *** hpe-hj has quit IRC | 16:28 | |
| *** jpich has quit IRC | 16:29 | |
| clarkb | right now its doing keystone things in devstack so probably about 5 minutes away? I forget how long it took | 16:29 |
| dougwig | clarkb: 14-15 mins from launch. | 16:29 |
| dougwig | but that might include a timeout, so i expect you'll know soonest. | 16:29 |
| *** rcernin has quit IRC | 16:29 | |
| clarkb | I am also going to hold the instance as a precaution | 16:29 |
| clarkb | 3831927 is held | 16:30 |
| *** Swami has quit IRC | 16:30 | |
| pabelanger | apuimedo: currently we don't have python35 on our centos-7 nodes. | 16:31 |
| *** claudiub has quit IRC | 16:31 | |
| apuimedo | pabelanger: is it for a lacking software collection? | 16:31 |
| clarkb | we don't install anything from software collections (I think thats the name for them) | 16:31 |
| pabelanger | apetrich: that said, there has been some talk about using python3.5 from software collections | 16:31 |
| apuimedo | Or where whould the python35 come from in EL7? | 16:31 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck: Remove unnecessary rm from uncat cron job https://review.openstack.org/359927 | 16:32 |
| *** hockeynu_ has joined #openstack-infra | 16:32 | |
| *** yolanda has quit IRC | 16:32 | |
| clarkb | epel only has 3.3 or 3.4 so it would have to be software collections if we needed taht for some reason. I think I am missing some context though | 16:32 |
| clarkb | is there some reason that the default python (python2.7) won't work there? | 16:32 |
| zaro | ianw: reviewed https://review.openstack.org/359683 i think something else is amiss. | 16:32 |
| pabelanger | clarkb: apuimedo: asked in a PM about using python3.5 on nodes, I redirected here | 16:32 |
| apuimedo | clarkb: well, we're now upstreaming the kuryr prototype | 16:33 |
| clarkb | we have python3.5 on xenial instances | 16:33 |
| apuimedo | for kubernetes | 16:33 |
| AJaeger | thanks, fungi for the zuul-env updates! | 16:33 |
| apuimedo | and I know that xenial is fine | 16:33 |
| apuimedo | I wanted to know how far are we for the el7 nodes | 16:33 |
| pabelanger | when I did ask Red Hat about python3.5 and centos-7, there wasn't a clear plan yet | 16:33 |
| apuimedo | since we'd really much prefer to do 3.5+ | 16:33 |
| *** hashar has quit IRC | 16:33 | |
| *** Hal1 has joined #openstack-infra | 16:33 | |
| *** Hal has quit IRC | 16:33 | |
| apuimedo | pabelanger: that's what I was afraid of | 16:34 |
| pabelanger | but software collections was suggested as a possible options | 16:34 |
| clarkb | I think my preference for that sort of setup would be that your jobs installed python3.5 from software collections if you need ti on centos for some reason | 16:34 |
| fungi | AJaeger: it will take a bit for testing to finish on 359352, but i'll self-approve with only one core vote on it if necessary at that point so it doesn't linger | 16:34 |
| clarkb | I don't think infra should be managing that | 16:34 |
| *** hashar has joined #openstack-infra | 16:34 | |
| apuimedo | pabelanger: would that be a first in RHOSP? To use SCs? | 16:34 |
| *** hashar has quit IRC | 16:34 | |
| clarkb | (we provide a valid python3.5 platform using available system packages, if you want to branch outside of that its fine but I don't think we will bake it in) | 16:34 |
| *** kaisers_ has quit IRC | 16:34 | |
| pabelanger | clarkb: yes, agreed. If apuimedo wants python3.5 for kuryr, adding software collections would be my suggestion | 16:34 |
| *** tqtran has joined #openstack-infra | 16:35 | |
| *** hockeynut has quit IRC | 16:35 | |
| *** tesseract- has quit IRC | 16:35 | |
| clarkb | though more and more with bindep we don't technically provide any base python | 16:35 |
| clarkb | (though due to image build deps we technically do) | 16:35 |
| pabelanger | apuimedo: not RHOSP, we don't manage that. But you could it upstream in your jobs | 16:35 |
| clarkb | dougwig: its running neutron setup now we should know soon | 16:35 |
| dougwig | clarkb: the executioner is climbing the stairs to the platform.... | 16:36 |
| pabelanger | clarkb: ya, I've been meaning to try software collections on centos-7 for a python35 job. never enough time | 16:36 |
| clarkb | dougwig: I think it just died creating that public subnet | 16:37 |
| dougwig | clarkb: my ping just froze | 16:37 |
| clarkb | I do knote it created the private one just fine about 20 seconds before hand | 16:37 |
| apuimedo | I was under the impression that SCs were at a very primitive stage for 3.5 | 16:37 |
| *** yamamoto has quit IRC | 16:37 | |
| apuimedo | in terms of available libs | 16:37 |
| dougwig | sc68cal: haleyb: ^^ anything on the node that will help us? | 16:38 |
| clarkb | dougwig: ya so I don't think this is a complete solution. But likely still a required piece. Something else in the subnet creation is tripping us up | 16:38 |
| *** ganesan has joined #openstack-infra | 16:38 | |
| pabelanger | apuimedo: possible? Not sure. At this point, nobody really know. python35 support for RedHat still work in progress | 16:39 |
| clarkb | it should be trivial to reproduce this right? has anyone from neutron/devstack tried yet? | 16:39 |
| *** tqtran has quit IRC | 16:39 | |
| *** Na3iL has joined #openstack-infra | 16:39 | |
| clarkb | I guess it requires a global ipv6 addr which isn't necessarily something everyone has handy | 16:39 |
| apuimedo | understood | 16:39 |
| clarkb | dougwig: the instance is held by nodepool so we can hop on it if we need to. THe tricky thing is devstack doesn't stop running when this happens | 16:39 |
| *** sdague has joined #openstack-infra | 16:40 | |
| dougwig | clarkb: yes, it should be. and no, i'm not sure anyone has a v6 only setup ready to roll, but if this drags out any longer, switching to reproing elsewhere will be a better plan. | 16:40 |
| clarkb | dougwig: so it and or tempest and friends can continue to change state on the host which might make debugging a little tricky. Its possible we could rig up a devstack change on top of mordreds that just stops doing anything after that point | 16:40 |
| sdague | can you repromote - 359939,2 - failed on unrelated tests | 16:40 |
| clarkb | make public ipv6 subnet and exit 1 to kill the rest of the job | 16:40 |
| sdague | fungi: ^^^ | 16:40 |
| fungi | sdague: on it | 16:41 |
| sdague | or force merge, it passed all the important stuff | 16:41 |
| fungi | sdague: can do--just a sec | 16:41 |
| sdague | and it would let people get back to work quick | 16:41 |
| sdague | the osic ipv6 delay also did trip it there | 16:42 |
| fungi | yeah, hopefully we're really close to pinning that one down now | 16:42 |
| clarkb | also this appears to be a legitimate neutron issue... not really anything osic is doing wrong | 16:43 |
| ganesan | I am getting Jenkins expection on the nodepool.log "JenkinsException: Error in request. Possibly authentication failed [500]: Server Error" | 16:43 |
| fungi | for some reason my gerrit superpowers are being a pain | 16:43 |
| sdague | so, one of the issues with osic ipv6 only nodes... that means the console doesn't work unless you as a user have ipv6 | 16:43 |
| ganesan | nodepool.log -- http://paste.openstack.org/show/563098/ | 16:43 |
| clarkb | sdague: that is correct | 16:44 |
| fungi | sdague: or something with ipv6 connectivity you can bounce through, or a v6 tunnel | 16:44 |
| jeblair | or a phone | 16:44 |
| fungi | hah, yep | 16:44 |
| fungi | phones seem to all be ipv6 these days | 16:44 |
| fungi | because of sheer volume | 16:44 |
| openstackgerrit | Merged openstack-infra/project-config: TripleO scenario001 experimental job https://review.openstack.org/356675 | 16:44 |
| sdague | a phone with telnet client | 16:44 |
| clarkb | I have been using my irc screen box in rackspace. fungi has an HE tunnel. Most/all comcast customers should have native ipv6 | 16:44 |
| *** AnarchyAo has joined #openstack-infra | 16:44 | |
| jeblair | sdague: i you know you have one ;) | 16:44 |
| sdague | clarkb: that's still a small percentage of our devs | 16:45 |
| fungi | and, of course, people who live in first world countries | 16:45 |
| fungi | someday maybe i'll move to one | 16:45 |
| *** derekh has quit IRC | 16:45 | |
| *** hockeynut has joined #openstack-infra | 16:45 | |
| cloudnull | at home i use an HE tunnel works really well | 16:45 |
| *** hockeynu_ has quit IRC | 16:45 | |
| clarkb | much of asia is all ipv6 native and has been for a long time | 16:45 |
| clarkb | that was a neat part of the tokyo summit | 16:45 |
| clarkb | yes, not everyone will have it working. But I think a significant number of people can make it work with minimal effort or already ahve it | 16:46 |
| dougwig | clarkb: asking for an osic node now, so i can jumpstart the setup phase of repro. | 16:46 |
| clarkb | and osic has been gracious enough to provide a lot of resources to us | 16:46 |
| *** shashank_hegde has quit IRC | 16:46 | |
| *** roxanagh_ has joined #openstack-infra | 16:47 | |
| *** jaosorior has quit IRC | 16:47 | |
| JayF | AJaeger: for https://review.openstack.org/#/c/356797/ you said all I needed was python-jobs template, but ironic-lib already had the python-jobs template and wasn't running a docs job. I'm curious what piece I'm misssing? | 16:47 |
| sdague | clarkb: that's fine, I'm just saying it moves burden back onto devs that already have to figure out a lot of things. | 16:47 |
| clarkb | dougwig: you should also be able to reproduce on a rax instance too fwiw | 16:48 |
| AJaeger | JayF: let me check again... | 16:48 |
| clarkb | dougwig: we don't notice it in our rax instances because we connect to them via ipv4 so ipv6 going away isn't noticed but it should still happen there too | 16:48 |
| dougwig | clarkb: heh, since i'm sitting at rax hq, by chance, that'd be a good way to go too. | 16:48 |
| *** Sukhdev has quit IRC | 16:48 | |
| fungi | sdague: also the websockets work, once it lands, will ease that since we can have a non-v6 web proxy serving it | 16:48 |
| sdague | ok, with that force merged, then promoting the ipv6 work around is the next order of business? | 16:48 |
| AJaeger | JayF: you have python34-jobs and python35-jobs but *no* python-jobs | 16:49 |
| sdague | fungi: yeh, this was mostly, can we get webui for this prioritized | 16:49 |
| EmilienM | hey infra, here's an easy patch to approve: https://review.openstack.org/#/c/359539/ - thanks! | 16:49 |
| clarkb | sdague: no because the ipv6 workaround does not work. There is somethign else neutron is doing to explode ipv6 when creating the public ipv6 subnet | 16:49 |
| JayF | AJaeger: line #3479 on https://review.openstack.org/#/c/356797/4/jenkins/jobs/projects.yaml says "python-jobs" | 16:49 |
| clarkb | sdague: so we need to figure that out first (sounds like dougwig is working on getting it reproduced) | 16:49 |
| sdague | clarkb: ok | 16:49 |
| AJaeger | JayF: projects.yaml != zuul | 16:49 |
| JayF | AJaeger: OH! in zuul layout | 16:49 |
| *** AnarchyAo has quit IRC | 16:49 | |
| AJaeger | JayF: Those are two complete different beasts. I'm talking zuul | 16:49 |
| *** Hal1 has quit IRC | 16:50 | |
| JayF | AJaeger: I got it now, will update, ty | 16:50 |
| *** asettle has quit IRC | 16:50 | |
| *** mtanino has joined #openstack-infra | 16:50 | |
| *** Hal has joined #openstack-infra | 16:50 | |
| clarkb | dougwig: also dreamhost and vexxhost provide native ipv6 to instances if you want more options :) | 16:50 |
| sdague | ok, sc68cal is sitting across from me | 16:50 |
| *** asettle has joined #openstack-infra | 16:50 | |
| *** jordanP has quit IRC | 16:51 | |
| AJaeger | bbl | 16:51 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 16:52 |
| *** yamahata has joined #openstack-infra | 16:52 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 16:52 |
| *** DrifterZA has quit IRC | 16:52 | |
| haleyb | clarkb: now i'm not so sure setting net.ipv6.conf.all.accept_ra=2 is doing what we think, not sure it's propagating to all the interfaces. the forwarding setting has special code to handle that case | 16:53 |
| *** csomerville has joined #openstack-infra | 16:54 | |
| *** agordeev has joined #openstack-infra | 16:55 | |
| *** asettle has quit IRC | 16:55 | |
| sc68cal | haleyb: we have code in neutron/agent/linux/interface.py that sets it for an interface - ['sysctl', '-w', 'net.ipv6.conf.%s.accept_ra=2' % dev_name]) | 16:55 |
| *** dtantsur is now known as dtantsur|afk | 16:55 | |
| haleyb | sc68cal: right, but does it do it for the public interface ? | 16:55 |
| dougwig | sc68cal: haleyb: no, it does not. | 16:56 |
| sc68cal | haleyb: nope, just for the namespace | 16:56 |
| *** Shrews has quit IRC | 16:56 | |
| haleyb | the "all" setting in ipv6 isn't like the ipv4 equivalents from waht i remember | 16:56 |
| *** phschwartz has quit IRC | 16:56 | |
| *** mordred has quit IRC | 16:56 | |
| dougwig | haleyb: because ipv6 wants to be special? | 16:56 |
| *** tonytan_brb has quit IRC | 16:57 | |
| haleyb | dougwig: it was added later, and noone noticed, so was too late to change without breaking things | 16:57 |
| *** tonytan4ever has joined #openstack-infra | 16:57 | |
| haleyb | again, that's if my mind is remembering from the discussion on the netdev list years ago | 16:58 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add query for get me a network fails https://review.openstack.org/359960 | 16:58 |
| haleyb | we should try adding a sysctl for that public interface, which i think we pull a few lines below | 16:58 |
| *** phschwartz has joined #openstack-infra | 16:58 | |
| jeblair | haleyb: how does 'default' factor into this? | 16:58 |
| clarkb | haleyb: would writing to default instead of all change anything? or do we need to do it for each interface we want specifically? | 16:58 |
| clarkb | jeblair: jinx | 16:58 |
| sc68cal | haleyb: agree, let's find the public iface and sysctl it | 16:59 |
| *** Shrews has joined #openstack-infra | 16:59 | |
| haleyb | clarkb: i think we need to do it individually, can't hurt to try. i can tweak the patch i think | 16:59 |
| *** mordred has joined #openstack-infra | 16:59 | |
| clarkb | also any idea why the subnet creation appears to be when we fail? is that the point in time where neutron will set forwarding on the interface? | 17:00 |
| haleyb | yes, when the interface is added to the router i believe | 17:00 |
| sc68cal | we do a couple route commands in _neutron_configure_router_v4, a couple lines up before the public v4 net is creatred | 17:01 |
| sc68cal | ahh, no we don't | 17:02 |
| *** sputnik13 has joined #openstack-infra | 17:03 | |
| *** AnarchyAo has joined #openstack-infra | 17:04 | |
| *** jerryz has joined #openstack-infra | 17:06 | |
| haleyb | i just pushed an update to https://review.openstack.org/#/c/359490/ | 17:07 |
| pabelanger | jeblair: did you by chance see my comment about nodepool not getting data from gearman (manager_name that zuul-launcher sets)? | 17:08 |
| *** berendt has quit IRC | 17:08 | |
| *** martinkopec has quit IRC | 17:08 | |
| *** greg-g has left #openstack-infra | 17:09 | |
| *** hockeynut has quit IRC | 17:10 | |
| sdague | so... I guess the question, can we get osic ipv6 only out of the test path for now so the debugging can happen orthoginal to people trying to land unrelated code? | 17:10 |
| fungi | sdague: we can, though it will cut our test capacity by half, and delay additional capacity we're probably bringing online in the next day or so | 17:11 |
| openstackgerrit | Merged openstack-infra/storyboard: Put the logic for hiding private things in storyboard/db/api/base.py https://review.openstack.org/359894 | 17:11 |
| sdague | fungi: ok, but if that capacity has the same issue, it means that the gate for anything running neutron gets extra sluggish | 17:11 |
| *** amitgandhinz has quit IRC | 17:12 | |
| pabelanger | mgagne_: about mtl01, is there a new set of credentials for that? | 17:12 |
| fungi | sdague: yep. it's a balancing act. i don't know whether rescheduling neutron jobs are decreasing our throughput by half, but reducing our quota by half definitely will | 17:12 |
| *** hockeynut has joined #openstack-infra | 17:12 | |
| *** jcoufal has joined #openstack-infra | 17:12 | |
| *** amitgandhinz has joined #openstack-infra | 17:12 | |
| pabelanger | fungi: we seem to be having some issues with rax-iad, launch node timeouts for ssh for example. Do you have a contact on IRC to help troubleshoot? | 17:13 |
| fungi | apparently server capacity is a lot easier for providers to give us than ipv4 addresses | 17:13 |
| sdague | fungi: sure. What if we moved it out of the dsvm class? | 17:13 |
| fungi | pabelanger: i usually end up just opening a trouble ticket with rackspace and waiting for them to eventually hand it off to someone not in their first tier of support | 17:14 |
| *** yolanda has joined #openstack-infra | 17:14 | |
| fungi | sdague: i don't know what that means | 17:14 |
| fungi | if you mean not run jobs with dsvm in the name on osic, we ceased having separate node types for dsvm jobs back in march or so | 17:15 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 17:15 |
| sdague | oh, right, we have the same node type between dsvm and unit tests now | 17:15 |
| sdague | right... | 17:15 |
| sdague | it just would be really nice to not be rushing to solve this issue while people are also trying to fix other release issues :( | 17:16 |
| haleyb | clarkb: i have to take off for a bit, but will check back regarding that patch, don't know how soon you'll know if connectivity is ok | 17:16 |
| fungi | sdague: yep. so anyway, we can go back to all ipv4 nodes. it just may take longer to run jobs | 17:16 |
| clarkb | haleyb: as soon as osic runs the neutron job for that patch and passes the neutron ipv6 subnet setup | 17:16 |
| fungi | sdague: which would be a shame if the ipv6 routing issue turns out to be quick-ish to fix | 17:16 |
| sc68cal | haleyb: do you think the sysctl for setting ipv6 forwarding to 1 has any side effects, like clearing the sysctls for accept_ra? | 17:17 |
| sdague | fungi: right, but then we can not grind while we get the fix, and once we know we have the fix put things back into operation. | 17:17 |
| *** bethwhite_ has quit IRC | 17:18 | |
| sc68cal | since we set accept_ra first, then forwarding? I have zero data to back it up, just spitballing | 17:18 |
| fungi | sdague: well, it's not failing jobs, just causing them to get rescheduled so delaying results/merging for some projects/changes and wasting some of the additional resources we have | 17:18 |
| fungi | i'm unconvinced things will get appreciably better if we stop using those additional resources, though perhaps they won't get any worse | 17:19 |
| sdague | fungi: ok | 17:19 |
| *** piet has joined #openstack-infra | 17:20 | |
| fungi | i agree it's an option worth entertaining | 17:20 |
| sdague | the impact that I see is it puts delay on landing gate fixes, which make things pass quicker in check (reduce our rechecks) | 17:20 |
| sdague | so the fix path being slow means harder to dig out of issues once we discover and address them | 17:21 |
| sdague | but, I'm also about to drop and go back to the conference as bat is running low. | 17:22 |
| *** ganesan has quit IRC | 17:22 | |
| *** piet has quit IRC | 17:22 | |
| cloudnull | using that same logic, isn't this an issue that has been "discovered" which can now be addressed? | 17:22 |
| * sc68cal is in same boat | 17:22 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:23 | |
| pabelanger | While I understand the need to roll back to ipv4, I do like the idea of iterating forward on ipv6. Especially since it gives us valid test failures | 17:23 |
| *** rcernin has joined #openstack-infra | 17:23 | |
| pabelanger | I would much rather remove the requeue job logic in zuul-launcher then move back to ipv4 :) | 17:23 |
| phschwartz | What is the best way to get the list of projects/reviews currently running in zuul? | 17:23 |
| sc68cal | iterating forward on ipv6 is a good thing, but not when it's causing serious disruption | 17:23 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 17:24 |
| clarkb | phschwartz: grab the status json file (the same thing the fancy status page uses to render the page) | 17:24 |
| sc68cal | and we have people scrambling | 17:24 |
| cloudnull | also asking because the OSIC has more room to give (more in cloud1 and cloud8 coming online soon) but if it's not helping then we'll hold off for now. | 17:24 |
| fungi | cloudnull: well, one of the tools at our disposal when we discover something we're testing is broken is to stop testing it until we work out the fix. i guess in that vein options are to stop testing the things in neutron which trigger this, or stop testing on ipv4 | 17:24 |
| *** hashar has joined #openstack-infra | 17:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Pass cacert param for admin-infracloud https://review.openstack.org/359934 | 17:25 |
| pabelanger | Right I think we also have another region in internap too | 17:25 |
| pabelanger | I'm on stand-by to launch cloud8 once that cloud is in place | 17:26 |
| *** hashar is now known as hasharAway | 17:26 | |
| *** pt_15 has quit IRC | 17:26 | |
| cloudnull | is internap using v6 too ? | 17:26 |
| clarkb | my preference would be to run with osic as is since all jobs that are not devstacky with neutron should be fine (except maybe kolla and osa?) and just get neutron fixed | 17:26 |
| cloudnull | OSA is happy :) | 17:27 |
| clarkb | neutron should support this and until someone has reproduced locally I am afraid that we will only set ourselves back by no longer testing it | 17:27 |
| pabelanger | cloudnull: ipv4 | 17:27 |
| clarkb | if neutron comes back and says "we have this reproduced and are workong on a fix" I can reconsider | 17:27 |
| pabelanger | clarkb: I agree with that statement | 17:27 |
| dougwig | i think "we are taking this seriously and not ignoring it" should be enough. the pressure has served its purpose. | 17:28 |
| *** tqtran has joined #openstack-infra | 17:28 | |
| cloudnull | so maybe we can remove the | 17:28 |
| cloudnull | dsvm header from the osa jobs | 17:28 |
| *** electrofelix has quit IRC | 17:29 | |
| cloudnull | which should allow it to continue running on OSIC? | 17:29 |
| cloudnull | or is it not that simple ? | 17:29 |
| clarkb | cloudnull: ya I Think its redundant at this point if osa isn't devstack gating (since we use the same image everywhere now) | 17:29 |
| *** fguillot has joined #openstack-infra | 17:29 | |
| *** kzaitsev_mb has quit IRC | 17:29 | |
| clarkb | cloudnull: well if we were to turn off osic temporarily it would likely be global | 17:29 |
| *** vhosakot_ has joined #openstack-infra | 17:29 | |
| clarkb | hence my apprehension when most jobs continue to work there | 17:29 |
| fungi | cloudnull: we can effectively remove "dsvm" from all jobs if we want. these days it mostly just means "this uses devstack-gate" but is purely cosmetic | 17:29 |
| cloudnull | ok | 17:29 |
| cloudnull | clarkb: :'( | 17:30 |
| dougwig | can you remove (neutron.*dsvm|dsvm.*neutron) ? | 17:30 |
| pabelanger | we'd have to drop the ubuntu-trusty / ubuntu-xenial job | 17:30 |
| pabelanger | err, label | 17:30 |
| *** csomerville has quit IRC | 17:30 | |
| clarkb | pabelanger: ya we would have to make new labels just for neutron jobs that osic doesn't provide | 17:30 |
| clarkb | it would likely be more work than just fixing the issue | 17:30 |
| clarkb | (because glance image uplaods are slow) | 17:30 |
| pabelanger | yes | 17:30 |
| fungi | "dsvm" used to mean that the job needed devstack-specific nodes, but we no longer have devstack and non-devstack nodes... one node type (of each distro/release) to rule them all | 17:30 |
| dougwig | we need to get this sorted before next week's feature freeze no matter what, IMO, or we're all doomed. | 17:30 |
| sdague | clarkb: the issue with that approach is what you're actually saying is that during release everything on any project that uses neutron for default networking on a job should be delayed until this is addressed. | 17:31 |
| fungi | dougwig: yes, i have doubts that we'll get the coming volume of change activity through without the additional capacity v6-only providers are offering us | 17:31 |
| clarkb | sdague: yes but shouldn't this be addressed in the next hour or two? | 17:31 |
| sdague | dougwig: right, the issue is how many other unrelated features in non neutron projects miss freeze for it | 17:31 |
| AJaeger | project-config cores, could you review a change for check-osc that I like to have in for next image build with mor debugging output: https://review.openstack.org/#/c/359937/1 - and a change to add os-api-ref under release team control: https://review.openstack.org/359916, please? Both have already one +2. | 17:32 |
| *** vhosakot has quit IRC | 17:32 | |
| *** senk has joined #openstack-infra | 17:32 | |
| sdague | clarkb: I don't know | 17:32 |
| pabelanger | clarkb: I feel we know what the issue is right? We are just waiting on patches at this point right? | 17:32 |
| pabelanger | also, right | 17:33 |
| clarkb | pabelanger: I think we have a sense of the issue, we are working on nailing down a fix that actually addresses it | 17:33 |
| sdague | the first spector of this was raised over a week ago with the mysterious restarting jobs that I noticed. If you think we're within 1 or 2 hours of a fix, cool. | 17:33 |
| sdague | as a non ipv6 expert, I do not know if that is true or not | 17:33 |
| clarkb | sdague: yes but no one debugged it at that point | 17:33 |
| clarkb | they handed it off to infra and said "not our issue" | 17:33 |
| clarkb | when in reality this is a neutron bug that should be trivially rerpduceable if anyone tried | 17:33 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 17:33 |
| dougwig | heh, it'll take me half that hour just to get this new node stacked. | 17:34 |
| cloudnull | dougwig: sdague: if you need resources on the OSIC to test things I'd be happy to spin things for you. | 17:34 |
| clarkb | dougwig: it should be as easy as nova boot && wget reproduce.sh && sudo ./reproduce.sh assuming you have a cloud that meets teh requirements | 17:34 |
| clarkb | reproduce.sh is pretty awesome | 17:34 |
| AJaeger | JayF: +2! | 17:35 |
| dougwig | my first attempts at getting a node didn't work, just got a rax node live. | 17:35 |
| *** lucasagomes is now known as lucas-dinner | 17:35 | |
| JayF | AJaeger: tyvm for the handholding :) | 17:35 |
| AJaeger | np | 17:35 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add query for cinder unit test OOM https://review.openstack.org/360023 | 17:36 |
| *** vhosakot_ has quit IRC | 17:37 | |
| pabelanger | dougwig: send along your public SSH key, I can setup a test node in osic-cloud1 | 17:37 |
| sc68cal | clarkb: we don't know if this is a neutron bug | 17:37 |
| clarkb | pabelanger: you'll need to set up two fwiw | 17:37 |
| *** vhosakot has joined #openstack-infra | 17:37 | |
| clarkb | pabelanger: one to be the proxy/bastion and the other to break | 17:37 |
| pabelanger | clarkb: sure, I can do that | 17:38 |
| *** yamamoto has joined #openstack-infra | 17:38 | |
| clarkb | sc68cal: it seems pretty definitive to me that creating an ipv6 subnet on neutron breaks the hosts ipv6 routes. But you are correct we haven't nailed down the exact fix/cause so could be something else | 17:38 |
| pabelanger | clarkb: trusty right? | 17:39 |
| clarkb | pabelanger: xenial or trusty, both exhibit the same behavior | 17:39 |
| AJaeger | project-config cores, a review to fix trove's tempest job would be nice as well: https://review.openstack.org/#/c/356999 | 17:40 |
| AJaeger | anteaya: thanks for reviewing! | 17:40 |
| pabelanger | clarkb: okay | 17:40 |
| *** jheroux has joined #openstack-infra | 17:42 | |
| *** sdague has quit IRC | 17:42 | |
| *** shashank_hegde has joined #openstack-infra | 17:43 | |
| pabelanger | clarkb: k, dougwig has 2 instances in osic-cloud1 | 17:43 |
| anteaya | AJaeger: welcome, thanks for doing the heavy lifting | 17:43 |
| AJaeger | ;) | 17:44 |
| *** rbrndt has quit IRC | 17:44 | |
| *** ihrachys has quit IRC | 17:44 | |
| *** yamamoto has quit IRC | 17:44 | |
| *** gyee has quit IRC | 17:47 | |
| AJaeger | pabelanger, jeblair, fungi: https://review.openstack.org/#/c/354098/ proposes a new periodic-frequent pipeline just for tripleo. didn't we want to move tripleo out of OpenStack CI instead of doing another specific change for them? | 17:47 |
| fungi | AJaeger: i think the ml discussion is ongoing | 17:48 |
| anteaya | AJaeger: I also wondered if we have any thoughts on patches to create bespoke pipelines | 17:48 |
| fungi | though i also don't see it coming to a resolution unless we force the issue | 17:48 |
| AJaeger | So, -2 the patch - or ignore it for now? | 17:48 |
| AJaeger | anteaya: could you review https://review.openstack.org/359289 as well, please? | 17:49 |
| fungi | AJaeger: i would just skip over it for now, noting that it adds additional complexity perhaps unnecessary pending outcome of the third-party discussion on the ml | 17:49 |
| dougwig | clarkb: what is this reproduce.sh script you're referring to? | 17:49 |
| *** tosky has quit IRC | 17:49 | |
| pabelanger | AJaeger: I am pretty sure a new pipeline is not needed, this could be achieved using the post pipeline with some changes to tripleo-ci | 17:50 |
| clarkb | dougwig: every devstack gate run writes out a reproduce.sh script that is copied into the logs | 17:50 |
| AJaeger | dougwig: it's in the log folder | 17:50 |
| AJaeger | dougwig: do you have a change with logs? | 17:50 |
| clarkb | dougwig: shouod make it easy to reproduce and devstack gate test run | 17:50 |
| dougwig | i have a log tarball from mordred that i can use. | 17:51 |
| pabelanger | AJaeger: Let me see what tripleo is trying to do and will get back to you | 17:51 |
| clarkb | any neutron tempest full run reptoduce.sh should work | 17:51 |
| *** piet has joined #openstack-infra | 17:51 | |
| clarkb | well I guess not one from a failing run | 17:51 |
| *** piet has quit IRC | 17:51 | |
| anteaya | AJaeger: okay my shell isn't great, but I feel the risk is low changing the location of the code whilst retaining the code | 17:51 |
| AJaeger | pabelanger: thanks | 17:52 |
| *** sdague has joined #openstack-infra | 17:52 | |
| *** piet has joined #openstack-infra | 17:52 | |
| AJaeger | anteaya: thanks. | 17:53 |
| *** rbrndt has joined #openstack-infra | 17:53 | |
| AJaeger | pabelanger: I commented and put a WIP on it - and wait for more discussion and comments | 17:53 |
| haleyb | sc68cal: yes, setting forwarding=1 will remove the default route if accept_ra is not 2, i had found the kernel code | 17:54 |
| anteaya | AJaeger: welcome | 17:54 |
| dougwig | haleyb: sc68cal: you're about to reset on the gate-grenade-dsvm-neutron job on your version of mordred's patch. the instance is down. | 17:55 |
| *** rbrndt has quit IRC | 17:55 | |
| *** ddieterly is now known as ddieterly[away] | 17:55 | |
| haleyb | dougwig: so it didn't work, sigh | 17:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add the exception string when a osc command fails to load https://review.openstack.org/359937 | 17:56 |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Move container job to check https://review.openstack.org/356421 | 17:57 |
| sc68cal | haleyb: ok cool | 17:57 |
| openstackgerrit | Merged openstack-infra/project-config: Move os-api-ref to release team https://review.openstack.org/359916 | 17:57 |
| openstackgerrit | Merged openstack-infra/project-config: Update trove job to install trove tempest plugin https://review.openstack.org/356999 | 17:57 |
| dougwig | haleyb: sc68cal: i'm trying to repro with a non-CI stack. what's your plan? | 17:58 |
| sc68cal | dougwig: I will be on the train tonight back to my home, I'll also work to reproduce tomorrow | 17:59 |
| *** Swami has joined #openstack-infra | 17:59 | |
| sc68cal | I have some nodes that I can make look like OSIC env, then run | 17:59 |
| *** guilherme has joined #openstack-infra | 18:00 | |
| haleyb | dougwig: i guess we need to make sure the external interface is as expected - i'm thinking it needs to set accept_ra=2 on eth0. i will start cobbling together a test config in the meantime | 18:01 |
| sdague | clarkb: is there a logstash processor that works with - http://logs.openstack.org/21/359721/1/gate/gate-tempest-dsvm-full-ubuntu-xenial/065baa6/_zuul_ansible/ansible_log.txt ? | 18:01 |
| dougwig | haleyb: if you note this hacky test i started 10 minutes before yours: https://review.openstack.org/#/c/360011/ , it also failed. and i hardcoded eth0 in that one. | 18:02 |
| clarkb | sdague: I am not aware of one but I would guess someone somewhere ahs written one we can borrow | 18:03 |
| clarkb | logstash used to have a recipes website let me see if that still exists | 18:03 |
| haleyb | dougwig: i'm running a new stack with that change to see which one it sets | 18:03 |
| sdague | clarkb: that would be great, we've got a class of early fails that are no longer in the console.html | 18:03 |
| *** kushal has quit IRC | 18:03 | |
| clarkb | https://github.com/logstash/cookbook/tree/gh-pages/recipes that looks old | 18:04 |
| nibalizer | thanks EmilienM for doign https://review.openstack.org/#/c/359539/ | 18:04 |
| nibalizer | and thanks Hunner for the fast turnaround on that | 18:04 |
| EmilienM | nibalizer: cool | 18:04 |
| jeblair | sdague: what's the class? | 18:05 |
| clarkb | and all the googles return are ansible playbooks to deploy logstash :/ | 18:06 |
| sdague | http://logs.openstack.org/27/348427/4/gate/gate-tempest-dsvm-full-devstack-plugin-ceph/97a875e/ | 18:06 |
| jeblair | clarkb: how do you stash the logs from that deployment ?!? ;) | 18:06 |
| *** guilherme has quit IRC | 18:06 | |
| sdague | jeblair: looks like an ssh connect failure in the zuul ansible | 18:06 |
| sdague | that one has a mitm warning, I've seen others which are just a hang / timeout after 10 minutes | 18:07 |
| jeblair | pabelanger: http://logs.openstack.org/27/348427/4/gate/gate-tempest-dsvm-full-devstack-plugin-ceph/97a875e/_zuul_ansible/ansible_log.txt may be of interest to you | 18:07 |
| sdague | but in trying to process the uncategorized board, we can't track those, because that file is not indexed | 18:07 |
| *** sputnik13 has quit IRC | 18:08 | |
| pabelanger | jeblair: looking | 18:09 |
| *** sputnik13 has joined #openstack-infra | 18:09 | |
| *** guilherme has joined #openstack-infra | 18:09 | |
| dougwig | fungi, jeblair, clarkb - both my stacks failed fetching cirros, and the current gerrit reviews don't work. and sean is out of pocket. we'll keep working, but realistically, this isn't a fix that's going to happen in an hour with any degree of assurance. i think we should plan around a fix by cob tomorrow, and do whatever is needed to stabilize until then. | 18:09 |
| dougwig | if we're early, great, but i want to set expectations. | 18:09 |
| *** Gorian|work has joined #openstack-infra | 18:09 | |
| *** guilherme has quit IRC | 18:10 | |
| clarkb | dougwig: ok | 18:10 |
| *** guilherme has joined #openstack-infra | 18:10 | |
| *** _nadya_ has joined #openstack-infra | 18:10 | |
| jeblair | sdague: thanks | 18:10 |
| *** pmalik_ has left #openstack-infra | 18:10 | |
| sdague | jeblair: let me see if I can find that second one | 18:10 |
| *** guilherme has quit IRC | 18:11 | |
| *** jamesden_ has joined #openstack-infra | 18:11 | |
| sdague | I think it's lost in my sea of tabs never to be found again | 18:12 |
| sdague | human logstash is extremely inefficient for repeat searches :) | 18:13 |
| *** hockeynut has quit IRC | 18:13 | |
| mwhahaha | Anyone know who the owner of 'Hitachi Manila HNAS CI' is and why it's trying to run on puppet-swift changes? | 18:13 |
| mwhahaha | i'm assuming it's misconfigured somewhere but i have no idea where | 18:13 |
| sdague | jeblair: oh, here is another one http://logs.openstack.org/30/357430/2/gate/gate-nova-docs-ubuntu-xenial/c6e2293/_zuul_ansible/ansible_log.txt | 18:13 |
| *** sambetts is now known as sambetts|afk | 18:13 | |
| sdague | 2016-08-22 18:44:48,708 p=21516 u=zuul | fatal: [node]: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."} | 18:13 |
| sdague | the console.html is just | 18:14 |
| *** Gorian|work has quit IRC | 18:14 | |
| clarkb | mwhahaha: the gerrit account may have an email addr assocaited wtih it if you hover over the name in the comments but also all third party CIs should have a page on the wiki with contact info | 18:14 |
| jeblair | sdague, pabelanger: the fingerprint it got back from the second connection is indeed different than the fingerprint of the file it got via the initial ssh-keyscan | 18:14 |
| sdague | 2016-08-22 18:45:02.304470 | [Zuul] Job complete, result: FAILURE | 18:14 |
| *** roxanag__ has joined #openstack-infra | 18:14 | |
| *** rbrndt has joined #openstack-infra | 18:14 | |
| pabelanger | jeblair: looks to be rax-dfw, I wonder if another node came back online with that IP address | 18:14 |
| clarkb | mwhahaha: google says https://wiki.openstack.org/wiki/ThirdPartySystems/Hitachi_Manila_HNAS_CI | 18:14 |
| mwhahaha | clarkb: thanks | 18:14 |
| anteaya | mwhahaha: sounds like it should be disabled, if you get no response from operators do say so, and I'll get it disabled | 18:15 |
| mwhahaha | ok thanks | 18:16 |
| anteaya | mwhahaha: thank you | 18:16 |
| pabelanger | jeblair: I think we seen this issue before in rax, and lead to us enabling ssh-keyscan for zuul-launcher. | 18:16 |
| *** roxanagh_ has quit IRC | 18:17 | |
| *** amotoki has quit IRC | 18:17 | |
| *** dims has quit IRC | 18:18 | |
| *** dizquierdo has joined #openstack-infra | 18:18 | |
| jeblair | pabelanger: yeah ... i think the next debugging step would be to see if we can find whether any host used 01:1b:17:f0:68:7d:c6:00:d2:af:61:04:e8:0d:e3 as a key | 18:18 |
| *** dims has joined #openstack-infra | 18:18 | |
| pabelanger | agreed | 18:19 |
| jeblair | pabelanger: unfortunately, while we do store the host key in the _zuul_ansible directory, we don't log its fingerprint under normal circumstances | 18:19 |
| clarkb | jeblair: pabelanger our images should have no key on boot forcing new keys to be generated. So I don't think the host is coming up with one key then generating a new one | 18:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Clean up OpenStack references and genericize https://review.openstack.org/360036 | 18:19 |
| clarkb | however clouds can and do reuse IPs so maybe some flaw with not storing fingerprints? | 18:19 |
| *** rhallisey has quit IRC | 18:20 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a wiki-dev.o.o server to test newer mediawiki https://review.openstack.org/358246 | 18:20 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Set wiki name and logo URL https://review.openstack.org/360037 | 18:20 |
| jeblair | clarkb: i agree with your assesment that the boot-up thing is unlikely, but can you elaborate on your second idea? | 18:20 |
| *** marcusvrn_ has joined #openstack-infra | 18:20 | |
| *** kushal has joined #openstack-infra | 18:20 | |
| tpsilva | mwhahaha, anteaya: marcusvrn_ is the owner | 18:21 |
| clarkb | jeblair: in $cloud we boot with IP foo, run tests, delete. Then some time later we reuse IP foo due to lack of ipv4 space. If we had stored a fingerprint somehow for that IP previously then it may not like it when things get recycled | 18:21 |
| tpsilva | marcusvrn_: do you know what's happening? | 18:21 |
| anteaya | tpsilva: great marcusvrn_ needs to attend to his system or it will be disabled | 18:21 |
| jeblair | clarkb: ah, right -- we don't store keys across runs. you're right, that would be a problem, and it turns out, we would hit it in about 15 minutes in our setup. :) | 18:21 |
| clarkb | jeblair: ya I would expect it to be a more common fail if that was happening | 18:21 |
| *** Na3iL has quit IRC | 18:22 | |
| jeblair | clarkb: we do store keys for a single run though -- and in the case sdague linked, we retrieved the key, ssh'd in at least once with it, then did it again and the key had changed | 18:22 |
| openstackgerrit | Dmitry Ilyin proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 18:23 |
| marcusvrn_ | anteaya: yep, I have already stopped the CI. I'm setting a nodepool environment and ran puppet script which configure zuul to listen all projects | 18:23 |
| jeblair | clarkb: (note the top of the ansible log, it performs a keyscan: http://logs.openstack.org/27/348427/4/gate/gate-tempest-dsvm-full-devstack-plugin-ceph/97a875e/_zuul_ansible/ansible_log.txt and we even store and upload the results of the keyscan: http://logs.openstack.org/27/348427/4/gate/gate-tempest-dsvm-full-devstack-plugin-ceph/97a875e/_zuul_ansible/known_hosts | 18:23 |
| *** flepied has quit IRC | 18:24 | |
| anteaya | marcusvrn_: good, have you considered pointing your system at http://git.openstack.org/cgit/openstack-dev/ci-sandbox/ while in testing mode | 18:25 |
| anteaya | mwhahaha: ^^ | 18:25 |
| marcusvrn_ | anteaya: Yes, that's what I did now | 18:25 |
| *** d34dh0r53 is now known as b3rnard0-b0n-h4r | 18:25 | |
| anteaya | marcusvrn_: great, thank you | 18:26 |
| *** sdague has quit IRC | 18:26 | |
| mwhahaha | marcusvrn_, anteaya: thanks | 18:26 |
| anteaya | marcusvrn_: just as a reality check, we are coming up to feature freeze so folks have very little patience with new ci systems running amok | 18:27 |
| asselin__ | marcusvrn_, which puppet script "configure zuul to listen all projects" ? | 18:27 |
| anteaya | marcusvrn_: https://releases.openstack.org/newton/schedule.html | 18:27 |
| anteaya | asselin__: glad you were here to see that, I also wondered | 18:28 |
| *** Sukhdev has joined #openstack-infra | 18:28 | |
| marcusvrn_ | anteaya: ok, sorry for the inconvenience | 18:29 |
| openstackgerrit | Merged openstack-infra/project-config: Run host lookup first for configure_mirror.sh https://review.openstack.org/359289 | 18:29 |
| openstackgerrit | Merged openstack-infra/project-config: Include dib-builddate.txt for configure_mirror.sh https://review.openstack.org/359290 | 18:30 |
| anteaya | marcusvrn_: thank you for your attentiveness, ci systems are hard, do consider attending a third party meeting and working with other operators | 18:30 |
| anteaya | marcusvrn_: there are many holes to fall down, sometimes it helps to chat with others with experience to avoid a few | 18:30 |
| *** hyakuhei has quit IRC | 18:30 | |
| marcusvrn_ | asselin__: I'm using this guide: http://docs.openstack.org/infra/openstackci/third_party_ci.html and start zuul before configure to sandbox project...so by defult, it listens all projects | 18:31 |
| *** claudiub has joined #openstack-infra | 18:31 | |
| *** cody-somerville has joined #openstack-infra | 18:31 | |
| marcusvrn_ | anteaya: sure! thanks | 18:32 |
| asselin__ | marcusvrn_, I see, so you used the upstream project-config repo instead of your own? | 18:32 |
| anteaya | marcusvrn_: thank you | 18:32 |
| *** pvaneck has joined #openstack-infra | 18:32 | |
| marcusvrn_ | asselin__: yep, now, I'm configuring mine | 18:32 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 18:32 |
| asselin__ | marcusvrn_, here is says to use your own: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n5 | 18:33 |
| asselin__ | marcusvrn_, what doc change would have helped you avoid this? | 18:33 |
| asselin__ | marcusvrn_, I ask b/c you aren't the first, but I would prefer you to be the last.... | 18:34 |
| pabelanger | jeblair: clarkb: I think we can easily update zuul-launcher to also log the fingerprint of the node. Working on a patch now. | 18:36 |
| jeblair | pabelanger: cool, thanks. i'm running ssh-keygen on all the known_hosts files on the log server to see if i can find where that key might have been used before. this will not be fast. | 18:37 |
| marcusvrn_ | asselin__: Maybe add the same step about create a local git project to save local changes like system-config step | 18:37 |
| marcusvrn_ | It would help, I think | 18:38 |
| asselin__ | marcusvrn_, So something added to here? http://docs.openstack.org/infra/openstackci/third_party_ci.html#create-an-initial-project-config-repository | 18:38 |
| asselin__ | reading again and not very clear | 18:39 |
| *** degorenko is now known as _degorenko|afk | 18:39 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for get me a network fails https://review.openstack.org/359960 | 18:39 |
| *** hyakuhei has joined #openstack-infra | 18:41 | |
| *** hyakuhei has quit IRC | 18:41 | |
| *** hyakuhei has joined #openstack-infra | 18:41 | |
| *** hyakuhei has quit IRC | 18:41 | |
| *** hyakuhei has joined #openstack-infra | 18:41 | |
| *** senk has quit IRC | 18:41 | |
| *** gyee has joined #openstack-infra | 18:41 | |
| *** senk has joined #openstack-infra | 18:41 | |
| *** zz_zz_ja has quit IRC | 18:41 | |
| *** ekhugen has quit IRC | 18:41 | |
| pabelanger | jeblair: I'm using: ssh-keygen -E md5 -lf known_hosts to extract the key currently | 18:42 |
| pabelanger | does that work for you? | 18:42 |
| marcusvrn_ | asselin__: yep...because, reading that, I created a local project-config, I edited it, but when configured my common.yaml, I added upstream project instead of local project....Maybe It was my fault too....some misunderstood | 18:43 |
| openstackgerrit | Merged openstack-infra/nodepool: Have buildZooKeeperHosts accept a config object https://review.openstack.org/357347 | 18:44 |
| jeblair | pabelanger: that seems good | 18:45 |
| *** adrian_otto has quit IRC | 18:48 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cinder unit test OOM https://review.openstack.org/360023 | 18:48 |
| *** _nadya_ has quit IRC | 18:50 | |
| dougwig | aha, repro'ed. | 18:52 |
| *** flepied1 has joined #openstack-infra | 18:52 | |
| *** hasharAway is now known as hashar | 18:52 | |
| mordred | dougwig: ooh! | 18:52 |
| * sc68cal perks up | 18:53 | |
| *** piet has quit IRC | 18:54 | |
| *** piet has joined #openstack-infra | 18:54 | |
| *** thiagop has quit IRC | 18:55 | |
| *** raunak has quit IRC | 18:55 | |
| *** piet has quit IRC | 18:56 | |
| *** piet has joined #openstack-infra | 18:56 | |
| *** esikachev has quit IRC | 18:57 | |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add Almanach IRC channel https://review.openstack.org/360049 | 18:58 |
| *** ekhugen has joined #openstack-infra | 18:58 | |
| *** zz_ja has joined #openstack-infra | 18:59 | |
| *** jcoufal has quit IRC | 18:59 | |
| *** piet has quit IRC | 19:00 | |
| *** piet has joined #openstack-infra | 19:01 | |
| *** gordc has joined #openstack-infra | 19:01 | |
| *** thiagop has joined #openstack-infra | 19:02 | |
| *** ddieterly[away] is now known as ddieterly | 19:04 | |
| *** Goneri has joined #openstack-infra | 19:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 19:07 |
| *** ccamacho has quit IRC | 19:07 | |
| *** piet has quit IRC | 19:07 | |
| sc68cal | dougwig: yeah no default route | 19:07 |
| haleyb | sc68cal: right, and accept_ra is all 1 | 19:08 |
| haleyb | think we do need to change to use the public interface | 19:08 |
| dougwig | sc68cal: yep. so, why? | 19:08 |
| *** esberglu has quit IRC | 19:09 | |
| *** piet has joined #openstack-infra | 19:09 | |
| sc68cal | haleyb: so, we set it accept_ra = 2 - then at some point in the future it gets set back to 1 | 19:09 |
| *** kzaitsev_mb has joined #openstack-infra | 19:09 | |
| dougwig | i expect if we added a default 6 right, it'd come right back on the air. so... how did it get removed? | 19:09 |
| dougwig | oh wait. | 19:10 |
| sc68cal | dougwig: most likely when we set ipv6.forwarding = 1 | 19:10 |
| dougwig | this was a repro case, it does *not* have the accept_ra 2 fix. | 19:10 |
| sc68cal | ah. | 19:10 |
| *** senk has quit IRC | 19:10 | |
| haleyb | sc68cal: it shouldn't do that, would just remove the default route | 19:10 |
| dougwig | so, can we turn off forwarding, let it get a route, then set 2 and reset forwarding? | 19:10 |
| haleyb | yes, set it to 2 on eth0 | 19:11 |
| *** senk has joined #openstack-infra | 19:11 | |
| *** ilyashakhat has joined #openstack-infra | 19:11 | |
| openstackgerrit | Pradeep Kilambi proposed openstack-infra/project-config: Create puppet-panko repository https://review.openstack.org/353662 | 19:11 |
| haleyb | who's driving? | 19:11 |
| dougwig | just set it | 19:11 |
| *** piet has quit IRC | 19:11 | |
| dougwig | i'm out, i'll let one of you drive. | 19:12 |
| dougwig | oh, it got a default route. | 19:12 |
| haleyb | ok, forwarding off, default route back | 19:12 |
| dougwig | k | 19:12 |
| sc68cal | dougwig: it just picked up a default route! | 19:12 |
| sc68cal | default via fe80::def dev eth0 proto ra metric 1024 expires 1761sec hoplimit 64 | 19:13 |
| sc68cal | I SEE IT | 19:13 |
| haleyb | so accept_ra=2, forwarding=1, still has default | 19:13 |
| sc68cal | it actually now has a default | 19:13 |
| sc68cal | here's I'll paste | 19:13 |
| sc68cal | it didn't have a default route | 19:13 |
| *** esikachev has joined #openstack-infra | 19:13 | |
| dougwig | i see it. | 19:14 |
| sc68cal | I ran it probably after dougwig ran the cmd, and the default route got added | 19:14 |
| haleyb | in my local test, PUBLIC_INTERFACE was blank, is it that way in the check jobs too? | 19:14 |
| haleyb | sc68cal: i was setting them ala Oz | 19:14 |
| dougwig | so, what was wrong with the devstack fix, then? the one where i *hard-coded* eth0 as a test? | 19:14 |
| *** rhallisey has joined #openstack-infra | 19:14 | |
| *** b3rnard0-b0n-h4r is now known as d34dh0r53 | 19:14 | |
| haleyb | dougwig: i don't know, that should have worked if that was the issue | 19:14 |
| *** ddieterly is now known as ddieterly[away] | 19:15 | |
| *** csomerville has joined #openstack-infra | 19:15 | |
| clarkb | dougwig: are you sure eth0 is the interface? | 19:15 |
| *** dizquierdo has quit IRC | 19:15 | |
| dougwig | clarkb: on osic it is. | 19:15 |
| clarkb | I think it may use biosname in some places | 19:15 |
| clarkb | depending on distro release | 19:15 |
| dougwig | clarkb: ok. | 19:15 |
| sc68cal | dougwig: I think it's the timing | 19:16 |
| haleyb | clarkb: that might be it, i think it's the PUBLIC_INTERFACE in devstack but need to verify | 19:16 |
| haleyb | sc68cal: oh, maybe add a sleep? | 19:16 |
| sc68cal | haleyb: my comment earlier - maybe we need to set accept_ra after we set forwarding = 1 | 19:16 |
| *** esberglu has joined #openstack-infra | 19:16 | |
| sc68cal | instead of before | 19:16 |
| clarkb | sc68cal: we troed tjat forst | 19:16 |
| *** cody-somerville has quit IRC | 19:16 | |
| dougwig | sc68cal: how so? going offline and then back in a few seconds should be seamless to clients. | 19:16 |
| clarkb | but using all not the specific interface name | 19:16 |
| haleyb | sc68cal: but forwarding=1 will have clobbered the default route | 19:16 |
| haleyb | sc68cal: do you think i can assume PUBLIC_INTERFACE is set here? | 19:18 |
| *** esikachev has quit IRC | 19:18 | |
| *** ddieterly[away] is now known as ddieterly | 19:18 | |
| sc68cal | haleyb: yes | 19:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Store ssh_host_key of remote node https://review.openstack.org/360057 | 19:19 |
| sc68cal | I think it's set in devstack-gate | 19:19 |
| sc68cal | in devstack it defaults to "" | 19:19 |
| pabelanger | jeblair: clarkb: ^ extract ssh_host_key for zuul-launcher | 19:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat https://review.openstack.org/358868 | 19:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Stop using NodePoolBuilder class https://review.openstack.org/359896 | 19:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for registering a watch. https://review.openstack.org/358837 | 19:19 |
| sc68cal | haleyb: hmm.... | 19:19 |
| dougwig | sc68cal: what about non-CI users? | 19:19 |
| sc68cal | I think all our docs state that you need to set PUBLIC_INTERFACE | 19:20 |
| sc68cal | but we don't provide a default | 19:20 |
| haleyb | dougwig: they're screwed for now, and yes it's "" but must be set in a yaml somewhere? | 19:20 |
| *** sdake has quit IRC | 19:20 | |
| *** esberglu has quit IRC | 19:20 | |
| sc68cal | We may need to do ip -6 r s and grep for the interface | 19:21 |
| sc68cal | before the forwarding = 1 fires and clobbers | 19:22 |
| sc68cal | in devstack | 19:22 |
| sc68cal | looks like we can't rely on PUBLIC_INTERFACE | 19:22 |
| dougwig | devstack is adding that masq crap. use the same variable that uses. | 19:22 |
| sc68cal | ah good point, yeah | 19:22 |
| dougwig | $default_dev | 19:22 |
| AJaeger | anybody else that wants to review fungi's system-config change to update zuul-env on persistent nodes? https://review.openstack.org/#/c/359352/ | 19:23 |
| dougwig | it looks it up at that point in the code. might need to copy that block, or bump its scope. | 19:23 |
| *** esberglu has joined #openstack-infra | 19:23 | |
| *** piet has joined #openstack-infra | 19:23 | |
| haleyb | dougwig: let me move it to the top, hang on | 19:23 |
| sc68cal | we'll need to copy the block or assign it to a global var, it's currently a local scoped var | 19:24 |
| clarkb | except the route isn't goig away when we set it to forwarding right? its going away when neutron creates the subnet | 19:24 |
| dougwig | it likely takes a certain amount of time to expire. | 19:25 |
| rcarrillocruz | pabelanger: created flavors and projects fine with the launucher on infracloud | 19:25 |
| haleyb | dougwig: setting forwarding=1 will remove the default route immediately | 19:25 |
| dougwig | hmm. | 19:25 |
| clarkb | I just worry we are focusing on the wrong code path. I do think we need to set accept_ra to 2 at some point but that isn't the point of failure currently | 19:25 |
| rcarrillocruz | brb | 19:25 |
| *** rvasilets___ has joined #openstack-infra | 19:25 | |
| clarkb | dougwig: haleyb sc68cal an easy way to test would be to make a new public ipv6 subnet on your reproduce box right? then see if the route goes away | 19:26 |
| dougwig | clarkb is right, 2+2 just ceased being 4. | 19:26 |
| haleyb | clarkb: that is what is causing the route to go away though | 19:26 |
| haleyb | we can do that on dougs box, thought we did? | 19:26 |
| clarkb | haleyb: sure but that implies neutron si doing it not devstack | 19:26 |
| pabelanger | rcarrillocruz: yay | 19:26 |
| dougwig | haleyb: then how are we seeing further activity in the log? | 19:27 |
| clarkb | haleyb: which is why I think its the wrong code path | 19:27 |
| *** _nadya_ has joined #openstack-infra | 19:27 | |
| *** dtardivel has quit IRC | 19:27 | |
| clarkb | (I do still think we need to address the area you are focusing on, I jsut think thats the next step after we figure out why the new subnet kills things) | 19:27 |
| dougwig | i mean, let's try the fix, but there's a mystery here. | 19:27 |
| haleyb | clarkb: it's happening when devstack sets forwarding=1, i can try again on dougs system | 19:27 |
| *** senk has quit IRC | 19:27 | |
| dougwig | haleyb: if you look at the paste, it doesn't happen then, it happens on the public v6 subnet create. | 19:28 |
| dougwig | that's clarkb's point. | 19:28 |
| *** senk has joined #openstack-infra | 19:28 | |
| clarkb | haleyb: my investigation shows it happens when neutron creates the public ipv6 subnet | 19:28 |
| sc68cal | clarkb: I thought that paste was when it was doing the public v4 ne | 19:29 |
| clarkb | sc68cal: my first paste was cut off short. dougwig pointed that out and I posted a nwe one that has less context but includes the important bits | 19:30 |
| haleyb | well, it does create the public v6 subnet right before setting forwarding=1 | 19:30 |
| dougwig | sc68cal: that was a truncated paste. look in the bug report | 19:30 |
| clarkb | http://paste.openstack.org/show/562831/ that one | 19:30 |
| *** Goneri has quit IRC | 19:30 | |
| clarkb | haleyb: ok so maybe its a buffering issue. In any case it would be super simple to rule out if you create a new subnet | 19:30 |
| haleyb | clarkb: i set forwarding=0, then accept_ra=1, then forwarding=1 on dougs system and default route went away | 19:31 |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add Almanach IRC channel https://review.openstack.org/360049 | 19:32 |
| clarkb | haleyb: yes but that doesn't prove that the subnet creation isn't also a problem... | 19:32 |
| clarkb | haleyb: only that forwarding=1 is also a problem | 19:32 |
| clarkb | I just want to rule out that we need to fix two things or learn that we do | 19:32 |
| sc68cal | I think it's a buffering issue | 19:32 |
| sc68cal | since it's dying at different points in the script run | 19:32 |
| sc68cal | one of your pastes it died at the v4 public net create | 19:33 |
| clarkb | sc68cal: mp | 19:33 |
| clarkb | er | 19:33 |
| clarkb | no | 19:33 |
| clarkb | that first paste was truncated | 19:33 |
| clarkb | and is invalid do not look at it | 19:33 |
| haleyb | i can give doug the change to try, just hacking now | 19:34 |
| *** DsAAzvTVQt has joined #openstack-infra | 19:34 | |
| *** esikachev has joined #openstack-infra | 19:34 | |
| clarkb | the paste I just pasted is the correct one from the same ssh connection. It did not get truncated | 19:34 |
| dougwig | haleyb: if you focus on updating the devstack patch, i'll check clarkb's concern real quick. | 19:34 |
| * sc68cal resists the urge to use the wall command | 19:35 | |
| dougwig | haleyb: sc68cal: did one of you set accept_ra back to 1 and nuke the default route on my box? | 19:35 |
| sc68cal | not i | 19:35 |
| *** e0ne has joined #openstack-infra | 19:35 | |
| fungi | clarkb: dougwig: haleyb: sc68cal: it's also possible the connectivity loss is a slightly delayed reaction to something changing a little earlier | 19:35 |
| haleyb | dougwig: yes, i did it | 19:36 |
| dougwig | haleyb: *phew*, scared me. | 19:36 |
| dougwig | haleyb: are you testing on that box? i don't want to step on you. | 19:36 |
| *** senk has quit IRC | 19:36 | |
| * sc68cal also breathes a sigh of relief | 19:36 | |
| haleyb | dougwig: let it get the default route bcak, hang on | 19:36 |
| dougwig | haleyb: i'll stay hands off until you tell me that you're done. | 19:37 |
| *** piet has quit IRC | 19:37 | |
| *** DsAAzvTVQt has quit IRC | 19:38 | |
| *** esikachev has quit IRC | 19:38 | |
| *** claudiub has quit IRC | 19:41 | |
| rcarrillocruz | actually | 19:42 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Remove initial resources creation from infracloud controller manifest https://review.openstack.org/360061 | 19:42 |
| rcarrillocruz | it was a bit of a mix | 19:42 |
| rcarrillocruz | pabelanger: ^ | 19:42 |
| rcarrillocruz | at the mid-cycle we put some stuff for creating resources (projects, users, etc) within puppet code | 19:42 |
| *** Goneri has joined #openstack-infra | 19:43 | |
| rcarrillocruz | let's remove it, i just realized i had openstackjenkins, when we should have now openstackzuul | 19:43 |
| *** VaUsfOUbRl has joined #openstack-infra | 19:43 | |
| *** VaUsfOUbRl has quit IRC | 19:44 | |
| dougwig | haleyb: i don't see it getting the route back | 19:45 |
| *** e0ne has quit IRC | 19:45 | |
| *** Apoorva has quit IRC | 19:45 | |
| haleyb | dougwig: i accidentally tweaked the sysctl wrong, and now it's waiting for another RA, which isn't frequent in rax i guess | 19:45 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Remove rcarrillocruz from infracloud users https://review.openstack.org/360063 | 19:45 |
| dougwig | osic, not rax | 19:46 |
| clarkb | its possible that cloudnull can change that for us if it will help | 19:46 |
| *** kzaitsev_mb has quit IRC | 19:46 | |
| clarkb | they are running liberty so liekly depends on what we can do with that | 19:46 |
| haleyb | give it another minute and it should return | 19:46 |
| cloudnull | ++ what needs to be tweaked? | 19:47 |
| haleyb | https://review.openstack.org/#/c/359490/ is updated | 19:47 |
| * cloudnull reading | 19:47 | |
| haleyb | cloudnull: RAs are just infrequent in OSIC apparently, do you know the interval? | 19:47 |
| * jamesden_ looking | 19:48 | |
| *** Goneri has quit IRC | 19:48 | |
| cloudnull | ^ | 19:48 |
| cloudnull | thanks jamesden_ | 19:48 |
| haleyb | dougwig: it's back | 19:48 |
| *** hockeynut has joined #openstack-infra | 19:49 | |
| jamesden_ | haleyb the default interval is 600 seconds | 19:49 |
| jamesden_ | i can lower it if you'd like | 19:49 |
| *** jamesden_ is now known as jamesdenton | 19:49 | |
| haleyb | typically the client will send an RS, in this case we deleted something and had to wait for another | 19:49 |
| *** xcGlQHdZGT has joined #openstack-infra | 19:50 | |
| dougwig | haleyb: looks like we lucked into one of the jobs starting immediate on an osic node (gate-tempest-dsvm-neutron-full-ubuntu-xenial) | 19:50 |
| mordred | woot | 19:50 |
| nibalizer | fungi: where are the docs on that release key attestation thing I need to do? | 19:52 |
| nibalizer | (as long as I'm doing gpg key things anyways) | 19:52 |
| jeblair | nibalizer: http://docs.openstack.org/infra/system-config/signing.html | 19:52 |
| haleyb | jamesdenton: is that the max? that defaults to 600, min would be .33 of that if not set, so 200 | 19:52 |
| jamesdenton | new interval is 200. hopefully that helps a little next time. | 19:52 |
| haleyb | jamesdenton: thanks, again, was that max interval you chaanged? | 19:53 |
| jeblair | nibalizer: that's where all the root docs are | 19:53 |
| jeblair | s/root// | 19:53 |
| jamesdenton | haleyb ra-interval range is <4-1800> | 19:53 |
| *** asettle has joined #openstack-infra | 19:53 | |
| jamesdenton | haleyb defaults to 600 | 19:53 |
| *** esikachev has joined #openstack-infra | 19:55 | |
| haleyb | jamesdenton: oh, it's a cisco device. their description of that value isn't like radvd (or anything i'm used to) that specifies min/max - but 200 is better, thanks | 19:55 |
| jamesdenton | ahh ok. Yes, it's a Cisco box | 19:55 |
| haleyb | the open source implementations all follow the RFC naming convention, cisco had to be different | 19:56 |
| sc68cal | heh. | 19:56 |
| sc68cal | freebsd calls it rtadvd ;) | 19:56 |
| sc68cal | not radvd | 19:57 |
| *** gordc has quit IRC | 19:57 | |
| haleyb | sc68cal: but i bet thety have min/max settings called MaxRtrAdvInterval, etc | 19:57 |
| *** piet has joined #openstack-infra | 19:58 | |
| sc68cal | oh yeah, they just added a "t" for clarity to the binary name :) | 19:58 |
| nibalizer | jeblair: ty ty | 19:58 |
| haleyb | dougwig: so when will we know if it worked? | 19:58 |
| cloudnull | tyvm jamesdenton | 19:59 |
| jamesdenton | sure. ping me if you want to tweak anything else. | 19:59 |
| * mordred is excited about our new RA overlords | 20:00 | |
| SpamapS | so ipv6. very RA. #dogeV6 | 20:00 |
| dougwig | haleyb: watch and see if 2001:4800:1ae1:18:f816:3eff:fe30:66c3 goes offline in the next 10 mins or so. | 20:01 |
| haleyb | started a ping | 20:01 |
| dougwig | i'm pinging and watching zuul | 20:01 |
| *** hockeynut has quit IRC | 20:01 | |
| dougwig | are you done with the test node for the moment? | 20:02 |
| haleyb | dougwig: except for running a ping there, yes | 20:02 |
| *** hockeynut has joined #openstack-infra | 20:03 | |
| *** markusry has quit IRC | 20:03 | |
| *** asettle has quit IRC | 20:03 | |
| *** markusry has joined #openstack-infra | 20:03 | |
| *** markusry has quit IRC | 20:03 | |
| *** jamesden_ has joined #openstack-infra | 20:04 | |
| *** jamesdenton has quit IRC | 20:05 | |
| *** jamesden_ is now known as jamesdenton | 20:05 | |
| sc68cal | what's the telnet trick for the new zuul to connect to the jenkins console | 20:05 |
| *** sdague has joined #openstack-infra | 20:07 | |
| *** cody-somerville has joined #openstack-infra | 20:07 | |
| *** vhosakot has quit IRC | 20:08 | |
| dougwig | haleyb: clarkb: sc68cal: the above job is now running tempest tests. | 20:08 |
| sc68cal | so it worked? | 20:08 |
| dougwig | cautiously optimistic. | 20:08 |
| *** dprince has quit IRC | 20:09 | |
| *** csomerville has quit IRC | 20:09 | |
| *** amitgandhinz has quit IRC | 20:09 | |
| *** raunak has joined #openstack-infra | 20:10 | |
| *** amitgandhinz has joined #openstack-infra | 20:10 | |
| *** cody-somerville has quit IRC | 20:12 | |
| *** vhosakot has joined #openstack-infra | 20:12 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: highlight when integrated_gate data is out of date https://review.openstack.org/359932 | 20:13 |
| *** dimtruck is now known as zz_dimtruck | 20:13 | |
| clarkb | so this isse was in not getting the interface name correct in previous attempts and needing to not use "all" | 20:14 |
| * clarkb double checks the run | 20:14 | |
| sc68cal | correct | 20:14 |
| russellb | sc68cal trick? i just grab the uri from the zuul status page and run telnet to that IP and port ... | 20:14 |
| sc68cal | russellb: ah. doh | 20:14 |
| clarkb | ya its running tempest. We should make sure it finishes and nothing else comes up but this looks good | 20:15 |
| sdague | progress on the ipv6 issue? | 20:15 |
| sdague | is there a patch to look at? | 20:15 |
| sc68cal | sdague: was about to highlight you | 20:15 |
| sc68cal | sdague: https://review.openstack.org/#/c/359490/ | 20:15 |
| dmsimard | I'm not a Zuul gate expert but some jobs have been there for almost 24hrs now ? Is this typical ? | 20:15 |
| dmsimard | I was surprised to see changes +W'd this morning still hadn't merged .. | 20:16 |
| *** Goneri has joined #openstack-infra | 20:16 | |
| pabelanger | dmsimard: which review? | 20:16 |
| dmsimard | pabelanger: particular reviews I'm interested in are https://review.openstack.org/#/c/359391/ and https://review.openstack.org/#/c/358670/ but looking at the gate in general shows a lot of jobs >20hrs http://status.openstack.org/zuul/ | 20:17 |
| clarkb | sc68cal: dougwig haleyb its probably worht a writeup to the dev list so that other deployment based tests know what they need to do too (if necssary) | 20:17 |
| *** piet has quit IRC | 20:17 | |
| clarkb | thinking kolla/chef/puppet/osa may potentially run into this as well | 20:17 |
| pabelanger | dmsimard: Ya, working on the fix now. 359490 should be the solution | 20:17 |
| dougwig | clarkb: and a doc bug on the install guide. | 20:18 |
| *** krtaylor has quit IRC | 20:18 | |
| clarkb | dougwig: ++ | 20:18 |
| dmsimard | pabelanger: ack | 20:18 |
| sdague | dmsimard: there are many bugs in the gate, this last one is one of them, but there are many | 20:18 |
| *** piet has joined #openstack-infra | 20:18 | |
| sdague | we've already addressed 2 others today as well | 20:18 |
| sc68cal | I'll propose a doc patch to the networking guide | 20:18 |
| dmsimard | sdague: wasn't aware of the gate issues and that's why I was inquiring, thanks. | 20:18 |
| dougwig | sc68cal: k, thanks | 20:18 |
| *** _nadya_ has quit IRC | 20:19 | |
| sdague | clarkb / sc68cal ok, so this is running tests fine on neutron on an ipv6 only node atm? | 20:19 |
| clarkb | sdague: yes, need to see if it finishes tempest cleanly but it got past where there was trouble | 20:20 |
| haleyb | i am still able to ping the node in question | 20:20 |
| *** piet has quit IRC | 20:20 | |
| sdague | I don't have ipv6 connectivity from here so blind on it | 20:20 |
| sdague | gah, http://logs.openstack.org/49/359449/1/gate/gate-tempest-python34/c178353/console.html | 20:21 |
| sdague | osic apt package mirror break? | 20:21 |
| clarkb | I doubt its osic specific if the mirror is broken since its all backed bythe same afs volume | 20:21 |
| sdague | clarkb: then, ug mirror break | 20:22 |
| *** piet has joined #openstack-infra | 20:22 | |
| *** tqtran has quit IRC | 20:22 | |
| pabelanger | they would have updates since that run | 20:22 |
| sdague | the 2 gate fails in tempest unit tests right now are osic | 20:22 |
| sdague | fwiw | 20:22 |
| sdague | http://logs.openstack.org/40/358140/1/gate/gate-tempest-python34/db92922/console.html#_2016-08-24_19_56_02_689271 | 20:22 |
| sdague | so, +A on https://review.openstack.org/#/c/359490/ | 20:23 |
| clarkb | osic is > half our resources right now I think so not really surprising it would see fails | 20:23 |
| pabelanger | Hmm, I do see an issue in mirror-update.o.o | 20:23 |
| pabelanger | let me find out why | 20:23 |
| pabelanger | The lock file '/afs/.openstack.org/mirror/ubuntu/db/lockfile' already exists. There might be another instance with the | 20:24 |
| sdague | I'm about to leave computers for the day, so if that actually fixes the world please promote | 20:24 |
| clarkb | sdague: ok I will enqueue and promote to gate as soon as this test ends happily | 20:24 |
| sdague | clarkb: thanks | 20:24 |
| sdague | pabelanger: yeh, would be great, that mirror failure is going to cause cascading fails | 20:24 |
| sdague | that will pile up fast | 20:24 |
| nibalizer | fungi: jeblair im not seeing our key on the keyservers | 20:25 |
| nibalizer | oh nvm | 20:25 |
| nibalizer | lol | 20:25 |
| *** chem has quit IRC | 20:25 | |
| clarkb | sdague: fwiw I have a smallish script to run the telnet/nc through an ssh tunnel if you have access via ssh to another host with ipv6 | 20:25 |
| phschwartz | jeblair: sorry about yesterday, got caught up in stuff for my son's school | 20:25 |
| phschwartz | jeblair: The issue came back after a restart and adding of missing projects to our zuul layout. I have 3 pastes that show the about 960 lines before the first exception in the log | 20:26 |
| *** kzaitsev_mb has joined #openstack-infra | 20:27 | |
| phschwartz | jeblair: http://paste.openstack.org/show/563129/ http://paste.openstack.org/show/563131/ http://paste.openstack.org/show/563133/ | 20:27 |
| asselin__ | rcarrillocruz, mordred I'm running into that issue again where I can't connect to a VM's floating ip address. I don't know if it's the same or not, but it happened after I switched back from ansible 2.1.1.0 to 2.1.0.0. Restoring ansible version back didn't help. I'm on shade 1.11.1 | 20:27 |
| pabelanger | sdague: clarkb: looks like the ubuntu mirror is stale by 4 days, cleaning it up now | 20:27 |
| phschwartz | jeblair: it is also good to note that in 22 hours we got just over 701k of the exceptions in the log. That is an average of about 9 per second. | 20:27 |
| erlon | mwhahaha: hey, do you have the link of the puppet-swift changes? | 20:28 |
| asselin__ | rcarrillocruz, mordred some debug logs: http://paste.openstack.org/show/563135/ let me know if you have any ideas. | 20:28 |
| mwhahaha | erlon: it was showing up on https://review.openstack.org/360032 but i believe it was fixed as there have not been any subsequent CI postings | 20:30 |
| openstackgerrit | Merged openstack-infra/shade: Batch calls to list_floating_ips https://review.openstack.org/315697 | 20:31 |
| * clarkb makes a note to delete all the held instances in osic if this fixes things | 20:31 | |
| clarkb | (I didn't leave hold reason notes on them because I was in a hurry to beat nodepool deleting them) | 20:32 |
| anteaya | erlon: this might also help: https://review.openstack.org/#/q/reviewer:%22Hitachi+Manila+HNAS+CI+%253COpenstackManilaCI%2540hds.com%253E%22 | 20:32 |
| anteaya | erlon: bit of a mixed bag there | 20:32 |
| haleyb | clarkb: i'm writing an email to the dev list now too | 20:32 |
| clarkb | haleyb: great, thank you | 20:33 |
| erlon | mwhahaha: anteaya: thanks, it shouldnt be posting in those other projects | 20:33 |
| *** Hal has quit IRC | 20:33 | |
| anteaya | erlon: agreed, thanks for the follow up | 20:33 |
| *** Jeffrey4l__ has joined #openstack-infra | 20:34 | |
| erlon | mwhahaha: anteaya: started from 2:56 pm. Ill check that tomorrow morning | 20:34 |
| openstackgerrit | Travis Truman (automagically) proposed openstack-infra/project-config: Add os_monasca repository to OpenStack-Ansible https://review.openstack.org/360074 | 20:34 |
| anteaya | erlon: hopefully marcusvrn_ can make time for the third party meetings in future: http://eavesdrop.openstack.org/#Third_Party_Meeting | 20:34 |
| anteaya | erlon: and you are welcome too | 20:34 |
| *** Jeffrey4l_ has quit IRC | 20:34 | |
| anteaya | erlon: great, vigilence on a ci system, especially a new one is greatly appreciated | 20:35 |
| erlon | anteaya: thanks, we will, we are in the process to move our Cis to nodepool | 20:35 |
| pabelanger | sdague: clarkb: okay, mirror.ubuntu updated and released. We had a stale lock in reprepro which stopped the crontab for running for a few days | 20:35 |
| clarkb | pabelanger: any idea how it ended up in an inconsistent state? | 20:35 |
| Shrews | asselin__: you might need to add a wait for the ssh port to become active before trying the ssh. haven't seen the playbook, so don't know if you're actually doing that | 20:35 |
| clarkb | my understanding of the afs setup is that we should either work or fail to update? | 20:35 |
| anteaya | erlon: awesome, many other third party operators will have experience with moving to nodepool | 20:36 |
| *** sdague has quit IRC | 20:36 | |
| erlon | anteaya: this one still in our old infra, but someone must have changed something | 20:36 |
| pabelanger | clarkb: it maybe be possible it was my fault. I need to check the timelines but I did manually hold the lock a few days ago to import the source packages | 20:36 |
| clarkb | ah | 20:36 |
| *** Apoorva has joined #openstack-infra | 20:36 | |
| *** tonytan4ever has quit IRC | 20:36 | |
| anteaya | erlon: something happened, I'll let you follow up | 20:36 |
| pabelanger | possible I didn't clean some things up | 20:36 |
| anteaya | erlon: glad the spam has stopped, thank you | 20:36 |
| asselin__ | Shrews, actually I can't connect even manually....the floating ip is just missing altogether. For reference here's the playbook: https://review.openstack.org/#/c/320466/15 | 20:36 |
| pabelanger | clarkb: and I didn't actually check, until today, if the mirror update happened | 20:36 |
| *** hongbin has joined #openstack-infra | 20:37 | |
| erlon | mwhahaha: anteaya: welcome, thanks for helping | 20:37 |
| Shrews | asselin__: ah, well that'd be a different problem | 20:37 |
| anteaya | erlon: mwhahaha thank you :) | 20:37 |
| asselin__ | Shrews, do you think it's an issue with the client or the cloud itself? | 20:37 |
| pabelanger | clarkb: fungi: speaking of mirrors, I could use some help landing: https://review.openstack.org/#/c/347058/ Fixes an issue with jessie-security in mirror.debian | 20:38 |
| Shrews | asselin__: not sure. i have 0 background on what you're doing :) | 20:38 |
| *** tonytan4ever has joined #openstack-infra | 20:40 | |
| *** ilyashakhat has quit IRC | 20:40 | |
| *** fguillot has quit IRC | 20:41 | |
| *** edtubill has quit IRC | 20:41 | |
| clarkb | pabelanger: As soon as this tempest test finishes and I can reorg the gate for it, I will try to review that and fungi's zuul env change | 20:41 |
| pabelanger | clarkb: sure, no rush. | 20:42 |
| clarkb | nah I want to feel like I got something done today :) | 20:42 |
| anteaya | I think possibly fixing ipv6 classifies as getting something done | 20:43 |
| anteaya | it does in my book anyway | 20:43 |
| clarkb | anteaya: ya I just thought I would be working on finishing up the xenialing of newton this week. I gues there is still a little time left :) | 20:43 |
| anteaya | that is true | 20:44 |
| anteaya | something something the best laid plans something something | 20:44 |
| anteaya | clarkb: sc68cal haleyb dougwig thanks for your hardwork on ipv6 | 20:44 |
| pabelanger | clarkb: The only other osic-cloud1 issue I am seeing, appears to be a gearman issue in nodepool. Some context: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2016-08-24T13:22:26 | 20:45 |
| *** ansmith has quit IRC | 20:46 | |
| pabelanger | clarkb: basically, we are seeing limiting launch failures in osic-cloud1 because nodepool didn't know which launcher was used to launch the ndoe | 20:46 |
| pabelanger | node* | 20:46 |
| pabelanger | I don't understand why the data would be missing in gearman | 20:46 |
| clarkb | pabelanger: huh, gearman does run inthe clear so one possible way to debug that is with tcpdump | 20:47 |
| clarkb | pabelanger: I think you can tell tcpdump to record a rolling log so ever X bytes it makes a new file then can delete old ones | 20:47 |
| clarkb | pabelanger: you could set it up to do that then wait for that to show up in nodepool then go find it in the pcap file | 20:47 |
| pabelanger | clarkb: jeblair it could also explain why some of our jobs have 'Failed to connect to the host via ssh' from ansible, since nodepool was the one who deleted the node | 20:48 |
| clarkb | pabelanger: essentially that is saying there was no data in the assignment packet I think | 20:48 |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Remove whitespaces and add parenthesis to print statements https://review.openstack.org/360080 | 20:48 |
| pabelanger | clarkb: right, which nodepool then deletes the node | 20:49 |
| *** vhosakot has quit IRC | 20:49 | |
| pabelanger | clarkb: I'll see if I can get a pcap going | 20:49 |
| *** vhosakot has joined #openstack-infra | 20:50 | |
| *** roxanag__ has quit IRC | 20:51 | |
| clarkb | pabelanger: tcpdump -s 1500 -i eth0 -w gearman.pcap-%Y-%m-%d_%H:%M:%S -W 10 -C 100 port 4730 | 20:51 |
| clarkb | pabelanger: something like that may work | 20:51 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 20:51 |
| *** Goneri has quit IRC | 20:52 | |
| jeblair | phschwartz: can you paste the log lines from the time zuul started until "Reconfiguration complete" ? | 20:52 |
| clarkb | that should save a gigabyte of pcap data in a rotation of 10 files | 20:52 |
| phschwartz | jeblair: yeah, will be a couple of moments | 20:53 |
| *** amitgandhinz has quit IRC | 20:54 | |
| *** amitgandhinz has joined #openstack-infra | 20:55 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove initial resources creation from infracloud controller manifest https://review.openstack.org/360061 | 20:55 |
| fungi | clarkb: on nodepool hold reasons, you can add/update them later (i often hold and then repeat the same command but add a --reason, so i can have a better shot at winning the race) | 20:56 |
| clarkb | 2016-08-24 20:56:35.333402 | [Zuul] Job complete, result: SUCCESS | 20:57 |
| clarkb | I am going to enqueue/promote now | 20:57 |
| clarkb | fungi: huh didn't know that | 20:57 |
| *** _ari_ has quit IRC | 20:57 | |
| fungi | clarkb: yeah, it's sort of a cheat, but nodepool hold holds a node in any state, including ones in a held state. so holding again with a --reason just replaces the old (possibly empty) reason when it overwrites the status | 20:58 |
| *** claudiub has joined #openstack-infra | 20:58 | |
| pabelanger | clarkb: thanks, running on nodepool in /tmp folder | 20:58 |
| fungi | clarkb: well, and also updates the held timestamp (so effectively resets the age in state) | 20:59 |
| *** jkilpatr has quit IRC | 21:00 | |
| clarkb | pabelanger: you might want to double check you are getting what you want after a few megabytes. Seems like I alwys have to fiddle with tcpdump in order to get what I need | 21:00 |
| clarkb | ok devstack change is promoted | 21:00 |
| *** tqtran has joined #openstack-infra | 21:01 | |
| phschwartz | jeblair: hmm, looking at the log I sent these from, it says the Reconfiguration complete is almost 230k lines into the file. | 21:01 |
| haleyb | clarkb: that doesn't need cherry-picking, right? | 21:01 |
| clarkb | haleyb: no, any job starting behind it in the gate will run with it and once it merges any jobs starting in the check queue will get it | 21:02 |
| *** roxanagh_ has joined #openstack-infra | 21:03 | |
| jeblair | phschwartz: i meant from the time of the restart... so the most recent set of lines between "Performing reconfiguration" and "Reconfiguration complete" | 21:03 |
| jeblair | phschwartz: (most recest set of those lines before the error) | 21:03 |
| jeblair | clarkb, haleyb: and any check jobs that "Depends-On" that change as well | 21:04 |
| haleyb | clarkb: i guess i was thinking of the stable devstack branches | 21:04 |
| clarkb | haleyb: oh yes stable will liekly need it as well | 21:04 |
| phschwartz | jeblair: http://paste.openstack.org/show/563140/ http://paste.openstack.org/show/563141/ | 21:07 |
| phschwartz | jeblair: 598 lines so split it | 21:07 |
| nibalizer | yah at least one keyserver has figured out that I signed the key http://keyserver.ubuntu.com/pks/lookup?op=vindex&search=openstack+infra&fingerprint=on | 21:07 |
| *** Hal has joined #openstack-infra | 21:08 | |
| robcresswell | When patches in a gate chain fail, do latter patches just get restarted? Horizon has a 24hr patch that got kicked back to Queueing :) | 21:08 |
| jeblair | phschwartz: that does not have the change from nibalizer to add requirements to the layout. | 21:08 |
| phschwartz | jeblair: yeah, we had an issue getting the server to pull that one update. But my concern is when this issue happens we are getting flooded with so many errors that in a little over 24 hours it was 22.4g of logs | 21:10 |
| *** edtubill has joined #openstack-infra | 21:10 | |
| jeblair | phschwartz: my concern is trying to fix the bug | 21:10 |
| SpamapS | jeblair: guessing it hasn't reloaded the layout just yet, because it's too busy in that loop | 21:10 |
| jeblair | phschwartz: i'm still trying to elucidate the circumstances around it | 21:10 |
| SpamapS | OR, the layout hasn't been delivered yet. :) | 21:10 |
| SpamapS | because some moron forgot to add his auth keys file ;) | 21:11 |
| jeblair | SpamapS: it's effectively never going to exit that loop and needs to be restarted | 21:11 |
| SpamapS | jeblair: that was my fear, makes sense. | 21:11 |
| phschwartz | jeblair: basically any time there is a depends-on in a review and that review it depends on is for a project not in the layout, it goes fubar | 21:11 |
| *** vhosakot has quit IRC | 21:11 | |
| jeblair | phschwartz: right, but as i said yesterday, that case is not only handled, but tested, so i'm trying to figure out what's different | 21:11 |
| jeblair | which is why i'm trying to find every detal | 21:11 |
| jeblair | whether the project is in the layout (i just learned it is not) is an important one | 21:11 |
| *** vhosakot has joined #openstack-infra | 21:11 | |
| fungi | it's a configurable option, right? | 21:13 |
| clarkb | fungi: re the keep zuul env up to date change. Do you think we need to use pip install --upgrade? I am trying to remember under which circumstances pip won't update a package | 21:13 |
| asselin__ | Shrews, have you ever seen this error before? http://paste.openstack.org/show/563144/ | 21:13 |
| clarkb | fungi: I just did a local test of zuul 2.0 install upgraded to latest master and that worked without -U | 21:13 |
| rcarrillocruz | hey folks, i just noticed there are unmerged changes on the puppetmaster private hiera | 21:14 |
| phschwartz | jeblair: so, I have been able to confirm that the issue is purely related to when the project from a depends-on is not in the layout | 21:14 |
| rcarrillocruz | a massive chmod on all files | 21:14 |
| phschwartz | jeblair: we missed some and we are trying to fix that | 21:14 |
| pabelanger | rcarrillocruz: Oh, that could be puppet | 21:14 |
| rcarrillocruz | anyone doing that? | 21:14 |
| pabelanger | 1 sec | 21:14 |
| jeblair | fungi: that would do it | 21:14 |
| clarkb | mordred: dstufft do you have a tl;dr for when -U is necessary when running pip install? is it just when you want newer pacakges that are available when older ones match requirements? | 21:15 |
| rcarrillocruz | ah k, i was about to do a change, but wanted to give a nudge to whoever to commit those | 21:15 |
| phschwartz | jeblair: but is a worring issue due to the volume of the errors going into the log. | 21:15 |
| pabelanger | rcarrillocruz: https://review.openstack.org/#/c/326649/ | 21:15 |
| dstufft | clarkb: yes | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | dstufft: tyty | 21:15 |
| pabelanger | rcarrillocruz: the issue is 0750? | 21:15 |
| dstufft | clarkb: specfically, when the one you already have installed matches requirements, but you want a newer one that is available | 21:15 |
| rcarrillocruz | yah | 21:15 |
| pabelanger | or they change to 0640 | 21:15 |
| jeblair | phschwartz: one error is too many | 21:15 |
| rcarrillocruz | old mode 100644 | 21:15 |
| clarkb | dstufft: ya that makes sense. thanks | 21:15 |
| rcarrillocruz | new mode 100755 | 21:15 |
| jeblair | fungi: i don't see an option for it | 21:16 |
| rcarrillocruz | pabelanger: do you want to commit that or should I and link to thta change | 21:16 |
| rcarrillocruz | ? | 21:16 |
| fungi | jeblair: yeah going back through the manual i'm missing it as well | 21:16 |
| pabelanger | rcarrillocruz: ya, I think we should | 21:16 |
| rcarrillocruz | k, gimme a sec | 21:16 |
| SpamapS | phschwartz jeblair: do you want to maybe move this to #zuul? | 21:16 |
| *** vhosakot has quit IRC | 21:17 | |
| * SpamapS doesn't want to interfere with infra business | 21:17 | |
| *** salv-orl_ has joined #openstack-infra | 21:17 | |
| jeblair | we can do that | 21:17 |
| *** matt-borland has quit IRC | 21:17 | |
| rcarrillocruz | pabelanger: done | 21:18 |
| JayF | JFYI gerrit is throwing 503s | 21:18 |
| JayF | "Service Unavailable" in the little grey popup window; just started about a minute ago | 21:18 |
| *** piet has quit IRC | 21:19 | |
| * rcarrillocruz just had a nostalgia moment for seeing hpcloud hiera keys on puppetmaster | 21:19 | |
| *** jheroux has quit IRC | 21:19 | |
| *** salv-orlando has quit IRC | 21:20 | |
| pabelanger | rcarrillocruz: thanks | 21:20 |
| rcernin | is it just me having problems with connecting to "https://review.openstack.org" | 21:20 |
| pabelanger | clarkb: confirmed tcpdump works as expected, thanks for the syntax | 21:20 |
| cloudnull | rcernin: +1 | 21:21 |
| rcernin | :( | 21:21 |
| cloudnull | looks like review.o.o is upset | 21:21 |
| rcernin | thanks | 21:21 |
| clarkb | ya I am trying to pull up its internal monitoring tools to see if it is in garbage collection pain | 21:21 |
| *** thorst_ has quit IRC | 21:21 | |
| *** roxanagh_ has quit IRC | 21:21 | |
| rcarrillocruz | pabelanger: in the clouds layout i don't see a user creation for bluebox | 21:21 |
| rcarrillocruz | was that created manually, for the openstackci/openstackzuul projects ? | 21:21 |
| rcarrillocruz | i.e. out of band , not with the launcher | 21:22 |
| asselin__ | rcarrillocruz, what is 'that' you're referring to? | 21:22 |
| clarkb | it doesn't look too bad on the memory front (though it appears to be getting clsoe to where things go south) | 21:22 |
| *** rhallisey has quit IRC | 21:23 | |
| *** gouthamr has quit IRC | 21:23 | |
| clarkb | rcernin: cloudnull http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=30&rra_id=all theres the problem I think | 21:23 |
| *** baoli_ has quit IRC | 21:23 | |
| clarkb | we need to update our apache to allow for more connections, its using the default apache mpm event total | 21:23 |
| fungi | oh yikes | 21:24 |
| fungi | probably could stand to see where that spike came from too | 21:24 |
| *** jkilpatr has joined #openstack-infra | 21:24 | |
| clarkb | gerrit itself has an fd limit of 8192 | 21:24 |
| clarkb | and it uses a bunch of those for jgit things. Somewhere in the range of 2k is probably safe | 21:24 |
| fungi | yeah, apache probably needs a pair to proxy every connection too | 21:25 |
| rcarrillocruz | asselin__: https://review.openstack.org/#/c/359807/1/cloud_launcher/clouds_layouts.yml | 21:25 |
| clarkb | haleyb: in fact I think we may need the backports for grenade to work properly too. Any chacne you pushed those yet? | 21:25 |
| rcarrillocruz | there is no user creation there | 21:25 |
| *** piet has joined #openstack-infra | 21:25 | |
| clarkb | haleyb: I can promote them as well once the master chagne gets through | 21:25 |
| clarkb | fungi: good point | 21:26 |
| fungi | clarkb: we're making 2984 connections to mysql according to netstat. that can't be right... | 21:26 |
| openstackgerrit | Merged openstack-infra/system-config: Update zuul-env on job nodes https://review.openstack.org/359352 | 21:26 |
| clarkb | fungi: wow | 21:26 |
| fungi | and climbing | 21:26 |
| asselin__ | rcarrillocruz, sorry, I don't have the context. Which comment of mine are you referring too? I don't think I'm using that file you referenced....but need to double-check | 21:26 |
| zaro | i think there's a bug with gerrit that it doesn't close stale conections | 21:26 |
| fungi | i wonder if there's a trove or backend networking issue in rackspace dfw right now | 21:26 |
| clarkb | fungi: fwiw I don't think the number of fds on the gerrit process has been the issue. apache is jsut capping us off at a very conservative level compared to what gerrit should be able to handle | 21:26 |
| clarkb | zaro: could that be the source of the leak too? | 21:27 |
| fungi | netstat -nt|grep -c :3306 | 21:27 |
| fungi | 4237 | 21:27 |
| clarkb | ouch | 21:27 |
| rcarrillocruz | i'm not commenting to any comment of you, i was talking to pabelanger about infra's puppetmaster :-) | 21:27 |
| anteaya | would this be a runaway third party ci? | 21:27 |
| asselin__ | rcarrillocruz, oh sorry...my bad | 21:28 |
| anteaya | yes: https://review.openstack.org/#/c/358065/ | 21:28 |
| zaro | here's the fix for it, https://git.openstack.org/cgit/openstack-infra/gerrit/commit/?h=upstream/stable-2.11&id=f57d2b8f830710d6e25d501f4dc8d76f2d2e71f2 | 21:28 |
| anteaya | fuel ci | 21:28 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add openstackci/openstackzuul oscc clouds to all-clouds https://review.openstack.org/360093 | 21:28 |
| fungi | anteaya: nah, that's how fuel-ci works | 21:28 |
| zaro | clarkb: maybe, but i think we restart gerrit enough to not have lingering connections though | 21:28 |
| anteaya | really? | 21:28 |
| anteaya | wow | 21:28 |
| fungi | it posts a comment for every job it starts on a change | 21:29 |
| fungi | however, it only tests fuel changes, so nobody's complained | 21:29 |
| anteaya | frightening | 21:29 |
| anteaya | okay | 21:29 |
| clarkb | zaro: we only restart it that often beceause of the leak though... | 21:29 |
| fungi | anyway, i've run some stats on the gerrit logs and don't see an excessive comment volume coming from anywhere | 21:29 |
| clarkb | zaro: so if it is teh cause of the leak that is how we are working around it | 21:29 |
| anteaya | fungi: okay thank you | 21:29 |
| fungi | whatever the spike on the graph was, it's subsided again | 21:30 |
| *** hashar has quit IRC | 21:30 | |
| clarkb | but I think the 503s happen when we hit that limit and apache has to say go away | 21:31 |
| anteaya | heck of a spike | 21:31 |
| clarkb | before that it will queue things so when people say gerrit is slow I think that is what is happening. Its actually their connection being parked on the proxy before being handled | 21:31 |
| *** vinaypotluri has quit IRC | 21:31 | |
| fungi | there was a large corresponding outbound traffic spike corresponding with the open connections spike | 21:31 |
| zaro | clarkb: well i don't know if that's is causing the memory issue, maybe we should check the ssh sessions over time? | 21:32 |
| *** thiagop has quit IRC | 21:32 | |
| zaro | or does cacti have that info? | 21:32 |
| clarkb | cacti only has the system level info | 21:33 |
| rcarrillocruz | clarkb: when you said yesterday about doing performance testing, did you mean scping a dib image from nodepool to the controller, upload to glance, spin a vm and do a reproduce.sh from some random job there ? | 21:33 |
| clarkb | which shows us at a pretty steady 400ish tcp connections during normal operations | 21:33 |
| zaro | well i guess there's the gerrit show-sessions command | 21:33 |
| fungi | also a largeish but not huge corresponding inbound spike on the internal nic, suggesting it was perhaps reading a lot of whatever it was serving from the database | 21:33 |
| clarkb | rcarrillocruz: yup basically. though you don't have to scp it to the controlelr first you should be able to upload directly from nodepool using openstackclient | 21:33 |
| pabelanger | rcarrillocruz: yes, I manually created them, because passwords | 21:33 |
| zaro | clarkb: maybe put that on a watch? | 21:33 |
| pabelanger | rcarrillocruz: we need to solve that | 21:34 |
| rcarrillocruz | clarkb: ah yeah, need to put the clouds on nodepool for infracloud tho | 21:34 |
| rcarrillocruz | i have put them just on puppetmaster for now | 21:34 |
| haleyb | clarkb: armax was working on the backport, need to review | 21:34 |
| rcarrillocruz | thx | 21:34 |
| clarkb | rcarrillocruz: or just have one in your homdir or something | 21:34 |
| rcarrillocruz | pabelanger: oh yeah, the stuff you asked about vault | 21:35 |
| rcarrillocruz | ++ | 21:35 |
| *** jamesdenton has quit IRC | 21:35 | |
| *** piet has quit IRC | 21:36 | |
| clarkb | fungi: I wonder if its someone doing a bunch of api queries? | 21:37 |
| clarkb | stackalytics maybe? | 21:37 |
| *** guilherme has joined #openstack-infra | 21:37 | |
| fungi | clarkb: it's possible but will need someone to have time to do some focused analysis of the apache logs | 21:37 |
| clarkb | in any case we should probably plan to increase the number of connectiosn apache will allow since I think the server can do more comfortably based on memory and cpu logs | 21:37 |
| fungi | agreed | 21:38 |
| clarkb | we are just queuing when we don't need to | 21:38 |
| fungi | yep, looks like the server's powerful enough to abosrb it if only it were configured to try | 21:38 |
| *** guilherme has quit IRC | 21:39 | |
| * clarkb doesn't want to get distracted from getting these devstack changes through, but can probably look at that tomorrow at some point | 21:41 | |
| zaro | how was the review.o.o CPU utilization during that time? | 21:41 |
| *** mdrabe has quit IRC | 21:41 | |
| clarkb | zaro: its pretty low like 10% | 21:41 |
| clarkb | zaro: you can see it on http://cacti.openstack.org | 21:41 |
| zaro | ok. there's a bug that will cause cpu to pin on certain queries. | 21:41 |
| clarkb | oh I guess it jumped to ~50% during the peak | 21:42 |
| *** gouthamr has joined #openstack-infra | 21:42 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 21:42 |
| zaro | https://review.openstack.org/#/c/356617/ | 21:42 |
| clarkb | still not too bad with room to breath if we increase the connection limits | 21:42 |
| zaro | yeah, that doesn't look like it would cause unreponsiveness | 21:43 |
| clarkb | zaro: ya but should be fixed anyways as I know more people are using those types of queries to track work. +2 | 21:44 |
| clarkb | fungi: maybe you want to look at ^ and we can get gerrit updated when we rename things? | 21:45 |
| zaro | maybe should include the ssh timeout fix too? | 21:46 |
| *** vinaypotluri has joined #openstack-infra | 21:46 | |
| *** sdake has joined #openstack-infra | 21:48 | |
| haleyb | clarkb: the mitaka and liberty patches are up, but since the code has changed they are not exact picks - https://review.openstack.org/#/c/359974/ and https://review.openstack.org/#/c/359975/ | 21:49 |
| *** roxanagh_ has joined #openstack-infra | 21:49 | |
| clarkb | haleyb: awesome. I am not devstack core but I think ianw jeblair mtreinish sdake and sc68cal are | 21:49 |
| *** sdake_ has joined #openstack-infra | 21:50 | |
| *** thorst_ has joined #openstack-infra | 21:51 | |
| *** aeng has quit IRC | 21:51 | |
| *** weshay is now known as weshay_bbiab | 21:53 | |
| ianw | clarkb: if sdague sc68cal doesn't jump on them, happy to push them in if consensus here is they're good | 21:53 |
| clarkb | ianw: the master change has been approved and is at the top of the gatequeue. As long as you are happy with the merge delta I say go for it | 21:54 |
| *** sdake has quit IRC | 21:54 | |
| *** thorst_ has quit IRC | 21:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a wiki-dev.o.o server to test newer mediawiki https://review.openstack.org/358246 | 21:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Set wiki name and logo URL https://review.openstack.org/360037 | 21:56 |
| *** rvasilets___ has quit IRC | 21:57 | |
| *** ddieterly is now known as ddieterly[away] | 21:59 | |
| *** Hal has quit IRC | 21:59 | |
| *** adriant has joined #openstack-infra | 21:59 | |
| rcarrillocruz | pabelanger: mind reviewing https://review.openstack.org/#/c/360093/ ? | 22:01 |
| ianw | haleyb: just a question on if the iptables stuff was deliberately left out of -> https://review.openstack.org/#/c/359975/3 | 22:01 |
| *** Hal has joined #openstack-infra | 22:01 | |
| *** sdake_ has quit IRC | 22:01 | |
| ianw | haleyb: i guess same question for the mikata one | 22:02 |
| *** rfolco has quit IRC | 22:03 | |
| *** sdake has joined #openstack-infra | 22:03 | |
| pabelanger | rcarrillocruz: looks right | 22:04 |
| haleyb | ianw: i didn't think those were backported, don't know why, but could be done in a separate patch | 22:04 |
| rcarrillocruz | thx | 22:04 |
| pabelanger | rcarrillocruz: also, think I am going to try my hand at a lets encrypt playbook | 22:05 |
| rcarrillocruz | oh? | 22:05 |
| rcarrillocruz | so people's cool with let's encrypt ? | 22:05 |
| rcarrillocruz | i remember cody or someone bringing it up in a meeting and people wasn't very excited about it | 22:05 |
| rcarrillocruz | i'm all over automating all things :-) | 22:05 |
| pabelanger | I think as long as we have it 100% automated | 22:06 |
| ianw | haleyb: sorry, not following ... i'm but a dilettante at neutron+ipv6 ... the sysctl stuff looks like it moved over fine, just wondering where the iptables fiddling comes into it, because it's not on the mikata and liberty patches | 22:06 |
| haleyb | ianw: oh, it was just not to duplicate code - that iptables call was needing the same interface value | 22:07 |
| fungi | rcarrillocruz: pabelanger: i'm not really keen on relying on letsencrypt in production, but if you want to play around with it be my guest | 22:07 |
| ianw | haleyb: ooohhhh, ok, yeah, i see that now | 22:08 |
| haleyb | ianw: someone pointed out we had just done this for iptables, so i just copied it | 22:08 |
| pabelanger | fungi: Ya, I wanted to give it a try first. See what can be done | 22:09 |
| *** Hal has quit IRC | 22:09 | |
| ianw | haleyb: cool. no worries, as expected sdague jumped on it anyway, so all good | 22:09 |
| pabelanger | fungi: however, it would be nice to not have a self-signed cert if possible. Even if a wildcard for ic.openstack.org | 22:09 |
| clarkb | I will promote those as soon as the master one finishes (that should happen soon) | 22:09 |
| haleyb | i hope it works on the stable branches the same way... | 22:09 |
| fungi | pabelanger: rcarrillocruz: if enough of the rest of our root team are comfortable with it then i'm just as happy to let them take that on, but i have some fundamental issues with how letsencrypt is implemented that make it not something i personally want to have to deal with | 22:10 |
| pabelanger | fungi: Ya, I don't want to force it if people object | 22:10 |
| *** notmyname has quit IRC | 22:10 | |
| rcarrillocruz | fungi: i guess we can do a POC | 22:11 |
| ianw | are we going to be the type of people who hit letsencrypt's renewal limits? | 22:11 |
| rcarrillocruz | fungi: but i have two personal projects i'd like to tackle at some point | 22:11 |
| ianw | although i guess you can ask for exceptions | 22:11 |
| rcarrillocruz | 1. put all infra in repo and have automation to automatically deploy it | 22:11 |
| clarkb | haleyb: telnet://2001:4800:1ae1:18:f816:3eff:fe27:9a2d:19885 (mitaka) and telnet://2001:4800:1ae1:18:f816:3eff:fe3b:80f0:19885 (liberty) should tell us | 22:11 |
| pabelanger | ianw: if we had wildcard, we can do up to 2000 hosts | 22:11 |
| pabelanger | per week | 22:12 |
| clarkb | I will keep an eye on those and make sure they get to tempest before enqueing and promoting to the gate | 22:12 |
| rcarrillocruz | 2. move over to designate (yeah, i'd need you or clark to check how the new designate looks like in RAX and if the other folks at the foundation are comfortable with the UI) | 22:12 |
| pabelanger | https://letsencrypt.org/docs/rate-limits/ | 22:12 |
| fungi | ianw: not sure what the renewal limits are... we'd presumably need to renew all certificates once every month or two so as to avoid exceeding the imposed three-month expiration (whereas right now i get certs good for 2 years, so an order of magnitude more frequent renewing) | 22:12 |
| ianw | pabelanger: i thought let'sencrypt was no wildcards? that seems to be the compalint every time it comes up on hackernews | 22:12 |
| fungi | ianw: though it does seem to allow subjectaltnames | 22:13 |
| clarkb | haleyb: actually they have both already started tempest so just need the master chagne to merge then I can enqueue and promote them | 22:13 |
| fungi | ianw: so could be worked around that way if needed | 22:13 |
| *** vhosakot has joined #openstack-infra | 22:13 | |
| pabelanger | ianw: it looks like they limit you to 100 Names per Certificate | 22:14 |
| pabelanger | up to 20 a week is your 2000 | 22:14 |
| *** roxanagh_ has quit IRC | 22:14 | |
| fungi | yeah, we're probably not in danger of exceeding that at least | 22:15 |
| haleyb | clarkb: cool, i'll stay online until they merge | 22:15 |
| clarkb | looks like master just merged | 22:15 |
| *** roxanagh_ has joined #openstack-infra | 22:15 | |
| clarkb | promoting now | 22:16 |
| *** sileht has quit IRC | 22:16 | |
| fungi | pabelanger: ianw: we have about two dozen certs in circulation at the moment, and the one with the most subjectaltnames has maybe a dozen | 22:16 |
| fungi | so we _would_ need to stagger renewals i think, couldn't fire them all in the same week fwiw | 22:17 |
| *** ddieterly[away] is now known as ddieterly | 22:17 | |
| clarkb | and thats done \o/ now just to get them merged | 22:17 |
| *** esberglu has quit IRC | 22:17 | |
| pabelanger | fungi: ya, we'd need some rotation schedule | 22:17 |
| clarkb | once they merge we should see the rate of ansible rc 3's fall off on the zuul launchers | 22:17 |
| *** aeng has joined #openstack-infra | 22:17 | |
| pabelanger | ++ | 22:17 |
| *** sileht has joined #openstack-infra | 22:17 | |
| fungi | pabelanger: but that goes back to my primary worry... complexity+automation=outages | 22:17 |
| openstackgerrit | Chris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available https://review.openstack.org/359471 | 22:17 |
| pabelanger | fungi: Ya, valid concern too | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Always create foreign projects if needed https://review.openstack.org/360105 | 22:19 |
| jeblair | SpamapS, phschwartz, fungi: ^ | 22:19 |
| jeblair | asselin__: ^ that may be of interest to third-party ci ops | 22:19 |
| fungi | right now, while our cert management is manual in nature and bears a nontrivial cost, there is a human involved in the process able to double-check that things are correct and fix/undo them if not. turn that over to a daemon and wait for your services to periodically go offline until someone's around to fix certificate renewal issues on them | 22:20 |
| fungi | jeblair: awesome, thanks | 22:20 |
| mordred | clarkb: ooh - I was afk for a bit ... the devstack fix wound up being the fix? | 22:20 |
| clarkb | mordred: yup but we had to do it to the specific interface and actually get the interface correct | 22:20 |
| fungi | mordred: yeah, just had to get moved around a little, sounds like | 22:20 |
| mordred | ahhh. neat. WOOT! | 22:21 |
| mordred | that's super exciting | 22:21 |
| clarkb | pabelanger: fungi another thing to keep in mind with letsencrypt is you have to run the renewal tool as root | 22:21 |
| clarkb | because why? who knows | 22:21 |
| clarkb | mordred: the master change just merged and I have put the stable branch fixes at the top of the queue so hopefllyt in the next hour or so this will be a thing of the past | 22:21 |
| fungi | that sounds solvable | 22:21 |
| pabelanger | clarkb: neat, didn't know that | 22:21 |
| fungi | the as root issue i mean | 22:22 |
| clarkb | fungi: oh I am sure, its just a thing that doesn't make me confident in the tooling | 22:22 |
| *** rfolco has joined #openstack-infra | 22:22 | |
| clarkb | (it wants to listen on port 443 or something to make sure you really own the ip:port combo? I dunno | 22:22 |
| jeblair | clarkb, fungi, pabelanger: you don't need to turn over management of apache to letsencrypt. it's just an api, and you can use tools to get the certs in a 'normal' way. | 22:22 |
| *** esikachev has quit IRC | 22:22 | |
| *** rfolco has quit IRC | 22:22 | |
| fungi | i also have philosophical/community concerns with the way letsencrypt came about, but i'm willing to keep my criticisms to technical issues | 22:22 |
| *** xarses has quit IRC | 22:23 | |
| *** fguillot has joined #openstack-infra | 22:23 | |
| pabelanger | Ya, having letsencrypt touch apache would be a no go for me | 22:23 |
| fungi | because not everybody is as bitter about the state of the certificate vending machine industry and browser collusion as i am | 22:23 |
| *** fguillot has quit IRC | 22:24 | |
| jeblair | i think jamielennox had some ansible stuff to use le in a less-insane manner; we might ask about that if we decide to go down that route. | 22:24 |
| clarkb | jeblair: correct, aiui even if it doesn't touch apache it needs or wants root | 22:24 |
| mordred | clarkb: _excellent_ | 22:24 |
| *** ddieterly has quit IRC | 22:24 | |
| ianw | fungi: so, how about we connect an arduino based electric buzzer up to your new firehose and insert it in your pants, it can monitor events such that it makes the voltage inversely proportionate to the time left before renewal of the letsencrypt certificates? keep a human in the loop :) | 22:24 |
| fungi | ianw: compelling | 22:25 |
| jeblair | plus, fungi would have a superhuman sense of cert expiration times | 22:25 |
| jhesketh | Morning | 22:25 |
| openstackgerrit | Merged openstack-infra/system-config: Add openstackci/openstackzuul oscc clouds to all-clouds https://review.openstack.org/360093 | 22:25 |
| fungi | yeah, after a while i could take off the training pants and just preternaturally *know* when certs are about to expire | 22:25 |
| mmedvede | jeblair: I just had my zuul explode yesterday with unkown project bug, filling up 40G with logs, thanks for the https://review.openstack.org/360105 | 22:26 |
| jeblair | mmedvede: well, that answers my question of "why aren't other folks seeing this?" :) | 22:26 |
| *** nwkarsten has quit IRC | 22:27 | |
| mmedvede | I should hang out in infra channel more :) | 22:27 |
| jeblair | that's probably going to be worth a 2.5.1 release too | 22:28 |
| mordred | ++ | 22:28 |
| mmedvede | that would be appreciated. Currently pinning to 2.5.0 | 22:29 |
| *** markvoelker has joined #openstack-infra | 22:29 | |
| *** fguillot has joined #openstack-infra | 22:29 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-log_processor: Give each job's console log its own crm classifier https://review.openstack.org/338558 | 22:32 |
| SpamapS | mmedvede: wow really? Interesting. | 22:34 |
| SpamapS | here we thought we were the only ones dumb enough to add unknown projects to dependency chains ;) | 22:35 |
| *** hockeynut has quit IRC | 22:35 | |
| *** yamahata has quit IRC | 22:36 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/log_processor: Give each job's console log its own crm classifier https://review.openstack.org/360112 | 22:36 |
| *** Julien-zte has joined #openstack-infra | 22:38 | |
| *** yamahata has joined #openstack-infra | 22:38 | |
| anteaya | morning jhesketh | 22:39 |
| *** fguillot has quit IRC | 22:40 | |
| * anteaya laughs at the vision of fungi in hawaiian shirts and buzzard pants | 22:40 | |
| fungi | well, they'd just be bermuda shorts with some custom wiring | 22:40 |
| anteaya | ha ha ha | 22:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix legend for 'Time to Ready' for nodepool dashboard https://review.openstack.org/360113 | 22:41 |
| fungi | that is to say, not at all wired the way you'd normally wire a pair of bermuda shorts | 22:41 |
| clarkb | do you normally wire bermuda shorts? | 22:42 |
| anteaya | the way I'd wire my bermuda shorts for instance | 22:42 |
| fungi | shhh | 22:42 |
| pleia2 | hehe | 22:42 |
| *** xarses has joined #openstack-infra | 22:43 | |
| *** fguillot has joined #openstack-infra | 22:46 | |
| *** yamamoto has joined #openstack-infra | 22:48 | |
| *** thorst has joined #openstack-infra | 22:48 | |
| *** vhosakot has quit IRC | 22:50 | |
| clarkb | an instance in osic running grenade behind the two devsatck backports is looking good. It got through the first stack run and is now upgrading services | 22:52 |
| *** rcernin has quit IRC | 22:53 | |
| anteaya | yay! | 22:54 |
| *** marcusvrn_ has quit IRC | 22:54 | |
| haleyb | clarkb: great. i do have to take off now though, will check-in later | 22:55 |
| *** rbrndt has quit IRC | 22:55 | |
| *** sheeprine has quit IRC | 22:55 | |
| *** rlandy is now known as rlandy|bbl | 22:55 | |
| *** zigo has quit IRC | 22:56 | |
| anteaya | thanks haleyb | 22:56 |
| *** sdake has quit IRC | 22:57 | |
| *** zigo has joined #openstack-infra | 22:57 | |
| *** nwkarsten has joined #openstack-infra | 22:58 | |
| *** sheeprine has joined #openstack-infra | 22:58 | |
| clarkb | and now it is tempesting so I think we should be set once these merge. It will be easyish for infra to check tomorrow by grepping zuul launcher logs | 22:59 |
| *** fguillot has quit IRC | 23:00 | |
| *** zz_dimtruck is now known as dimtruck | 23:00 | |
| jamielennox | jeblair: there are some alternative clients out there for letsencrypt that i think would be easier to automate than the official certbot one if you're looking around | 23:02 |
| jamielennox | the certbot config files are kind of crazy and unnecessary | 23:02 |
| *** fguillot has joined #openstack-infra | 23:03 | |
| *** markvoelker has quit IRC | 23:04 | |
| *** cody-somerville has joined #openstack-infra | 23:05 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix ready node launch attemps panel for nodepool https://review.openstack.org/360118 | 23:05 |
| nibalizer | ssss | 23:05 |
| clarkb | woot that grenade job just passed | 23:05 |
| *** pradk has quit IRC | 23:06 | |
| *** hongbin has quit IRC | 23:06 | |
| anteaya | yay! | 23:07 |
| *** sdake has joined #openstack-infra | 23:07 | |
| *** tpsilva has quit IRC | 23:08 | |
| *** tqtran has quit IRC | 23:10 | |
| *** oomichi has quit IRC | 23:11 | |
| *** oomichi has joined #openstack-infra | 23:13 | |
| *** apetrich has quit IRC | 23:18 | |
| *** apetrich has joined #openstack-infra | 23:18 | |
| clarkb | and they just merged, so any job in the gate now or in check starting after nowish should be good | 23:19 |
| *** shashank_hegde has quit IRC | 23:19 | |
| pabelanger | nice work everybody | 23:20 |
| *** edmondsw has quit IRC | 23:20 | |
| pabelanger | also happy we didn't revert to ipv4 | 23:20 |
| *** kzaitsev_mb has quit IRC | 23:20 | |
| clarkb | I started a tail on zl01 that may pop up a few last stragglers from teh check queue but within an hour we should see the error fall off to zero | 23:21 |
| mordred | pabelanger: ++ | 23:21 |
| phschwartz | jeblair: that patch looks great, ty for the help. | 23:21 |
| clarkb | pabelanger: is the ubuntu mirror looking happy again? I am going to guess that it is due to the number of happy jobs out there | 23:23 |
| clarkb | pabelanger: btu I think that was the only other known issue? | 23:24 |
| clarkb | oh wait the ssh key issue is still outstanding I Think | 23:24 |
| pabelanger | clarkb: yes, we should be good now | 23:24 |
| pabelanger | which ssh key issue? | 23:24 |
| mordred | what's the ssh key issue? | 23:25 |
| zaro | ianw: ping | 23:26 |
| jroll | is the ssh thing going on from everywhere or specific locations? | 23:28 |
| pabelanger | clarkb: mordred: https://review.openstack.org/#/c/360057/ only change I proposed today for SSH. To help debug ssh_host_key changing | 23:28 |
| * jroll sees pabelanger in #rackspace | 23:28 | |
| pabelanger | jroll: which ssh thing? The question I asked in #rackspace? | 23:28 |
| jroll | pabelanger: yeah | 23:28 |
| *** fguillot has quit IRC | 23:29 | |
| pabelanger | jroll: seems to be limited to rax-iad | 23:29 |
| pabelanger | seeing a lot of SSH timeouts there | 23:29 |
| jroll | pabelanger: yeah, but only coming from zuul or can you reproduce from everywhere? | 23:29 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add sshnaidm to tripleo-cd-admins https://review.openstack.org/357217 | 23:29 |
| pabelanger | jroll: no, I couldn't access the IP from my local internet either | 23:30 |
| clarkb | mordred: tl;dr is ansible complains about the host key changing implying mitm or similar | 23:30 |
| *** hockeynut has joined #openstack-infra | 23:30 | |
| mordred | clarkb: oh. yeah. I saw discussion of that | 23:30 |
| jroll | pabelanger: ok. what's the failure rate? | 23:30 |
| oomichi | pleia2: hi, thanks for reviewing on https://review.openstack.org/#/c/358148/ | 23:30 |
| clarkb | pabelanger: +2 for the zl change | 23:31 |
| oomichi | pleia2: it is fine to drop that if not interested in | 23:31 |
| mordred | pabelanger: me too | 23:31 |
| pleia2 | oomichi: sure, I know there are a couple others to look at, but I'm not really certain about the logic in those (why we're doing the thing you're changing) | 23:31 |
| clarkb | and with that I need to go grocery shopping | 23:31 |
| *** xyang1 has quit IRC | 23:31 | |
| *** krtaylor has joined #openstack-infra | 23:32 | |
| pleia2 | oomichi: like, I'm not sure if there's a reason for having the for loop in https://review.openstack.org/#/c/358147/ before the exception | 23:32 |
| pleia2 | s/exeption/except | 23:33 |
| *** fguillot has joined #openstack-infra | 23:33 | |
| oomichi | pleia2: At changed code, line 60-65 never raises IOError. | 23:34 |
| *** nwkarsten has quit IRC | 23:34 | |
| pleia2 | ah, hm | 23:34 |
| pabelanger | clarkb: other issue I am tracking is gearman missing data, tcpdump in progress but haven't had the IndexError again | 23:35 |
| jroll | pabelanger: poking some internal support folks | 23:35 |
| ianw | zaro: hey | 23:36 |
| pabelanger | jroll: cool, thanks! | 23:36 |
| clarkb | pabelanger: that isnt causing failures right? | 23:36 |
| clarkb | just less efficient | 23:36 |
| *** AnarchyAo has quit IRC | 23:36 | |
| *** jcoufal has joined #openstack-infra | 23:36 | |
| pabelanger | jroll: http://grafana.openstack.org/dashboard/db/nodepool-rackspace?panelId=11&fullscreen shows the failure rates, you can filter on iad | 23:36 |
| jroll | cool, ty | 23:37 |
| pabelanger | clarkb: ya, less efficient | 23:37 |
| pabelanger | clarkb: it's only happen 1 today | 23:37 |
| pabelanger | and only chasing it because it was osic-cloud1 | 23:37 |
| *** jd__ has quit IRC | 23:37 | |
| jroll | pabelanger: looks like over 50% :/ | 23:38 |
| pabelanger | also still having DNS issues in bluebox, but going to wait until we reset nodepool for shade fixes | 23:38 |
| *** jd__ has joined #openstack-infra | 23:38 | |
| zaro | ianw: i checked gerrit functionality and it seems to work ok for me when i create a new site. could you please explain to me what you did to get the error? | 23:38 |
| pabelanger | jroll: Ya, rax-iad has been problematic since Aug. 3rd: http://grafana.openstack.org/dashboard/db/nodepool-rackspace?from=1469489937977&to=1472081937977 | 23:39 |
| pabelanger | only starting to debug it now | 23:39 |
| jroll | oh wow | 23:39 |
| jroll | pabelanger: so someone mentioned ssh host keys changing earlier, that's completely different, right? | 23:40 |
| pabelanger | jroll: it is | 23:40 |
| jroll | cool | 23:40 |
| jroll | pabelanger: well, I've got irc messages out, but you might get a quicker response via ticket | 23:40 |
| zaro | ianw: is puppet running on new gerrit site or on existing gerrit site? | 23:40 |
| zaro | puppet/puppet running `java -jar gerrit.war ...` | 23:41 |
| openstackgerrit | Merged openstack-infra/zuul: Store ssh_host_key of remote node https://review.openstack.org/360057 | 23:42 |
| *** nwkarsten has joined #openstack-infra | 23:42 | |
| ianw | zaro: this is launching a new node our launch scripts in system-config | 23:43 |
| zaro | actually i'm pretty sure it's running on existing site. ok. just wanted to confirm because that would mean my theory was correct. | 23:43 |
| *** Swami has quit IRC | 23:43 | |
| pabelanger | jroll: sure, can do that too | 23:43 |
| ianw | zaro: the host is up i think, although i've manually created the index but we can check logs | 23:43 |
| ianw | just a sec... | 23:43 |
| *** jcoufal has quit IRC | 23:44 | |
| ianw | zaro: try ssh root@104.130.159.161 | 23:46 |
| zaro | even against existing site i'm not really sure why you needed to run offline reindex at all. the gerrit version didn't change so it shouldn't have been necessary | 23:46 |
| *** nwkarsten has quit IRC | 23:47 | |
| zaro | ianw: you want me to see the logs? | 23:47 |
| ianw | zaro: we can poke at it, yeah | 23:47 |
| zaro | i'm not infra root | 23:48 |
| dmsimard | Ah, I'm sure it's been brought up but the issue I see with the ipv6 VMs is that if you're in a network that's not ipv6 enabled you can't follow the logs through telnet :( | 23:48 |
| *** dingyichen has joined #openstack-infra | 23:48 | |
| *** yolanda has quit IRC | 23:48 | |
| *** claudiub has quit IRC | 23:48 | |
| clarkb | dmsimard: yes | 23:49 |
| * dmsimard is going to lobby his ISP for v6 | 23:49 | |
| *** cody-somerville has quit IRC | 23:49 | |
| clarkb | dmsimard: zuulv3 will address that but un the meantime you can get an ipv6 (I have used a rax instance but dreamhost and vexxhost both ipv6) also HE tunnrls are a thing | 23:50 |
| jeblair | dmsimard: he tunnels are easy to get and set up here: https://tunnelbroker.net/ | 23:50 |
| pabelanger | clarkb: because of the websockets? | 23:50 |
| dmsimard | thanks for the suggestions :) | 23:51 |
| *** shashank_hegde has joined #openstack-infra | 23:51 | |
| jeblair | pabelanger: yes -- websockets will need a proxy | 23:52 |
| jeblair | mordred has a plan that involves that | 23:52 |
| pabelanger | jeblair: Ya, figured it was something along that lines | 23:52 |
| *** jerryz has quit IRC | 23:53 | |
| jeblair | something about in-browser javascript and connecting to random ports on random other hosts :) | 23:53 |
| clarkb | pabelanger: ya | 23:53 |
| *** zhurong has joined #openstack-infra | 23:54 | |
| asselin__ | jeblair, thanks for the heads up. We recently recommended folks to pin to zuul to 2.5.0. http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n31 Are we expecting another tag after that merges? | 23:56 |
| jeblair | asselin__: yes | 23:56 |
| *** roxanagh_ has quit IRC | 23:57 | |
| pabelanger | yay, 7 patches merging now | 23:58 |
| pabelanger | such wow | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
