| *** edmund has quit IRC | 00:01 | |
| *** edmund has joined #tripleo | 00:01 | |
| tchaypo | │error: unsupported configuration: SATA is not supported with this QEMU binary | 00:12 |
|---|---|---|
| tchaypo | so exciting. | 00:12 |
| *** nati_ueno has joined #tripleo | 00:12 | |
| *** nati_ueno has quit IRC | 00:17 | |
| mordred | tchaypo: that seems, you know, like an error | 00:21 |
| *** nati_uen_ has joined #tripleo | 00:22 | |
| *** matsuhashi has joined #tripleo | 00:26 | |
| *** nati_ue__ has joined #tripleo | 00:27 | |
| *** nati_uen_ has quit IRC | 00:30 | |
| *** nati_ue__ has quit IRC | 00:32 | |
| *** matsuhashi has quit IRC | 00:33 | |
| *** matsuhashi has joined #tripleo | 00:34 | |
| *** nati_uen_ has joined #tripleo | 00:35 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Add tchaypo to tripleo-cd-admins https://review.openstack.org/95614 | 00:35 |
| *** nati_uen_ has quit IRC | 00:39 | |
| openstackgerrit | Steve Baker proposed a change to openstack/diskimage-builder: Always bind-mount pypi mirror if dir exists https://review.openstack.org/95397 | 00:45 |
| *** nati_uen_ has joined #tripleo | 00:52 | |
| *** nati_ue__ has joined #tripleo | 00:54 | |
| *** nati_uen_ has quit IRC | 00:57 | |
| *** nati_ueno has joined #tripleo | 00:57 | |
| *** nati_ue__ has quit IRC | 00:59 | |
| *** nati_ueno has quit IRC | 00:59 | |
| *** nati_ueno has joined #tripleo | 01:00 | |
| StevenK | tchaypo: http://www.theregister.co.uk/2014/05/26/review_hp_zbook_17_mobile_workstation/ | 01:00 |
| lifeless | there's a special level of hell reserved for and now I have a etherpad to track etherpads | 01:01 |
| tchaypo | that reminds me, my SSD turned up last week, I should install that | 01:02 |
| *** nati_ueno has quit IRC | 01:04 | |
| tchaypo | ah, excellent. the installation manual starts of saying that "To reduce risk of data loss, initiate standby"; then has a caution that "To prevent system lockup and loss of information ... do not remove the hard drive while the computer is .. in Standby" | 01:05 |
| lifeless | the 15" 3.2K horizontal screen looks lovely | 01:07 |
| mordred | tchaypo: wow | 01:07 |
| mordred | tchaypo: that's truly special | 01:07 |
| mordred | lifeless: I believe you have invented evil | 01:07 |
| lifeless | mordred: oh? | 01:09 |
| *** nati_ueno has joined #tripleo | 01:09 | |
| mordred | lifeless: lifeless | there's a special level of hell reserved for and now I have a etherpad to track etherpads | 01:24 |
| lifeless | oh right :) | 01:24 |
| StevenK | Blah | 01:25 |
| StevenK | 2014-05-26 23:19:49.573 | Cloning into 'devstack-gate'... | 01:25 |
| StevenK | 2014-05-26 23:19:59.980 | fatal: Unable to look up git.openstack.org (port 9418) (Name or service not known) | 01:25 |
| *** matsuhashi has quit IRC | 01:33 | |
| *** nati_ueno has quit IRC | 01:33 | |
| *** matsuhashi has joined #tripleo | 01:34 | |
| *** yamahata has joined #tripleo | 01:37 | |
| *** edmund has quit IRC | 01:41 | |
| *** edmund1 has joined #tripleo | 01:41 | |
| *** nosnos has joined #tripleo | 01:44 | |
| *** edmund1 has quit IRC | 01:45 | |
| *** edmund has joined #tripleo | 01:45 | |
| *** nati_ueno has joined #tripleo | 02:00 | |
| SpamapS | o/ | 02:03 |
| StevenK | SpamapS: How is sunny, downtown Bangalore? | 02:06 |
| SpamapS | StevenK: muggy and slightly clouded | 02:10 |
| *** nati_ueno has quit IRC | 02:10 | |
| SpamapS | StevenK: 25C and thunderstorms | 02:10 |
| SpamapS | I did get to watch the sun rise over the city while in transit from the airport | 02:10 |
| SpamapS | it's a lot like LA.. spread out A LOT | 02:11 |
| lifeless | SpamapS: oh HAI | 02:11 |
| SpamapS | lifeless: greetings from UTC+0530 | 02:15 |
| SpamapS | lifeless: our cloud is screwed.. | 02:15 |
| SpamapS | lifeless: I wonder if we should just delete all the instances, backup the db, and re-deploy | 02:15 |
| SpamapS | lifeless: and then instead of os-cloud-config we just restore the db | 02:16 |
| lifeless | what sort of screwed - is the network thing resolved? | 02:17 |
| SpamapS | lifeless: AFAICT we had a bridging loop, but I couldn't get anybody in network engineering to diagnose that fo rme. | 02:18 |
| SpamapS | lifeless: once I ran os-collect-config --force --one on the controller it went away | 02:18 |
| lifeless | interesting. | 02:18 |
| lifeless | terrifyingly so | 02:19 |
| *** nati_ueno has joined #tripleo | 02:19 | |
| SpamapS | lifeless: yeah I was thinking maybe the IP only got half moved and thus was on the bridge and the physical port or something weird like that | 02:19 |
| SpamapS | lifeless: I susect it is the weird vlan25 crap | 02:19 |
| SpamapS | It will never make sense to me how that works. :-P | 02:19 |
| lifeless | hah | 02:21 |
| lifeless | vlans are a header strip on top of a raw device | 02:22 |
| StevenK | SpamapS: Canonical uses vlans extensively, you never had to deal with IS with vlan fun? | 02:24 |
| *** nati_ueno has quit IRC | 02:25 | |
| lifeless | btw | 02:27 |
| lifeless | https://etherpad.openstack.org/p/tripleo-etherpads | 02:27 |
| lifeless | meta | 02:27 |
| lifeless | ok time to figure out what I'm cooking for dinner | 02:28 |
| SpamapS | lifeless: anyway, without the switching loop, now GRE and openvswitch just aren't working. I don't know why. | 02:29 |
| SpamapS | looks like MTU issues to me. | 02:29 |
| SpamapS | I can SSH into instances, but then when the traffic starts flowing it just stalls out | 02:30 |
| SpamapS | gro is off | 02:30 |
| *** nati_ueno has joined #tripleo | 02:33 | |
| SpamapS | anyway time to go do heaty stuff | 02:36 |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1321943 | 03:00 |
| uvirtbot | Launchpad bug 1321943 in tripleo "Ceilometer Swift polling on overcloud control node fails with a 403 forbidden error" [Undecided,In progress] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1321563 | 03:00 |
| uvirtbot | Launchpad bug 1321563 in diskimage-builder "Add support in ramdisk element to fetch IP from dhcp" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1318767 | 03:00 |
| uvirtbot | Launchpad bug 1318767 in tripleo "apache element SSL cert check fails" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1314978 | 03:00 |
| uvirtbot | Launchpad bug 1314978 in tripleo "Cloud vm not pingable after overcloud upgrade " [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1315355 | 03:00 |
| uvirtbot | Launchpad bug 1315355 in tripleo "Upgrade of overcloud failed with "Connection to neutron failed: Maximum attempts reached"" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1323167 | 03:00 |
| uvirtbot | Launchpad bug 1323167 in tripleo "overcloud can't create an instance by neutron error" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1319473 | 03:00 |
| uvirtbot | Launchpad bug 1319473 in tripleo "devtest_testenv.sh doesn't honour overcloud_computescale or overcloud_controlscale" [Undecided,In progress] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1316675 | 03:00 |
| uvirtbot | Launchpad bug 1316675 in tripleo "Saving a devtest VM results in error" [Undecided,In progress] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1320090 | 03:00 |
| uvirtbot | Launchpad bug 1320090 in tripleo "pxe_ephemeral_format': u'ext4 left on nodes after deploys deleted" [Undecided,New] | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| *** nosnos has quit IRC | 03:25 | |
| *** eghobo has joined #tripleo | 03:30 | |
| *** akuznetsov has joined #tripleo | 03:48 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/os-cloud-config: Clamp pm_password to 255 for nova-bm nodes https://review.openstack.org/95441 | 04:06 |
| *** akuznetsov has quit IRC | 04:16 | |
| *** akuznetsov has joined #tripleo | 04:18 | |
| *** rakesh_hs has joined #tripleo | 04:25 | |
| *** nati_ueno has quit IRC | 04:31 | |
| *** nati_ueno has joined #tripleo | 04:32 | |
| *** TravT has joined #tripleo | 04:33 | |
| *** tzumainn has quit IRC | 04:39 | |
| *** nosnos has joined #tripleo | 04:46 | |
| *** e0ne has joined #tripleo | 04:48 | |
| *** e0ne has quit IRC | 04:49 | |
| *** e0ne has joined #tripleo | 04:49 | |
| tchaypo | SpamapS: why are are you in bangalore? | 04:52 |
| SpamapS | tchaypo: THE DOSA | 04:52 |
| tchaypo | I'm trying to dig up memories, but all I can remember is Hyderabad | 04:52 |
| *** e0ne has quit IRC | 04:53 | |
| SpamapS | tchaypo: also because HP has committed a team of 10 engineers that live and work here to help improve Heat. | 04:53 |
| tchaypo | I've somehow never heard of dosa before, but http://en.wikipedia.org/wiki/Dosa makes it look tasty | 04:53 |
| SpamapS | tchaypo: Dosa is southern indian food.. and most indian food you get in the UK and the US (and likely AU too) is northern. | 04:54 |
| tchaypo | I've been to bangalore and still not encountered dosa :p | 04:54 |
| *** edmund has quit IRC | 04:55 | |
| lifeless | SpamapS: <3 good indian food | 04:55 |
| tchaypo | Oh, hang on, the memories just came back | 04:55 |
| tchaypo | The first time I had a biryani was in Bangalore | 04:55 |
| tchaypo | ah, and now there are two of you here | 04:56 |
| tchaypo | I finally have the hpcs vpn working. I don't know what happens now. greghaynes is offering to do something with an ssh key, but as he said he's on holiday i'm ignoring him | 04:56 |
| tchaypo | actually he said that it is a holiday, not that he's on holiday. still ignoring him :) | 04:57 |
| greghaynes | haha | 04:57 |
| greghaynes | I did just jump through the hoops to get on hpcs though | 04:57 |
| greghaynes | so effort already expended :) | 04:57 |
| lifeless | tchaypo: probably he's offering to add it to the jump host(s) we've been using. | 04:57 |
| lifeless | but I need to make some green chicken curry right now | 04:57 |
| greghaynes | yep | 04:57 |
| greghaynes | to both those things | 04:58 |
| StevenK | You want green chicken curry at 10pm? | 04:58 |
| SpamapS | they're quite well along the way to fattening me up just 7 hours after I arrived.. I'm getting some sort of butter + flour + sugar sweet that is melting me. | 04:58 |
| greghaynes | StevenK: ofc | 04:58 |
| tchaypo | SpamapS: that stuff is fantastic. | 04:59 |
| SpamapS | once inside the HP office, it is indistignuishable from the Fort Collins Colorado and Galway offices btw. ;) | 04:59 |
| SpamapS | except the unaturally immense city ou tthe window. | 05:00 |
| StevenK | Not sure what is for dinner tonight. Maybe some honey and soy beef stir fry | 05:00 |
| StevenK | SpamapS: And Seattle, and Moffatt ... ? :-) | 05:01 |
| tchaypo | SpamapS: are there any slums within view? | 05:01 |
| SpamapS | no, Moffet is unbelievably sterile | 05:01 |
| SpamapS | and I haven't been to the new Seattle, but the old one was like a shoebox full of tiny cubes. | 05:01 |
| StevenK | Hm, speaking of | 05:02 |
| SpamapS | tchaypo: there are mostly slums and giant apartment buildings in view | 05:02 |
| StevenK | Let's see if I can find a suitable meeting room for Friday | 05:02 |
| clarkb | new one is moffet cubes with seattle amenities | 05:02 |
| SpamapS | clarkb: Seattle wins the location war. | 05:02 |
| SpamapS | hands down. | 05:02 |
| SpamapS | Can't beat 7th and Pike. | 05:02 |
| greghaynes | I discovered I can walk to amtrak in portland, train to seattle, walk from seattle amtrak to hp office | 05:03 |
| clarkb | also windows | 05:03 |
| tchaypo | SpamapS: If I'm remembering the various places correctly, the apartment I stayed in in Bangalore was part of a complex of giant apartment blocks that had slums right up to the walls of the complex, and western-style shopping malls directly across the freeway | 05:03 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Add support for source-repos gerrit refs https://review.openstack.org/90890 | 05:03 |
| greghaynes | tchaypo: Im all setup to add your key at this point, so its really no extra effort :) | 05:03 |
| greghaynes | NobodyCam: ^ relevant merge | 05:04 |
| StevenK | greghaynes: How long is the amtrak from Portland to Seattle? | 05:05 |
| greghaynes | 3ish hr | 05:05 |
| StevenK | Not as bad as I was thinking | 05:05 |
| greghaynes | Its not very reliable though, basically +/- 1hr | 05:05 |
| tchaypo | how frequent is the service? | 05:06 |
| greghaynes | theres about 6 a day | 05:08 |
| *** vkdrao has joined #tripleo | 05:12 | |
| * StevenK continues trying to work out how to book a conf room | 05:16 | |
| *** rakesh_hs has quit IRC | 05:24 | |
| *** rakesh_hs has joined #tripleo | 05:25 | |
| *** lazy_prince has joined #tripleo | 05:27 | |
| *** lazy_prince has quit IRC | 05:31 | |
| *** lazy_prince has joined #tripleo | 05:32 | |
| *** TravT has quit IRC | 05:33 | |
| *** eghobo has quit IRC | 05:44 | |
| openstackgerrit | Juerg Haefliger proposed a change to openstack/diskimage-builder: Fix package removal https://review.openstack.org/93275 | 05:51 |
| *** mrunge has joined #tripleo | 06:03 | |
| rpodolyaka | morning tripleo | 06:07 |
| marios | o/ | 06:09 |
| lazy_prince | Hi All.. Wanted to check if deploy element is depreciated from diskimage-builer. If so, how can I create a deployment image for use with Nova-baremetal..? | 06:11 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/os-cloud-config: Check for relevant environment variables https://review.openstack.org/95396 | 06:11 |
| lazy_prince | sorry.. found that deploy has been renamed to deploy-baremetal.. | 06:14 |
| lifeless | lazy_prince: yup | 06:23 |
| StevenK | lifeless: Re: your comment about ssh key to ironic, you mean in -incubator, or in os-cloud-config? | 06:24 |
| lifeless | StevenK: os-cloud-config would be cool | 06:25 |
| lifeless | StevenK: since we're about to move to that | 06:25 |
| lazy_prince | lifeless: right.. but the Readme file still points to deploy element. Can I offer a patch to fix it..? or is it already taken care of.. | 06:26 |
| lifeless | lazy_prince: please do | 06:26 |
| lazy_prince | Thanks.. will post a patch soon.. | 06:26 |
| StevenK | lifeless: The property in driver_info is ssh_key_filename, though ? | 06:27 |
| *** rdopiera has joined #tripleo | 06:28 | |
| *** dtantsur|afk is now known as dtantsur | 06:31 | |
| StevenK | Ah | 06:32 |
| StevenK | ssh_key_contents | 06:32 |
| *** rakesh_hs has quit IRC | 06:37 | |
| StevenK | lifeless: Hmm, passed as pm_password , or something else? | 06:38 |
| *** rakesh_hs has joined #tripleo | 06:39 | |
| *** mrunge has quit IRC | 06:43 | |
| openstackgerrit | Om Kumar proposed a change to openstack/diskimage-builder: Replacing deploy to deploy-baremetal in README.md https://review.openstack.org/95655 | 06:45 |
| lifeless | StevenK: thats right | 06:46 |
| lifeless | StevenK: pm_password ->ssh_key_contents | 06:46 |
| StevenK | Right, easy | 06:46 |
| lifeless | its kindof weird Ironic doesn't have a standard field for password | 06:46 |
| lifeless | but thats a different discussion | 06:47 |
| lifeless | when the Ironic devs start using Ironic, I imagine it will get fixed PDQ :) | 06:47 |
| lifeless | the lenovo yoga2 looks /nice/ | 06:47 |
| lifeless | 13.3" QHD+ (3200 x 1800) IPS wide-view display with 10-point multitouch technology | 06:47 |
| lifeless | I wish we had something equivalent | 06:48 |
| lifeless | thats with an i7 ... | 06:48 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/os-cloud-config: Switch to ssh_key_contents for ironic https://review.openstack.org/95657 | 06:48 |
| *** ifarkas has joined #tripleo | 06:50 | |
| *** jtomasek has joined #tripleo | 06:52 | |
| tchaypo | lifeless: didn't you see the review of the 17" z-book that someone pasted me this morning" | 06:57 |
| StevenK | I pasted it | 06:59 |
| *** boris-42 has quit IRC | 07:03 | |
| lifeless | tchaypo: I did, I quoted the 3.2K 15" reference from it. | 07:07 |
| lifeless | tchaypo: the lenovo above is 13.3" 3.2K | 07:07 |
| lifeless | whats the bug for | 07:08 |
| lifeless | 2014-05-27 06:52:48.725 | Cannot fetch index base URL http://pypi.openstack.org/openstack/? | 07:08 |
| lifeless | 1292141 | 07:09 |
| lifeless | jogo: would -really- love your help getting tripleo-ci test logs into elasticsearch | 07:10 |
| *** boris-42 has joined #tripleo | 07:11 | |
| *** jtomasek has quit IRC | 07:15 | |
| *** jtomasek has joined #tripleo | 07:15 | |
| *** jcoufal has joined #tripleo | 07:17 | |
| *** nati_ueno has quit IRC | 07:18 | |
| *** jprovazn has joined #tripleo | 07:18 | |
| *** lazy_prince has quit IRC | 07:19 | |
| *** nati_ueno has joined #tripleo | 07:19 | |
| *** hashar has joined #tripleo | 07:19 | |
| *** lazy_prince has joined #tripleo | 07:21 | |
| *** nati_ueno has quit IRC | 07:23 | |
| *** d0ugal_ has quit IRC | 07:25 | |
| *** jtomasek has quit IRC | 07:29 | |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-image-elements: Change stunnel priority and binding addresses https://review.openstack.org/95663 | 07:30 |
| jang | https://review.openstack.org/#/c/95177/ could do with a +A if possible. Just a missing package from an install-packages line. | 07:40 |
| *** akuznetsov has quit IRC | 07:43 | |
| tchaypo | I'm logged in as cetest | 07:50 |
| *** nati_ueno has joined #tripleo | 07:50 | |
| tchaypo | and i think that's enough for today. Tomorrow I can figure out where to go from there. | 07:50 |
| tchaypo | also, tomorrow I'm having my first driving lessing in a decade or so. | 07:50 |
| tchaypo | also, I've discovered a "compromise" position with my laptop today, useful for cramped environments where i don't have space for the keyboard and stand | 07:52 |
| tchaypo | I can stick the keyboard on top of the laptop. Still means my neck is bent downard awkwardly but it's tons better than using the built-in laptop flatboard | 07:53 |
| *** nati_ueno has quit IRC | 07:54 | |
| lifeless | jang: it has no functional test results, so it can't get a +A. | 07:54 |
| lifeless | jang: I've poked zuul to get some. | 07:54 |
| jang | what? | 07:55 |
| jang | oh, righto | 07:55 |
| * jang swaps it back in | 07:55 | |
| jang | cheers. | 07:55 |
| openstackgerrit | Gerry Drudy proposed a change to openstack/tripleo-image-elements: Enable rsync daemon on swift-storage https://review.openstack.org/95173 | 07:58 |
| *** viktors|afk has quit IRC | 08:09 | |
| *** akuznetsov has joined #tripleo | 08:09 | |
| *** pblaho has joined #tripleo | 08:13 | |
| *** pblaho has quit IRC | 08:13 | |
| *** jistr has joined #tripleo | 08:14 | |
| *** pblaho has joined #tripleo | 08:14 | |
| lifeless | SpamapS: ok so I've finally reviewed all specs again etc; cloud still naffed I guess? | 08:14 |
| *** viktors has joined #tripleo | 08:15 | |
| *** derekh_ has joined #tripleo | 08:17 | |
| Ng | morning | 08:19 |
| Ng | SpamapS: were you LHR'ing on your way to India? | 08:19 |
| lifeless | Ng: SpamapS is in India now | 08:23 |
| *** lucasagomes has joined #tripleo | 08:29 | |
| Ng | :) | 08:30 |
| lifeless | Ng: SpamapS says our ci cloud is entirely down | 08:31 |
| lifeless | Ng: see IRC backscroll | 08:31 |
| lifeless | Ng: I have just surfaced enough to be able to in principle look | 08:31 |
| lifeless | but 2030... tired | 08:31 |
| Ng | lifeless: I'll have a look | 08:32 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Properly enabling and restarting snmpd https://review.openstack.org/95689 | 08:47 |
| *** nosnos has quit IRC | 08:48 | |
| *** nosnos has joined #tripleo | 08:48 | |
| *** martyntaylor has joined #tripleo | 08:50 | |
| *** nati_ueno has joined #tripleo | 08:50 | |
| *** mrunge has joined #tripleo | 08:52 | |
| *** nosnos has quit IRC | 08:53 | |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/tuskar: Synced jsonutils from oslo-incubator https://review.openstack.org/95700 | 08:53 |
| *** nati_ueno has quit IRC | 08:55 | |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Install psmisc on TE hosts https://review.openstack.org/91535 | 08:56 |
| *** lazy_prince2 has joined #tripleo | 08:59 | |
| *** andreaf has joined #tripleo | 08:59 | |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1321943 | 09:00 |
| uvirtbot | Launchpad bug 1321943 in tripleo "Ceilometer Swift polling on overcloud control node fails with a 403 forbidden error" [Undecided,In progress] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1321563 | 09:00 |
| uvirtbot | Launchpad bug 1321563 in diskimage-builder "Add support in ramdisk element to fetch IP from dhcp" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1318767 | 09:00 |
| uvirtbot | Launchpad bug 1318767 in tripleo "apache element SSL cert check fails" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1314978 | 09:00 |
| uvirtbot | Launchpad bug 1314978 in tripleo "Cloud vm not pingable after overcloud upgrade " [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1315355 | 09:00 |
| uvirtbot | Launchpad bug 1315355 in tripleo "Upgrade of overcloud failed with "Connection to neutron failed: Maximum attempts reached"" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1323167 | 09:00 |
| uvirtbot | Launchpad bug 1323167 in tripleo "overcloud can't create an instance by neutron error" [Undecided,New] | 09:00 |
| *** lazy_prince has quit IRC | 09:00 | |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1319473 | 09:00 |
| uvirtbot | Launchpad bug 1319473 in tripleo "devtest_testenv.sh doesn't honour overcloud_computescale or overcloud_controlscale" [Undecided,In progress] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1316675 | 09:00 |
| uvirtbot | Launchpad bug 1316675 in tripleo "Saving a devtest VM results in error" [Undecided,In progress] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1320090 | 09:00 |
| uvirtbot | Launchpad bug 1320090 in tripleo "pxe_ephemeral_format': u'ext4 left on nodes after deploys deleted" [Undecided,New] | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| *** jp_at_hp has joined #tripleo | 09:08 | |
| *** lazy_prince2 is now known as lazy_prince | 09:11 | |
| *** jtomasek has joined #tripleo | 09:14 | |
| *** nosnos has joined #tripleo | 09:35 | |
| *** lazy_prince2 has joined #tripleo | 09:38 | |
| *** lazy_prince has quit IRC | 09:40 | |
| *** lokesh184 has joined #tripleo | 09:47 | |
| *** nati_ueno has joined #tripleo | 09:51 | |
| openstackgerrit | Gerry Drudy proposed a change to openstack/tripleo-image-elements: Move swift logs from /var/log/syslog into their own log files https://review.openstack.org/91533 | 09:53 |
| *** akrivoka has joined #tripleo | 09:53 | |
| *** nati_ueno has quit IRC | 09:55 | |
| *** matsuhashi has quit IRC | 09:58 | |
| *** matsuhashi has joined #tripleo | 10:00 | |
| *** hewbrocca has joined #tripleo | 10:06 | |
| *** matsuhashi has quit IRC | 10:07 | |
| *** nosnos has quit IRC | 10:07 | |
| *** matsuhashi has joined #tripleo | 10:07 | |
| *** matsuhashi has quit IRC | 10:12 | |
| *** tzumainn has joined #tripleo | 10:15 | |
| *** markmc has joined #tripleo | 10:16 | |
| * TheJulia yawns | 10:16 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Install lsof during Percona install https://review.openstack.org/93031 | 10:17 |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/tuskar: Synced jsonutils from oslo-incubator https://review.openstack.org/95700 | 10:22 |
| *** akrivoka has quit IRC | 10:23 | |
| *** lazy_prince2 has quit IRC | 10:24 | |
| openstackgerrit | Coleman Corrigan proposed a change to openstack/os-apply-config: Fix for string key traversal bug #1321827 https://review.openstack.org/94810 | 10:25 |
| uvirtbot | Launchpad bug 1321827 in os-apply-config "os-apply-config fails with TypeError: string indices must be integers when traversing strings " [Medium,Triaged] https://launchpad.net/bugs/1321827 | 10:25 |
| *** lokesh184 has quit IRC | 10:27 | |
| *** lazy_prince has joined #tripleo | 10:28 | |
| *** vkdrao has quit IRC | 10:29 | |
| *** lokesh184 has joined #tripleo | 10:43 | |
| *** nati_ueno has joined #tripleo | 10:52 | |
| *** akrivoka has joined #tripleo | 10:55 | |
| *** nati_ueno has quit IRC | 10:56 | |
| *** e0ne has joined #tripleo | 10:59 | |
| *** e0ne has quit IRC | 11:04 | |
| jprovazn | derekh_: Hi, I wonder why this change https://review.openstack.org/#/c/94354/1/scripts/devtest_overcloud.sh doesn't break tests (which passed), because according to this line special IP setting is required for overcloud: https://github.com/openstack/tripleo-image-elements/blob/master/elements/tripleo-cd/deploy-ci-overcloud#L46 | 11:15 |
| jprovazn | do you have an idea why tests passed then? | 11:15 |
| *** lokesh184 has quit IRC | 11:19 | |
| derekh_ | jprovazn: that file is used to deploy a new overcloud on which to run CI, it isn't actually used as part of a CI test | 11:23 |
| derekh_ | jprovazn: but the change your referencing may prevent us from deploying a new ci overcloud using devtest_overcloud in future when we need to | 11:24 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Choose whether to deploy/update using heat. https://review.openstack.org/92344 | 11:25 |
| *** 20WAAGQA1 is now known as shadower | 11:28 | |
| openstackgerrit | Jon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-incubator: Revert to correctly determining nodes needed https://review.openstack.org/93597 | 11:30 |
| openstackgerrit | Jon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-incubator: Revert to correctly determining nodes needed https://review.openstack.org/93597 | 11:31 |
| jprovazn | derekh_: ah, thanks | 11:32 |
| jp_at_hp | slagle: GheRivero derekh_ - can I get a 2nd opinion on https://review.openstack.org/93597 - I disagree with the -2, and I haven't yet been given any good reason why that change shouldn't go through. As far as I can tell I have jumped through all of the hoops to satisfy all use-cases, rather than leaving the code broken for a particular use-case... | 11:34 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-specs: Tripleo CI improvements https://review.openstack.org/95026 | 11:34 |
| StevenK | jp_at_hp: You will also need to convince greghaynes to remove his -2 | 11:36 |
| *** yamahata has quit IRC | 11:36 | |
| jp_at_hp | StevenK: yes - but a 2nd opinion that says the change is good makes for an easier argument. If I don't get a 2nd opinion that thinks the change should go through I abandon it, simples. I'm not adverse to abandoning it, but it is a bug and one I think needs fixing, and I just don't understand why there is an aversino to fixing it. | 11:40 |
| dshulyak | derekh_: how that patch can affect deploying new ci overcloud? | 11:41 |
| StevenK | jp_at_hp: Sounds fair, carry on. | 11:42 |
| derekh_ | dshulyak: we're using NeutronPublicInterfaceIP to set the IP of the ci overcloud along with using it in the endpoint, the patch looks like its removing support for putting it in the endpoint , | 11:45 |
| derekh_ | dshulyak: although I'm not sure (which is why I said *may*) | 11:45 |
| *** rlandy has joined #tripleo | 11:49 | |
| *** e0ne has joined #tripleo | 11:49 | |
| *** lucasagomes is now known as lucas-hungry | 11:54 | |
| *** e0ne has quit IRC | 11:54 | |
| *** nati_uen_ has joined #tripleo | 11:55 | |
| *** weshay has joined #tripleo | 11:58 | |
| *** nati_uen_ has quit IRC | 11:59 | |
| *** IvanBerezovskiy has joined #tripleo | 12:09 | |
| *** e0ne has joined #tripleo | 12:16 | |
| *** rakesh_hs has quit IRC | 12:17 | |
| *** dprince has joined #tripleo | 12:18 | |
| *** rakesh_hs has joined #tripleo | 12:19 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Properly enabling and restarting snmpd https://review.openstack.org/95689 | 12:20 |
| *** dprince has quit IRC | 12:25 | |
| *** dprince has joined #tripleo | 12:26 | |
| *** rcarrillocruz has left #tripleo | 12:30 | |
| *** rcarrillocruz has joined #tripleo | 12:30 | |
| NobodyCam | greghaynes: yea, lifeless point that one out to me yesterday :) | 12:30 |
| *** rbrady has joined #tripleo | 12:32 | |
| *** jdob has joined #tripleo | 12:33 | |
| *** martyntaylor has left #tripleo | 12:39 | |
| *** yamahata has joined #tripleo | 12:42 | |
| *** yamahata has quit IRC | 12:45 | |
| *** yamahata has joined #tripleo | 12:46 | |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-specs: Haproxy configuration options https://review.openstack.org/94907 | 12:49 |
| *** nati_ueno has joined #tripleo | 12:55 | |
| *** jistr has quit IRC | 12:57 | |
| *** noslzzp has joined #tripleo | 12:58 | |
| *** nati_ueno has quit IRC | 13:00 | |
| *** pblaho has quit IRC | 13:00 | |
| *** pblaho has joined #tripleo | 13:01 | |
| *** sseago__ has joined #tripleo | 13:01 | |
| *** sseago_ has quit IRC | 13:01 | |
| *** jistr has joined #tripleo | 13:03 | |
| *** julim has joined #tripleo | 13:05 | |
| greghaynes | NobodyCam: Yep, just letting you know its merged now :) | 13:06 |
| NobodyCam | :) w00t Ty greghaynes | 13:06 |
| *** lucas-hungry is now known as lucasagomes | 13:07 | |
| *** jcoufal has quit IRC | 13:07 | |
| *** jcoufal_ has joined #tripleo | 13:07 | |
| *** pblaho has quit IRC | 13:08 | |
| *** pblaho has joined #tripleo | 13:10 | |
| *** pblaho has quit IRC | 13:17 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Fix to Ironic gate job's baremetal config change https://review.openstack.org/94803 | 13:21 |
| openstackgerrit | therese-mchale proposed a change to openstack/tripleo-image-elements: Use Percona Packages for Ubuntu https://review.openstack.org/90134 | 13:29 |
| *** dtantsur has quit IRC | 13:32 | |
| *** sballe_ has joined #tripleo | 13:37 | |
| *** matty_dubs|gone is now known as matty_dubs | 13:39 | |
| *** sballe has quit IRC | 13:39 | |
| *** dtantsur has joined #tripleo | 13:41 | |
| *** sballe_ has quit IRC | 13:42 | |
| openstackgerrit | therese-mchale proposed a change to openstack/tripleo-image-elements: Add extra pinning for Percona Package install to work from a Mirror which includes both MySQL and Percona packages. Use the following environment variable: DIB_PERCONA_VERSION to pin to the specific version of these Percona libraries: libmysqlclient https://review.openstack.org/94346 | 13:46 |
| *** jcoufal has joined #tripleo | 13:49 | |
| *** jcoufal_ has quit IRC | 13:52 | |
| *** jistr has quit IRC | 13:53 | |
| *** sballe has joined #tripleo | 13:54 | |
| *** nati_ueno has joined #tripleo | 13:56 | |
| *** pblaho has joined #tripleo | 13:57 | |
| *** sballe has quit IRC | 13:59 | |
| *** nati_ueno has quit IRC | 14:00 | |
| *** beekneemech is now known as bnemec | 14:01 | |
| *** jistr has joined #tripleo | 14:04 | |
| *** hashar has quit IRC | 14:06 | |
| *** pblaho has quit IRC | 14:12 | |
| *** rdopiera has quit IRC | 14:13 | |
| *** Kiall_ is now known as Kiall | 14:14 | |
| *** ddieterly has joined #tripleo | 14:20 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Properly enabling and restarting snmpd https://review.openstack.org/95689 | 14:31 |
| *** sballe_ has joined #tripleo | 14:31 | |
| *** TravT has joined #tripleo | 14:39 | |
| *** sballe_ has quit IRC | 14:45 | |
| *** rakesh_hs has quit IRC | 14:51 | |
| openstackgerrit | therese-mchale proposed a change to openstack/tripleo-image-elements: Add extra pinning for Percona Package install to work from a Mirror https://review.openstack.org/94346 | 14:54 |
| *** nati_ueno has joined #tripleo | 14:57 | |
| *** al has quit IRC | 14:59 | |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1321943 | 15:00 |
| uvirtbot | Launchpad bug 1321943 in tripleo "Ceilometer Swift polling on overcloud control node fails with a 403 forbidden error" [Undecided,In progress] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1321563 | 15:00 |
| uvirtbot | Launchpad bug 1321563 in diskimage-builder "Add support in ramdisk element to fetch IP from dhcp" [Undecided,New] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1318767 | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1314978 | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1315355 | 15:00 |
| uvirtbot | Launchpad bug 1318767 in tripleo "apache element SSL cert check fails" [Undecided,New] | 15:00 |
| uvirtbot | Launchpad bug 1314978 in tripleo "Cloud vm not pingable after overcloud upgrade " [Undecided,New] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1323167 | 15:00 |
| uvirtbot | Launchpad bug 1315355 in tripleo "Upgrade of overcloud failed with "Connection to neutron failed: Maximum attempts reached"" [Undecided,New] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1319473 | 15:00 |
| uvirtbot | Launchpad bug 1323167 in tripleo "overcloud can't create an instance by neutron error" [Undecided,New] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1316675 | 15:00 |
| uvirtbot | Launchpad bug 1319473 in tripleo "devtest_testenv.sh doesn't honour overcloud_computescale or overcloud_controlscale" [Undecided,In progress] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1320090 | 15:00 |
| uvirtbot | Launchpad bug 1316675 in tripleo "Saving a devtest VM results in error" [Undecided,In progress] | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| uvirtbot | Launchpad bug 1320090 in tripleo "pxe_ephemeral_format': u'ext4 left on nodes after deploys deleted" [Undecided,New] | 15:00 |
| *** nati_ueno has quit IRC | 15:01 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack/tripleo-image-elements: Make the image name configurable https://review.openstack.org/95791 | 15:03 |
| *** morganfainberg_Z is now known as morganfainberg | 15:04 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack/tripleo-image-elements: Make the image name configurable https://review.openstack.org/95791 | 15:05 |
| *** blamar has quit IRC | 15:06 | |
| *** jprovazn has quit IRC | 15:07 | |
| lxsli | greghaynes: https://review.openstack.org/#/c/93041/ please? | 15:09 |
| lxsli | If you're happy with that, https://review.openstack.org/#/c/94856/ would be great too | 15:09 |
| *** blamar has joined #tripleo | 15:10 | |
| *** BadCub has joined #tripleo | 15:13 | |
| *** al has joined #tripleo | 15:14 | |
| *** viktors is now known as viktors|afc | 15:18 | |
| *** ccorrigan has quit IRC | 15:19 | |
| *** akuznetsov has quit IRC | 15:20 | |
| greghaynes | lxsli: ah - the removal of line 267 is what I was confused by, not the change to +. Was that line superfluous since we read in from existing ENV_JSON? | 15:21 |
| lxsli | greghaynes: It got moved up to line 245 | 15:22 |
| greghaynes | and also changed to an explicit 100 rather than jq magic that looks like an attempt to preserve an existing val | 15:23 |
| lxsli | Well, there was always an explicit 100 | 15:24 |
| openstackgerrit | Gerry Drudy proposed a change to openstack/tripleo-image-elements: Move swift logs from /var/log/syslog into their own log files https://review.openstack.org/91533 | 15:24 |
| *** morazi_ is now known as morazi | 15:24 | |
| lxsli | but now it will be overridden by any existing value in ENV_JSON | 15:24 |
| lxsli | In the old code, you got 100 whether you liked it or not | 15:25 |
| *** dtantsur is now known as dtantsur|afk | 15:25 | |
| lxsli | (unintentionally) | 15:25 |
| *** akuznetsov has joined #tripleo | 15:25 | |
| *** ddieterly has quit IRC | 15:25 | |
| *** mkerrin1 has quit IRC | 15:26 | |
| greghaynes | hehe, ok. Give me a sec to fiddle with because obviously I misunderstood some of this jq magic | 15:26 |
| *** lazy_prince has quit IRC | 15:26 | |
| lxsli | Yeah there's a lot of that going on (me incl.) | 15:26 |
| *** edmund has joined #tripleo | 15:30 | |
| *** mkerrin has joined #tripleo | 15:30 | |
| *** dshulyak_ has joined #tripleo | 15:37 | |
| derekh_ | How have we reviews with minus scores on them form weeks back https://review.openstack.org/#/q/status:open+(project:openstack/tripleo-image-elements+OR+project:openstack/diskimage-builder+OR+project:openstack/tripleo-heat-templates+OR+project:openstack/os-collect-config+OR+project:openstack/os-apply-config+OR+project:openstack/os-refresh-config+OR+project:openstack-infra/tripleo-ci+OR+project:openstack/tripleo-incubator+OR+project:openstack/os-cloud-confi | 15:39 |
| derekh_ | g+OR+project:openstack/tripleo-specs),n,002d073d0001486c | 15:39 |
| derekh_ | should they not be getting auto abandoned ? | 15:39 |
| ccrouch | derekh_: was this last update before the gerrit upgrade, perhaps that caused a wrinkle | 15:42 |
| derekh_ | ccrouch: yum I think it was, possibly the reason | 15:43 |
| *** eghobo has joined #tripleo | 15:43 | |
| *** hashar has joined #tripleo | 15:56 | |
| *** shakamunyi has joined #tripleo | 15:56 | |
| *** ddieterly has joined #tripleo | 15:59 | |
| *** IvanBerezovskiy has left #tripleo | 15:59 | |
| mordred | derekh_: autoabandon is no longer | 16:00 |
| mordred | derekh_: with new gerrit, core teams can WIP someone else's patch, which we couldnt' do before | 16:00 |
| mordred | so making something go away has a mechanism available to people should they want it | 16:00 |
| derekh_ | mordred: ahh ok, thanks, I kinda liked autoabandon, its kept the review clean of things people weren't working on, | 16:02 |
| derekh_ | so I guess we should all add a todo item , | 16:02 |
| derekh_ | check you patches and abandon if their no longer relevant | 16:02 |
| derekh_ | *your | 16:02 |
| *** matty_dubs is now known as matty_dubs|lunch | 16:03 | |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Name 99-setup-first-boot uniquely https://review.openstack.org/95808 | 16:03 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Delete redhat-common 00-usr-local-bin-secure-path https://review.openstack.org/95809 | 16:03 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Name 01-install-bin uniquely https://review.openstack.org/95810 | 16:03 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Name 10-rhel-cloud-image uniquely https://review.openstack.org/95811 | 16:03 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Name deploy-ironic and deploy-baremetal files uniq https://review.openstack.org/95812 | 16:03 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Add top level 'tests' dir for element testing https://review.openstack.org/95813 | 16:03 |
| *** cwolferh_ has joined #tripleo | 16:07 | |
| *** nati_ueno has joined #tripleo | 16:14 | |
| openstackgerrit | Loganathan Parthipan proposed a change to openstack/tripleo-incubator: Make disk bus type configurable https://review.openstack.org/92400 | 16:15 |
| *** markmc has quit IRC | 16:16 | |
| ccrouch | derekh_: i guess the core team also need a tidy up task: to set to WIP patches other people have forgotten about | 16:20 |
| derekh_ | ccrouch: ya, I think people are doing that already in some places, but I'd like them abandoned :-) | 16:22 |
| *** e0ne has quit IRC | 16:22 | |
| derekh_ | ccrouch: having a person doing it seems a little aggressive, but a autobot is just process .. | 16:22 |
| *** e0ne has joined #tripleo | 16:23 | |
| *** derekh_ has quit IRC | 16:26 | |
| *** e0ne has quit IRC | 16:27 | |
| *** jistr has quit IRC | 16:30 | |
| *** jcoufal has quit IRC | 16:32 | |
| *** BadCub has left #tripleo | 16:38 | |
| *** jtomasek has quit IRC | 16:39 | |
| *** martyntaylor has joined #tripleo | 16:45 | |
| *** lucasagomes is now known as lucas-dinner | 16:45 | |
| *** lucas-dinner has quit IRC | 16:48 | |
| *** olaph has joined #tripleo | 16:48 | |
| *** w_ has joined #tripleo | 16:48 | |
| *** w_ has quit IRC | 16:49 | |
| mordred | hrm | 16:49 |
| mordred | I wonder if we should look in to having the auto-abaondon (or even the abaondon bot modified to auto-wip) be an opt-in thing | 16:50 |
| mordred | like, similar to channel ops and whatnot - a team could add an entry to a yaml file somewhere saying "hey, we'd like autoabanon on our repos" | 16:50 |
| *** nati_ueno has quit IRC | 16:53 | |
| *** BadCub has joined #tripleo | 16:54 | |
| *** matty_dubs|lunch is now known as matty_dubs | 16:54 | |
| *** akrivoka has quit IRC | 16:54 | |
| *** yamahata has quit IRC | 16:55 | |
| *** nati_ueno has joined #tripleo | 16:56 | |
| ccrouch | mordred: that sounds like something worth exploring. I dont recall anyone complaining about how the auto-abandon used to work, so auto-wip might be the best alternative | 16:59 |
| ccrouch | i'll see if folks have strong/any feelings in the meeting today | 17:00 |
| *** BadCub has quit IRC | 17:00 | |
| *** hashar is now known as hasharOut | 17:00 | |
| mordred | ccrouch: there were actually a bunch of people who hated autoabandon - so there's potential feelings on both sides | 17:02 |
| slagle | ccrouch: fyi, no meeting today, it's the first week of the alt meeting schedule | 17:05 |
| morganfainberg | mordred, ++ on making auto abandon/auto-wip/auto-cleanup being a opt-in per project (perhaps even project can specify the window) | 17:17 |
| * morganfainberg realizes that it was random to pop into this channel. | 17:18 | |
| ccrouch | slagle: doh! | 17:19 |
| ccrouch | thanks, i'll need to update my reminders | 17:19 |
| ccrouch | mordred: ok, good to know | 17:19 |
| *** andreaf has quit IRC | 17:20 | |
| *** Lingo has joined #tripleo | 17:22 | |
| lifeless | morning :) | 17:26 |
| lifeless | I hate autoabandon when it happens to my branches | 17:26 |
| lifeless | I love it when it happens to those of drive-by contributors | 17:27 |
| lifeless | who have driven by and not come back | 17:27 |
| vinsh | dump the baby at the firestation devs. | 17:27 |
| vinsh | ;) | 17:27 |
| lifeless | Thats a terrible analogy, but yes. | 17:27 |
| * gothicmindfood just spit her coffee | 17:27 | |
| gothicmindfood | thanks vinsh ;) | 17:28 |
| vinsh | :D | 17:28 |
| * TheJulia suddenly realizes she needs coffee | 17:28 | |
| *** hasharOut has quit IRC | 17:28 | |
| *** nati_ueno has quit IRC | 17:31 | |
| bnemec | I'm generally +1 on auto-abandon. Maybe the time to abandon could be raised a bit to reduce the annoyance? | 17:31 |
| *** jprovazn has joined #tripleo | 17:37 | |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Use devtest_testenv default when building CI TE https://review.openstack.org/92489 | 17:39 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Reduce resource requirements set aside for testenvs https://review.openstack.org/92490 | 17:39 |
| *** eghobo has quit IRC | 17:41 | |
| *** eghobo has joined #tripleo | 17:42 | |
| *** nati_ueno has joined #tripleo | 17:42 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix a pep8 error in tempest/tests2skip.py https://review.openstack.org/94708 | 17:43 |
| *** pblaho has joined #tripleo | 17:45 | |
| *** dtantsur|afk is now known as dtantsur | 17:49 | |
| *** pblaho has quit IRC | 17:54 | |
| *** e0ne has joined #tripleo | 17:55 | |
| *** mrunge has quit IRC | 18:07 | |
| *** hashar has joined #tripleo | 18:09 | |
| *** nati_ueno has quit IRC | 18:13 | |
| *** nati_ueno has joined #tripleo | 18:14 | |
| *** jp_at_hp has quit IRC | 18:16 | |
| *** hashar is now known as hasharOut | 18:18 | |
| *** dtantsur is now known as dtantsur|afk | 18:19 | |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Factor out error behavior in dib-lint https://review.openstack.org/95850 | 18:19 |
| *** rlandy_ has joined #tripleo | 18:36 | |
| *** chuckC has quit IRC | 18:38 | |
| *** rlandy has quit IRC | 18:38 | |
| *** rpodolyaka1 has joined #tripleo | 18:45 | |
| *** chuckC has joined #tripleo | 18:46 | |
| *** akuznetsov has quit IRC | 18:46 | |
| *** cwolferh_ has quit IRC | 18:49 | |
| openstackgerrit | Phil Neal proposed a change to openstack/tripleo-incubator: Add Swift roles required to poll for Swift usage https://review.openstack.org/94730 | 18:50 |
| openstackgerrit | Phil Neal proposed a change to openstack/tripleo-incubator: Add Swift roles required to poll for Swift usage https://review.openstack.org/94730 | 18:52 |
| *** jesusaurus has quit IRC | 18:53 | |
| *** hasharOut is now known as hashar | 18:53 | |
| greghaynes | tchaypo: Thought you might find this link interesting http://coreos.com/docs/cluster-management/setup/etcd-cluster-discovery/ | 18:53 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Set -o pipefail new scripts https://review.openstack.org/95203 | 18:54 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Check for set -o pipefail https://review.openstack.org/83929 | 18:55 |
| *** tzumainn has quit IRC | 18:56 | |
| *** tzumainn has joined #tripleo | 18:57 | |
| lifeless | Ng: how did region1 recovery go ? | 18:57 |
| *** nati_ueno has quit IRC | 18:58 | |
| *** nati_ueno has joined #tripleo | 18:59 | |
| Ng | lifeless: terribly, I poked around, got very lost and confused and then started new starter stuff :/ | 18:59 |
| lifeless | ok | 18:59 |
| lifeless | I will see what I can see | 18:59 |
| lifeless | after ze meeting | 18:59 |
| Ng | I'm so completely out of touch with the CI stuff, I wonder if we could get a "here is what it's made of and where it lives" wiki page? | 18:59 |
| Ng | arg, ze meeting | 18:59 |
| Ng | alvays viz ze meetings | 18:59 |
| lifeless | Ng: how about some docs? | 19:00 |
| lifeless | you could write up your understanding and let folk correct it | 19:00 |
| Ng | wait, is it still happening now today? I thought tchaypo decided he hates me and moved to half past the middle of the night? | 19:00 |
| lifeless | oh yeah | 19:00 |
| lifeless | this is the alt week | 19:00 |
| lifeless | \o/ | 19:00 |
| lifeless | I can get shit down | 19:00 |
| ccrouch | iiuc meeting is in 12hrs from now | 19:01 |
| *** derekh_ has joined #tripleo | 19:02 | |
| lifeless | yes | 19:03 |
| lifeless | I failed at memory | 19:03 |
| rpodolyaka1 | oops | 19:03 |
| lifeless | derekh_: hi | 19:03 |
| derekh_ | lifeless: hello | 19:03 |
| lifeless | derekh_: I'm about to dive into the region1 issue | 19:03 |
| lifeless | derekh_: got a quick status summary for me? | 19:03 |
| SpamapS | lifeless: \o/ | 19:03 |
| *** jang has quit IRC | 19:04 | |
| SpamapS | lifeless: do you have a general plan? | 19:04 |
| lifeless | SpamapS: oh hai | 19:04 |
| lifeless | SpamapS: figure out whats up, then attempt to fix. | 19:04 |
| *** michchap_ has quit IRC | 19:04 | |
| lifeless | OODA basically | 19:04 |
| *** Penick has joined #tripleo | 19:04 | |
| *** Lingo is now known as BadCub | 19:05 | |
| derekh_ | last I looked was monday, eventually got the controller back after a reboot, SpamapS was looking at it since then but basically any time we get it back running again, jenkins keeps loosing connections to VM's and jobs fail | 19:05 |
| derekh_ | lifeless: ^ | 19:05 |
| *** CLOUDOUTAGE has joined #tripleo | 19:05 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- R1 is down for an unknown amount of time due to networking issues - https://etherpad.openstack.org/p/cloud-outage | 19:05 |
| *** CLOUDOUTAGE has quit IRC | 19:05 | |
| lifeless | derekh_: thanks | 19:06 |
| derekh_ | have we a meeting or did I miss a memo ? | 19:06 |
| lifeless | in 12 hours | 19:06 |
| SpamapS | yeah that jives with what I saw | 19:06 |
| jdob | there's an email about it, this week starts the alternating | 19:06 |
| lifeless | SpamapS: did you get monitoring up? | 19:07 |
| SpamapS | telnetting to port 22 on an instance works, but ssh'ing in fails.. traffic stops flowing after a bit | 19:07 |
| SpamapS | lifeless: I have a nagios running on the bastion | 19:07 |
| SpamapS | lifeless: but I wasn't able to get it tied into openid so everyone can access | 19:07 |
| lifeless | ah | 19:07 |
| lifeless | openid ? | 19:07 |
| lifeless | we have the cloud admin credentials we autogenerate already, could use that | 19:08 |
| derekh_ | jdob: ok, thanks, I guess I'll try and set an early alarm but I may be getting kids up etc... | 19:09 |
| SpamapS | lifeless: there's no passwords in there are there? | 19:09 |
| SpamapS | lifeless: anyway, yeah lots of stuff I could do.. just got distracted | 19:09 |
| jdob | ya, if my daughter keeps up her trend of waking us up in the middle of the night, I may be there | 19:09 |
| jdob | and as punishment for coming in and waking us for something dumb, i'm gonna make her attend too | 19:09 |
| lifeless | SpamapS: there are on the host that runs tripleo-cd in / | 19:10 |
| jdob | so if you see <ldob> o/ , you know who that is. | 19:10 |
| lifeless | jdob: LOL :) | 19:10 |
| lifeless | jdob: how old? | 19:10 |
| jdob | going to be 7 in a month, assuming she makes it that long | 19:10 |
| jdob | the boy is 2.5, and he's not on a great track to reach 3 either | 19:10 |
| jdob | been an interesting time in the Dobies household lately :D | 19:11 |
| SpamapS | lifeless: yeah thats hard to get over to the bastion. But openid would let us stay out of credential management. Anyway, it was just an idea. We can also just have a nagiosadmin password. | 19:11 |
| SpamapS | jdob: are you suggesting that murder is an appropriate answer to insomnia? Not saying I haven't contemplated it... | 19:12 |
| derekh_ | lifeless: also R2 config changed last week to run 18 nodes (it was 10), I've never once seen it run more then 10 jobs at a time so something is up, will talk to infra in a few minutes when im back | 19:12 |
| *** jang1 has joined #tripleo | 19:12 | |
| *** rpodolyaka1 has quit IRC | 19:12 | |
| jdob | SpamapS: honestly, it beats buying stuff off infomercials | 19:13 |
| jdob | and I'm willing to bet that would hold up in a court | 19:13 |
| *** jang has joined #tripleo | 19:13 | |
| lifeless | derekh_: did we up quotas on it ? | 19:16 |
| *** jang has quit IRC | 19:20 | |
| derekh_ | lifeless: will double check now but I don't think we needed to, the RAM qouta was the one constraining infra which should remain the same | 19:26 |
| *** ifarkas has quit IRC | 19:27 | |
| *** markmc has joined #tripleo | 19:33 | |
| *** nati_ueno has quit IRC | 19:33 | |
| lifeless | derekh_: default instance limit is 10 | 19:35 |
| *** CLOUDOUTAGE has joined #tripleo | 19:36 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- R1 is down for an unknown amount of time due to networking issues - https://etherpad.openstack.org/p/cloud-outage | 19:36 |
| *** CLOUDOUTAGE has quit IRC | 19:36 | |
| SpamapS | do we really need the bot spamming us for that? | 19:37 |
| derekh_ | lifeless: I had it chaged to 1000 | 19:37 |
| lifeless | derekh_: ah, ack | 19:37 |
| derekh_ | lifeless: actually looks like its the port quota | 19:37 |
| derekh_ | {"overLimit": {"message": "Maximum number of ports exceeded", "code": 413, "retryAfter": "0"}} | 19:37 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: dpkg: support pkg-map in bin/install-packages https://review.openstack.org/91601 | 19:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Yum: support pkg-map in bin/install-packages https://review.openstack.org/91600 | 19:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: opensuse: support pkg-map in bin/install-packages https://review.openstack.org/91602 | 19:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Update base element to make use of pkg-map https://review.openstack.org/91880 | 19:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Add pkg-map element. https://review.openstack.org/91598 | 19:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Set DISTRO_NAME in OS environment.d https://review.openstack.org/91599 | 19:39 |
| lifeless | SpamapS: its a communications tool to expose status; I thought you wanted more folk to know where things were at? | 19:39 |
| derekh_ | lifeless: argh, I hadn't factor in the testenv ports when calcullating a quota for openstack-nodepool (they belong to the nodepool tenant also) | 19:41 |
| dprince | rbrady: https://review.openstack.org/#/c/91598/ now supports a new OS family option per our conversation at the summit | 19:41 |
| *** jang has joined #tripleo | 19:42 | |
| rbrady | dprince: thanks! I'll take a look | 19:43 |
| SpamapS | lifeless: I do. I don't want people to be overwhelmed with the obvious. | 19:43 |
| SpamapS | Have seen nagios notifications become noise for the same reason many times. | 19:43 |
| *** rlandy_ has quit IRC | 19:43 | |
| dprince | rbrady: appreciate that feedback. Also, see here for how it plays out: https://review.openstack.org/#/c/91880/6/elements/base/pkg-map | 19:43 |
| * dprince things the pkg-map ship has now sailed! | 19:44 | |
| lifeless | SpamapS: so if I hadn't known to ask | 19:44 |
| * dprince thinks even! | 19:44 | |
| lifeless | SpamapS: I wouldn't have known we were still down | 19:44 |
| rbrady | dprince: I saw the new layout in the script comments | 19:44 |
| *** dshulyak_ has quit IRC | 19:45 | |
| *** dprince has quit IRC | 19:45 | |
| SpamapS | lifeless: mmmk. If it takes telling people 6 ways.. so be it. I think the problem is people not prioritizing it. | 19:46 |
| *** andreaf has joined #tripleo | 19:46 | |
| greghaynes | SpamapS: See hipchat? | 19:47 |
| greghaynes | before you pass out again :) | 19:47 |
| SpamapS | greghaynes: ah yes. | 19:47 |
| SpamapS | greghaynes: o-a-c has a misfeature .. which is that it lets you run it without OS_CONFIG_FILES or OS_CONFIG_FILES_LIST set, so it finds them itself.. | 19:48 |
| SpamapS | greghaynes: so if you run it from os-collect-config ... you may get a different set of json files | 19:48 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-specs: Configurable /mnt/state https://review.openstack.org/94876 | 19:49 |
| greghaynes | ah, fun | 19:50 |
| *** nati_ueno has joined #tripleo | 19:50 | |
| jdob | so far I am really liking this specs repo and review process | 19:55 |
| *** jang has quit IRC | 19:55 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Separate out BareMetalNode from IronicNode https://review.openstack.org/95881 | 19:58 |
| *** nati_ueno has quit IRC | 20:00 | |
| openstackgerrit | Martin Geisler proposed a change to openstack/tuskar: Use Emacs-friendly file variable to set file encoding https://review.openstack.org/95886 | 20:02 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-specs: Unit Testing TripleO Projects https://review.openstack.org/95887 | 20:02 |
| goneri | bnemec: for https://review.openstack.org/#/c/91818/ , indeed, the $rc is missing. Can I prepare the fault() patch or maybe you are already working on that? | 20:05 |
| bnemec | goneri: I already pushed one: https://review.openstack.org/#/c/95850/ | 20:06 |
| bnemec | It'll need a rebase already though because a couple of changes were in the gate when I pushed it. | 20:07 |
| *** CLOUDOUTAGE has joined #tripleo | 20:07 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- R1 is down for an unknown amount of time due to networking issues - https://etherpad.openstack.org/p/cloud-outage | 20:07 |
| *** CLOUDOUTAGE has quit IRC | 20:07 | |
| *** BadCub01_ has joined #tripleo | 20:08 | |
| goneri | bnemec: thanks :) | 20:08 |
| *** BadCub has quit IRC | 20:08 | |
| *** nati_ueno has joined #tripleo | 20:12 | |
| *** hashar has quit IRC | 20:12 | |
| *** chuckC has quit IRC | 20:13 | |
| jrist | tzumainn: question about your 95881 change | 20:19 |
| jrist | tzumainn: is there a blueprint? is that the nodes-cleanup? | 20:19 |
| *** boris-42 has quit IRC | 20:21 | |
| *** boris-42 has joined #tripleo | 20:21 | |
| tzumainn | jrist, yep | 20:23 |
| tzumainn | the review doesn't link to it? | 20:23 |
| tzumainn | I may have failed in my commit message | 20:23 |
| tzumainn | woah, I did fail | 20:23 |
| tzumainn | one sec | 20:23 |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Separate out BareMetalNode from IronicNode https://review.openstack.org/95881 | 20:24 |
| tzumainn | jrist, okay, it's fixed :P | 20:25 |
| *** hewbrocca has quit IRC | 20:38 | |
| *** CLOUDOUTAGE has joined #tripleo | 20:38 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- R1 is down for an unknown amount of time due to networking issues - https://etherpad.openstack.org/p/cloud-outage | 20:38 |
| *** CLOUDOUTAGE has quit IRC | 20:38 | |
| jrist | tzumainn: did I ask how you associated it to the bp? | 20:39 |
| jrist | tzumainn: if not I am now | 20:39 |
| tzumainn | oh, in the commit message, you add | 20:40 |
| tzumainn | Implements: blueprint nodes-cleanup | 20:40 |
| jrist | tzumainn: thanks | 20:40 |
| *** jtomasek has joined #tripleo | 20:41 | |
| tzumainn | no problem! | 20:41 |
| *** BadCub has joined #tripleo | 20:41 | |
| *** BadCub has quit IRC | 20:42 | |
| *** jprovazn has quit IRC | 20:44 | |
| *** hashar has joined #tripleo | 20:50 | |
| *** jdob has quit IRC | 20:50 | |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1321943 | 21:00 |
| uvirtbot | Launchpad bug 1321943 in tripleo "Ceilometer Swift polling on overcloud control node fails with a 403 forbidden error" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1318767 | 21:00 |
| uvirtbot | Launchpad bug 1318767 in tripleo "apache element SSL cert check fails" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1314978 | 21:00 |
| uvirtbot | Launchpad bug 1314978 in tripleo "Cloud vm not pingable after overcloud upgrade " [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1315355 | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1323167 | 21:00 |
| uvirtbot | Launchpad bug 1315355 in tripleo "Upgrade of overcloud failed with "Connection to neutron failed: Maximum attempts reached"" [Undecided,New] | 21:00 |
| uvirtbot | Launchpad bug 1323167 in tripleo "overcloud can't create an instance by neutron error" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1319473 | 21:00 |
| uvirtbot | Launchpad bug 1319473 in tripleo "devtest_testenv.sh doesn't honour overcloud_computescale or overcloud_controlscale" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1316675 | 21:00 |
| uvirtbot | Launchpad bug 1316675 in tripleo "Saving a devtest VM results in error" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1320090 | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| uvirtbot | Launchpad bug 1320090 in tripleo "pxe_ephemeral_format': u'ext4 left on nodes after deploys deleted" [Undecided,New] | 21:00 |
| *** hashar has quit IRC | 21:01 | |
| *** julim has quit IRC | 21:01 | |
| *** Penick has quit IRC | 21:02 | |
| tchaypo | I totes intended to be here at not-the-meeting-time to make sure there was no confusion | 21:02 |
| lifeless | ok, ensure-bridge is being called with the wrong interface | 21:03 |
| tchaypo | But it seems my body was overly excited about not having to get up at 5am, so it didn't | 21:03 |
| lifeless | erm, no, no its not | 21:03 |
| *** hashar has joined #tripleo | 21:03 | |
| lifeless | br-ex is layered on vlan25 | 21:03 |
| *** Penick has joined #tripleo | 21:04 | |
| *** chuckC has joined #tripleo | 21:07 | |
| *** nati_ueno has quit IRC | 21:08 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack/tripleo-image-elements: Make the image name configurable https://review.openstack.org/95791 | 21:08 |
| *** CLOUDOUTAGE has joined #tripleo | 21:09 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- R1 is down for an unknown amount of time due to networking issues - https://etherpad.openstack.org/p/cloud-outage | 21:09 |
| *** CLOUDOUTAGE has quit IRC | 21:09 | |
| *** nati_ueno has joined #tripleo | 21:11 | |
| *** e0ne has quit IRC | 21:11 | |
| *** markmc has quit IRC | 21:11 | |
| andreaf | derekh_: so sorry I uploaded the wrong patchset - I fixed it now https://review.openstack.org/95791 | 21:11 |
| *** e0ne has joined #tripleo | 21:11 | |
| *** jtomasek has quit IRC | 21:12 | |
| derekh_ | andreaf: ok thanks, will take a look in a bit | 21:12 |
| *** e0ne has quit IRC | 21:14 | |
| lifeless | region one is back | 21:20 |
| lifeless | we hope | 21:20 |
| lifeless | hp1 that is | 21:20 |
| *** e0ne has joined #tripleo | 21:21 | |
| derekh_ | lifeless: did we do anything about the high failure rate ? | 21:21 |
| lifeless | derekh_: AFAICT this is fallout from the stuff done to fix it. | 21:22 |
| derekh_ | lifeless: ok cool | 21:22 |
| lifeless | I will look at http://goodsquishy.com/downloads/tripleo-jobs.html in an hour or so | 21:22 |
| *** matty_dubs is now known as matty_dubs|gone | 21:22 | |
| derekh_ | cooleo | 21:23 |
| derekh_ | lifeless: you might not see anything looks like zuul is running nothing at the moment http://status.openstack.org/zuul/ | 21:25 |
| lifeless | I believe its being restarted | 21:28 |
| lifeless | per discussion in -infra | 21:28 |
| lifeless | jogo: ping | 21:28 |
| *** e0ne has quit IRC | 21:29 | |
| *** jesusaurus has joined #tripleo | 21:34 | |
| -openstackstatus- NOTICE: Zuul is offline due to an operational issue; ETA 2200 UTC. | 21:34 | |
| *** ChanServ changes topic to "Zuul is offline due to an operational issue; ETA 2200 UTC." | 21:34 | |
| *** nati_ueno has quit IRC | 21:37 | |
| *** Penick has quit IRC | 21:38 | |
| *** jang1 has quit IRC | 21:39 | |
| *** davidlenwell_ has joined #tripleo | 21:40 | |
| *** noslzzp has quit IRC | 21:41 | |
| *** nati_ueno has joined #tripleo | 21:43 | |
| *** davidlenwell_ has quit IRC | 21:44 | |
| *** davidlenwell has quit IRC | 21:45 | |
| *** davidlenwell has joined #tripleo | 21:47 | |
| *** ChanServ sets mode: +v davidlenwell | 21:50 | |
| *** edmund has quit IRC | 21:57 | |
| *** shakamunyi has quit IRC | 22:03 | |
| *** hashar has quit IRC | 22:04 | |
| *** nati_ueno has quit IRC | 22:05 | |
| *** yamahata has joined #tripleo | 22:07 | |
| lifeless | adam_g: ping | 22:09 |
| adam_g | lifeless, hi | 22:10 |
| lifeless | adam_g: are you poking at the 306? rack | 22:10 |
| adam_g | lifeless, yes | 22:10 |
| lifeless | cool | 22:10 |
| *** shakamunyi has joined #tripleo | 22:10 | |
| adam_g | lifeless, well, greg is. im on the other | 22:10 |
| lifeless | whats the situation at the moment, and can I be of any assistance | 22:10 |
| greghaynes | situation is waiting for an overcloud to deploy | 22:10 |
| greghaynes | there is a theory that we were hitting an issue with resource timeouts after the ironic patch to serialise dd's to nodes | 22:11 |
| adam_g | what greghaynes said | 22:11 |
| *** ChanServ changes topic to "ci-overcloud for HP region is down due to network issues | https://etherpad.openstack.org/p/tripleo-ci-r1-trusty | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 22:11 | |
| -openstackstatus- NOTICE: Zuul is started and processing changes that were in the queue when it was stopped. Changes uploaded or approved since then will need to be re-approved or rechecked. | 22:11 | |
| adam_g | lifeless, there also appears to be a considerable lag between when a node has applied its config and sent its callback to heat, and when heat actually registers the resource as having completed creation. | 22:12 |
| lifeless | ok | 22:13 |
| lifeless | so the patch to ironic | 22:13 |
| lifeless | do we have timing data (at the bare ironic layer) showing it improves tings? | 22:13 |
| adam_g | lifeless, timing data? wrt time, it doesnt improve anything. but limiting the amount of concurrent i/o prevents conductor from falling over in weird ways | 22:14 |
| greghaynes | it actually makes timing a ton worse | 22:14 |
| lifeless | so that implies we're not utilising the network all that well today | 22:15 |
| lifeless | adam_g: what ways was the conductor falling over? | 22:15 |
| adam_g | coincidentally, dan hit what seems to be a very similar on the front end of the deploy and has proposed the same for image download/conversion https://review.openstack.org/#/c/95213/ | 22:15 |
| adam_g | lifeless, RPCs timing out, locking issues, tftp connections failing | 22:16 |
| adam_g | i was seeing a ~10% fail rate on deploying ~30 instances at once | 22:17 |
| adam_g | lifeless, i dont think completely serializing the writeouts is the correct long term solution. id prefer ironic allow deploy(), or at least the i/o heavy image writeout, to happen in a worker pool of configurable size, to allow admins to tweak to their environment. | 22:18 |
| greghaynes | Sounds like we either need a magic number for rate limiting, network prioritization, or to do the pie in the sky multicast-style distribution.... | 22:18 |
| lifeless | there's a spec | 22:18 |
| lifeless | :) | 22:18 |
| greghaynes | "Theres a spec for that"? | 22:18 |
| lifeless | yeah | 22:18 |
| lifeless | so the thing that confuses me | 22:19 |
| lifeless | is that we didn't see that 10% error rate at the summit | 22:19 |
| lifeless | on friday | 22:19 |
| adam_g | no idea? | 22:22 |
| *** ddieterly has quit IRC | 22:23 | |
| *** noslzzp has joined #tripleo | 22:25 | |
| greghaynes | Could our heat performance improvements exacerbated the problem? | 22:26 |
| greghaynes | *shrug* | 22:26 |
| adam_g | greghaynes, i was able to hit failures without heat, simply 'nova boot --num-instances 30' | 22:28 |
| greghaynes | ugh, so the cloud-init people seem to want to find a way to let that cloudsigma stay on by default | 22:28 |
| lifeless | http://www.gossamer-threads.com/lists/linux/kernel/1426057 may be have an answer for the cmd_alloc failures | 22:28 |
| lifeless | greghaynes: aahahahahahahaha | 22:28 |
| greghaynes | adam_g: Yea, but im wondering if we were causing some kind of serialisation via abusing heat | 22:28 |
| greghaynes | Im not really sure you can sanely spew data over ttyS1 without knowing whats on the other end and not expect to cause all sorts of random problems | 22:29 |
| greghaynes | I await the person who cloud-init's their electon microscope control machine and wonders why their hardware starts going bazerk every time they turn their computer on | 22:30 |
| adam_g | greghaynes, well, the contention i was hitting happens long before cloud-init (assuming you're talking about the cloud-init / ttyS thing) | 22:30 |
| greghaynes | adam_g: Its just the data source in cloud-init polling ttyS1 for datas, right? | 22:31 |
| adam_g | greghaynes, yeah, on first boot. | 22:31 |
| adam_g | that will eventually be fixed in cloud-init in trusty, but ive thrown up https://review.openstack.org/#/c/95598/ in the meantime (and have been polling this down into our rack tests to avoid the bug for now) | 22:32 |
| *** nati_ueno has joined #tripleo | 22:32 | |
| greghaynes | adam_g: Yep, so there was a proposed fix of just turning off that data source by default, now theres talk of trying to find a way to detect with the data source being on | 22:32 |
| *** nati_uen_ has joined #tripleo | 22:42 | |
| *** nati_ueno has quit IRC | 22:42 | |
| *** Penick has joined #tripleo | 22:43 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Add details of which OS releases are tested in CI https://review.openstack.org/92749 | 22:48 |
| *** derekh_ has quit IRC | 22:51 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Reformat README as ReST and include in built docs https://review.openstack.org/95308 | 22:51 |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Reformat README as ReST and include in built docs https://review.openstack.org/95308 | 22:53 |
| jogo | lifeless: pong | 22:54 |
| jogo | lifeless: I would be happy to help | 22:54 |
| jogo | lifeless: what is the current status of the logs? | 22:55 |
| lifeless | they get captured like so http://logs.openstack.org/78/93078/3/check-tripleo/check-tripleo-overcloud-precise/fcb065e/ | 22:56 |
| lifeless | AFAICT even console.html isn't indexed | 22:56 |
| lifeless | we need the seed_logs.tar.xz to be indexed | 22:57 |
| lifeless | (and the undercloud and overcloud node logs too, but thats incremental wins | 22:57 |
| lifeless | jogo: ^ | 22:59 |
| *** andreaf has quit IRC | 22:59 | |
| lifeless | greghaynes: adam_g: so how did that deploy go? | 23:01 |
| adam_g | lifeless, still going | 23:01 |
| lifeless | I'm interested in the error nick reported | 23:01 |
| lifeless | a couple days back | 23:01 |
| jogo | lifeless: cool so that shouldn't be too bad I think | 23:01 |
| adam_g | lifeless, which? | 23:02 |
| jogo | lifeless: https://etherpad.openstack.org/p/tripleo-logs-elasticsearch | 23:02 |
| greghaynes | got to CREATE_IN_PROGRESS on all the things, some are starting to CREAT_COMPLETE (structured deployment resources) | 23:02 |
| greghaynes | so looking good | 23:02 |
| greghaynes | gotta afk for a bit, feel free to poke at rack 306, theres a screen open there | 23:03 |
| lifeless | adam_g: ' | 23:04 |
| lifeless | >However, heat failed at the config stage on all compute hosts, | 23:04 |
| lifeless | >controller was fine. (system left with this fail state) Think Adam is ' | 23:04 |
| adam_g | lifeless, i believe the heat stack was hitting its default 1hr time out | 23:04 |
| jogo | lifeless: so I don't think getting console logs added should be too bad | 23:05 |
| lifeless | adam_g: / greghaynes: so whats the timing data with serialisation today ? | 23:05 |
| lifeless | jogo: cool | 23:05 |
| jogo | but seed_logs.tar.gz may be a litte more challanging | 23:05 |
| adam_g | lifeless, we're testing an increased timeout on both racks now, to account for the serialized image writeouts | 23:05 |
| jogo | as those are seperate logs | 23:05 |
| jogo | I'll start with just getting console logs added | 23:05 |
| lifeless | jogo: yes, and in fact tarballs with a whole machines logs under it | 23:05 |
| adam_g | lifeless, ill hopefully know in a bit. running a timed overcloud deploy now. | 23:06 |
| lifeless | ok | 23:06 |
| jogo | lifeless: yeah, well lets talk after I can get console logs added | 23:06 |
| lifeless | kk | 23:06 |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Add details of which OS releases are tested in CI https://review.openstack.org/92749 | 23:08 |
| adam_g | looks like rack 306 stack creation is about done | 23:09 |
| *** edmund has joined #tripleo | 23:12 | |
| lifeless | so a bunch of this would be better if we had an HA control plane | 23:13 |
| lifeless | 3x the heat capacity, 3x the ironic conductors | 23:13 |
| *** yamahata has quit IRC | 23:16 | |
| adam_g | lifeless, yeah | 23:18 |
| mordred | lifeless: I thought greghaynes fixed HA like 2 months ago :) | 23:19 |
| greghaynes | hahaha | 23:19 |
| greghaynes | ETIME | 23:19 |
| adam_g | greghaynes, FYI the cloud-init fix didnt get pulled into image build on 306, one of the nodes (.162) hit the issue and i had to cycle power | 23:20 |
| greghaynes | :_( | 23:20 |
| mordred | greghaynes: dude. last time I made something HA, it took me 4 hours. what's the problem? ;) | 23:20 |
| mordred | and that was uphill both ways in the snow! | 23:20 |
| lifeless | mordred: we were | 23:21 |
| greghaynes | If we just used mongodb and redis it would be done by now | 23:21 |
| lifeless | then this virtual team thing came up and sidetracked | 23:21 |
| adam_g | greghaynes, | 4ac984e6-3874-4344-8958-54b59816b09c | overcloud | CREATE_COMPLETE | 2014-05-27T22:02:15Z | | 23:22 |
| greghaynes | WAT! | 23:22 |
| adam_g | thoughts on making the overcloud 'heat stack-create' timeout configurable in tripleo-incubator? | 23:23 |
| greghaynes | Dont we have an extra opts var we could use there? | 23:24 |
| greghaynes | I think the actual fix for that should go in at a lower layer, but extra_opts is probably a fine band-aid for now | 23:26 |
| lifeless | the timeout is a fugly thing | 23:28 |
| lifeless | perhaps we should just make it 5hours or something | 23:28 |
| lifeless | and document 'if your deploy layer cannot deploy to all ndoes in your cloud in < 5 hours, your deploy layer is undersized' ? | 23:28 |
| *** noslzzp has quit IRC | 23:29 | |
| adam_g | id propose we make the actual heat timeout very long/unlimited, and allow tripleo to timeout on the client side based on some knob | 23:29 |
| *** Penick has quit IRC | 23:30 | |
| *** michchap has joined #tripleo | 23:32 | |
| *** TravT has quit IRC | 23:35 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Clean up any existing environment first https://review.openstack.org/92038 | 23:37 |
| lifeless | greghaynes: so seriously | 23:43 |
| lifeless | greghaynes: where are we at? The vip stuff seems like the next logical step, no ? | 23:43 |
| StevenK | lifeless: Do you want to approve and land https://review.openstack.org/#/c/95445/ , or recheck due to f20 breaking due to name resolution? | 23:46 |
| *** ddieterly has joined #tripleo | 23:49 | |
| jogo | lifeless: for https://bugs.launchpad.net/python-novaclient/+bug/1323862 sigmavirus24 (Ian Cordasco) may be able to help | 23:49 |
| uvirtbot | Launchpad bug 1323862 in python-novaclient "novaclient can get hung when used for extended periods" [Undecided,New] | 23:49 |
| jogo | if this is an issue with how we use the requests library | 23:50 |
| *** eguz has joined #tripleo | 23:51 | |
| greghaynes | lifeless: Yes, and its fairly far along, I think the biggest blocker right now is the HAProxy spec which seems like its near complete | 23:51 |
| mordred | jogo: I feel like I filed that bug ... | 23:52 |
| mordred | oh - bah. lifeless filed that bug | 23:53 |
| lifeless | jogo: it is apparently supported in the plumbing | 23:53 |
| lifeless | mordred: I could not find a existing bug | 23:53 |
| lifeless | jogo: but something goeth wrong | 23:53 |
| *** ddieterly has quit IRC | 23:53 | |
| mordred | lifeless: we hit this in nodepool, and traced it down to novaclient - but I think we decided there is actually a bug in requests | 23:53 |
| openstackgerrit | Adam Gandelman proposed a change to openstack/tripleo-incubator: Allow user-specified timeout for stack creation https://review.openstack.org/95968 | 23:54 |
| lifeless | mordred: yes, I know :) | 23:54 |
| jogo | lifeless: so I have it on good word that sigmavirus24 will be joining openstack soon | 23:54 |
| lifeless | mordred: but we need a bug where it surfaces | 23:55 |
| lifeless | jogo: cool | 23:55 |
| jogo | and he maintains requests, requests-kerberos, betamax, requests-toolbelt | 23:55 |
| *** eghobo has quit IRC | 23:55 | |
| adam_g | lifeless, 71m12.080s on a 28 node overcloud /w serialized dd's | 23:55 |
| adam_g | well 71m till a post-stack-creation Neutron error | 23:56 |
| greghaynes | dshulyak: Would you be offended if I try and parameterize the ports in https://review.openstack.org/#/c/93554 ? | 23:57 |
| mordred | lifeless: https://review.openstack.org/#/c/74943/ | 23:57 |
| mordred | lifeless: here is where we came up against it before | 23:58 |
| mordred | jogo: ^^ | 23:58 |
| mordred | and then we abandoned the patch because it was a hack around a behavior that either novaclient or requests should do better ... but I believe we didn't get aroudn to filing another bug | 23:58 |
| jogo | mordred: oh right the firewall fun | 23:59 |
| mordred | yah | 23:59 |
| mordred | but it's the same thing | 23:59 |
| mordred | requests is trying to be "helpful" - but something in the middle decides to make the tcp conenction go away | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!