| tchaypo | Lotus907efi: https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/ubuntu/README.md suggests that the ubuntu element will grab the cloud image | 00:01 |
|---|---|---|
| Lotus907efi | ok, I will go read that | 00:01 |
| Lotus907efi | thanks | 00:01 |
| *** Hefeweiz1n has quit IRC | 00:02 | |
| tchaypo | it looks like https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/ubuntu/root.d/10-cache-ubuntu-tarball does the downloading | 00:02 |
| Lotus907efi | hmm, not really much there | 00:02 |
| *** andreaf has quit IRC | 00:02 | |
| Lotus907efi | it doesn't say for instance how those cloud images are used | 00:02 |
| greghaynes | woooo 'overcloud | DELETE_FAILED' | 00:03 |
| *** eghobo has quit IRC | 00:03 | |
| *** newell has quit IRC | 00:07 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Standardise location of environment password/rc files. https://review.openstack.org/83250 | 00:08 |
| tchaypo | 6 days, 12 revisions, I've settled into a comfortable habit of checking that each morning and doing a new update based on the new nits people found yesterday | 00:13 |
| *** cwolferh_ has quit IRC | 00:13 | |
| tchaypo | To me this feels like a good thing - it's getting thorough review and when it lands it will be solid. I wonder if this would be disconcerting to someone who wasn't used to this review process though | 00:14 |
| mordred | tchaypo: almost certainly - it's a learning curve for new folks for sure | 00:15 |
| tchaypo | I can't think of anything we could do to make that less of a curve, beyond looking out for new people and giving them a bit of hand-holding through their first review or two | 00:17 |
| tchaypo | I'm guessing most people will already be familiar with doing thorough pre-commit reviews though, so it's probably not too big of an issue anyway | 00:18 |
| mordred | tchaypo: we do send people a welcome email on their first commit sent in | 00:19 |
| mordred | :) | 00:19 |
| tchaypo | yeah | 00:20 |
| tchaypo | I beleive that started just *after* my first commit | 00:20 |
| mordred | hehe | 00:20 |
| tchaypo | *grumps* | 00:20 |
| mordred | you want me to send you a copy of the text in an email? | 00:20 |
| tchaypo | I'd rather you sent me a pony | 00:21 |
| clarkb | tchaypo: really? seems like most people don't do pre commit reviews | 00:21 |
| tchaypo | I even have a nice patch of grass outside where it could graze | 00:21 |
| mordred | tchaypo: I will attempt to fit a pony into an email | 00:21 |
| tchaypo | clarkb: maybe my perspective is skewed; my second-last employer used it (and as far as i can tell the workflow I got used to there also inspired the gerrit workflow we use on openstack); my last employer used pre-commit reviews in some cases (I always tried to get pre-commit reivew on my own changes, but it wasn't mandatory on my team) | 00:24 |
| tchaypo | mordred: Be careful with providing ponies, it just raises expectations | 00:26 |
| tchaypo | at second-last-job someone arranged a pony for our team for sysadmin appreciation day | 00:26 |
| tchaypo | but then Vint Cerf wandered past and one of my workmates got a photo with him *and* the pony | 00:26 |
| *** matsuhashi has joined #tripleo | 00:27 | |
| tchaypo | https://www.flickr.com/photos/joanne-psi/482443475/?rb=1 | 00:27 |
| tchaypo | so if you end up providing the pony and it doesn't come with a side-helping of Father of the Internet I'm just going to be disappointed | 00:28 |
| mordred | fair | 00:28 |
| lifeless | StevenK: what was the pony name @ Canonical ? | 00:30 |
| StevenK | Woody | 00:34 |
| tchaypo | I can't see any problems with requesting that someone give you that particular pony. | 00:34 |
| tchaypo | none at all. | 00:34 |
| *** giulivo has quit IRC | 00:47 | |
| *** CaptTofu has joined #tripleo | 00:49 | |
| *** UtahDave has left #tripleo | 00:53 | |
| *** CaptTofu has quit IRC | 00:54 | |
| *** zigo has quit IRC | 00:59 | |
| *** zigo has joined #tripleo | 01:01 | |
| *** morazi has quit IRC | 01:16 | |
| *** CaptTofu has joined #tripleo | 01:17 | |
| *** kiall has joined #tripleo | 01:19 | |
| *** nosnos has joined #tripleo | 01:26 | |
| tchaypo | lifeless: When you were investigating the dhcp issues yesterday, I think you mentioned you found some rules weren't beint re-applied - can you tell me more about that? | 01:38 |
| tchaypo | I'm pretty sure you found that they weren't being set up even after restarting neutron-openvswitch-agent though, right? | 01:39 |
| lifeless | tchaypo: n-o-a was crashing on startup | 01:39 |
| lifeless | tchaypo: due to a missing lock path setting in the undercloud | 01:39 |
| lifeless | woooo billions of slaves | 01:40 |
| tchaypo | yep, that's not my problem. thanks | 01:40 |
| tchaypo | i mean, not the problem I'm seeing. "not my problem" could be read other ways. | 01:41 |
| lifeless | indeed | 01:41 |
| *** fandi has joined #tripleo | 01:42 | |
| *** kiall has quit IRC | 01:47 | |
| *** CaptTofu has quit IRC | 01:51 | |
| *** CaptTofu has joined #tripleo | 01:58 | |
| *** noslzzp has quit IRC | 02:05 | |
| *** weshay has quit IRC | 02:06 | |
| *** slagle has quit IRC | 02:13 | |
| *** slagle has joined #tripleo | 02:15 | |
| *** slagle has quit IRC | 02:28 | |
| *** lifeless changes topic to "tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 02:30 | |
| *** fandi has quit IRC | 02:30 | |
| *** spzala has quit IRC | 02:33 | |
| *** fandi has joined #tripleo | 02:43 | |
| lifeless | StevenK: https://review.openstack.org/#/c/79043/ needs rebasing | 02:46 |
| *** fandi has quit IRC | 02:51 | |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 03:00 |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Make it possible to pass real hardware in. https://review.openstack.org/83188 | 03:04 |
| *** fandi has joined #tripleo | 03:05 | |
| lifeless | SpamapS: reviewed https://review.openstack.org/#/c/83614/ | 03:08 |
| *** killer_prince has quit IRC | 03:08 | |
| lifeless | tchaypo: http://paste.openstack.org/show/74692/ is what I was seeing | 03:08 |
| StevenK | lifeless: derekh would like the incubator portion to land first | 03:11 |
| lifeless | StevenK: yes, but it still needs a rebase :) | 03:11 |
| *** ramishra has joined #tripleo | 03:12 | |
| *** yamahata has joined #tripleo | 03:13 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/tripleo-ci: ensure-test-env now uses devtest_testenv https://review.openstack.org/83285 | 03:14 |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Populate seed.ip in the testenv JSON https://review.openstack.org/79043 | 03:15 |
| *** ramishra_ has joined #tripleo | 03:18 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Add some clarity to the first-time user experience https://review.openstack.org/83294 | 03:20 |
| *** ramishra has quit IRC | 03:21 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Fixup testenv config for interface names. https://review.openstack.org/84326 | 03:21 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Fixup HP region testenv config. https://review.openstack.org/84075 | 03:21 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Remove options not present in the heat template. https://review.openstack.org/84074 | 03:21 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Performance tweaks for testenv deploy script. https://review.openstack.org/84073 | 03:21 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Tune deploy-ci-overcloud a little. https://review.openstack.org/84076 | 03:21 |
| *** rpodolyaka1 has joined #tripleo | 03:21 | |
| lifeless | boom | 03:22 |
| *** matsuhashi has quit IRC | 03:22 | |
| lifeless | SpamapS: Ng: GheRivero: thoughts solicited: https://review.openstack.org/#/c/84049/ | 03:24 |
| lifeless | SpamapS: https://review.openstack.org/#/c/83659/ | 03:25 |
| *** rpodolyaka1 has quit IRC | 03:26 | |
| lifeless | Ng: GheRivero: https://review.openstack.org/#/c/82741/ | 03:29 |
| *** fandi has quit IRC | 03:31 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/os-cloud-config: Add CLI scripts for init-{heat,keystone,swift} https://review.openstack.org/84330 | 03:35 |
| lifeless | jogo: can you re-eyeball https://review.openstack.org/#/c/74600/ ? | 03:36 |
| *** nosnos has quit IRC | 03:37 | |
| lifeless | devananda: NobodyCam: might want to look into why the undercloud job on https://review.openstack.org/#/c/84043 failed | 03:38 |
| lifeless | also we need a get-state-from-hosts fix for ironic | 03:39 |
| *** CaptTofu has quit IRC | 03:41 | |
| StevenK | Blah, the call with Paul is at 0300Z | 03:42 |
| StevenK | Er, 0300 AEST | 03:42 |
| *** ramishra_ has quit IRC | 03:44 | |
| *** eghobo has joined #tripleo | 03:44 | |
| *** ramishra has joined #tripleo | 03:45 | |
| tchaypo | really? i thought it was 9am. | 03:47 |
| tchaypo | maybe my OWA calendar doesn't know what timezone i'm in | 03:48 |
| StevenK | I was assuming it was 9am PST | 03:50 |
| tchaypo | i was assuming that OWA was showing me that part of the message in a localized format | 03:51 |
| tchaypo | especially when it offers an "add this to my calendar" button which puts it in my calendar at 9am | 03:51 |
| StevenK | That may be quite false :-) | 03:51 |
| tchaypo | i would be entirely unsurprised to be wrong | 03:55 |
| tchaypo | clarkb: still around? did you get the email about the meeting with paul from the ospo? | 03:56 |
| clarkb | tchaypo: did they arrive after 0000UTC? | 03:57 |
| clarkb | if so I didn't and I don't remember getting anything | 03:57 |
| tchaypo | i got it at 10:18am | 03:58 |
| tchaypo | I can't remember if I'm currently GMT+10 or GMT+11 | 03:58 |
| clarkb | tchaypo: date -u for great good | 03:59 |
| StevenK | % date -R | 03:59 |
| StevenK | Tue, 01 Apr 2014 14:58:59 +1100 | 03:59 |
| clarkb | or that | 03:59 |
| tchaypo | so that'd be before 0000, but i think it was only sent to people who said they were interested in talking to him, so maybe you didn't get it | 04:01 |
| tchaypo | oh, it looks like greghaynes got the message | 04:01 |
| tchaypo | so he shold be able to confirm whether it shows 9am for him as well or if it's in his afternoon | 04:01 |
| clarkb | they tried to get me to commit to a thing later this month and do a bunch of paperwork for it today but ERELEASE, ESUMMIT, ENEXTRELEASECYCLE | 04:02 |
| greghaynes | tchaypo: which one? | 04:02 |
| greghaynes | tchaypo: the original calendar event? | 04:02 |
| tchaypo | the updated calendar event | 04:03 |
| greghaynes | 3:15pm | 04:03 |
| greghaynes | in who knows what timexone | 04:03 |
| greghaynes | timezone | 04:03 |
| tchaypo | i think that's time sent | 04:03 |
| tchaypo | but what time does the meeting show as? | 04:03 |
| tchaypo | for the the original was 8am-9am and the update is 9am-10am | 04:03 |
| greghaynes | 3pm | 04:03 |
| greghaynes | 3-4 | 04:04 |
| tchaypo | excellent | 04:04 |
| StevenK | greghaynes: On which day? | 04:04 |
| greghaynes | oh, you AUSers | 04:04 |
| tchaypo | fairly confident OWA is being nice and showing it to us in our local time | 04:04 |
| tchaypo | thanks greghaynes | 04:04 |
| greghaynes | wed the 2nd | 04:05 |
| tchaypo | greghaynes: the preferred term is "aussies" | 04:05 |
| StevenK | Orzies | 04:05 |
| greghaynes | Ausome | 04:05 |
| tchaypo | pronounced the same as "ozzies", there's nothing weird than a USAnian calling us "osseys" | 04:05 |
| greghaynes | wow, so ive been saying it wrong all along | 04:05 |
| tchaypo | y'all do :) | 04:06 |
| *** rpodolyaka1 has joined #tripleo | 04:06 | |
| * tchaypo leaves cafe to check mail | 04:06 | |
| *** killer_prince has joined #tripleo | 04:11 | |
| *** matsuhashi has joined #tripleo | 04:14 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/os-cloud-config: Add CLI scripts for init-{heat,keystone,swift} https://review.openstack.org/84330 | 04:22 |
| *** nosnos has joined #tripleo | 04:23 | |
| *** akuznetsov has joined #tripleo | 04:30 | |
| *** killer_prince2 has joined #tripleo | 04:35 | |
| *** akuznetsov has quit IRC | 04:42 | |
| *** akuznetsov has joined #tripleo | 04:43 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Store seed details in the JSON https://review.openstack.org/79051 | 04:54 |
| StevenK | lifeless: Feel like putting a +A on https://review.openstack.org/#/c/81691/ ? | 04:54 |
| StevenK | lifeless: I'm happy to recheck no bug if you'd like it go through CI again | 04:55 |
| *** cody-somerville has quit IRC | 05:01 | |
| *** matsuhas_ has joined #tripleo | 05:02 | |
| *** matsuhas_ has quit IRC | 05:02 | |
| *** cody-somerville has joined #tripleo | 05:02 | |
| *** cody-somerville has joined #tripleo | 05:02 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Completely subsume POWER_MANAGER into testenv https://review.openstack.org/81691 | 05:03 |
| *** matsuhashi has quit IRC | 05:05 | |
| *** ramishra has quit IRC | 05:05 | |
| *** matsuhashi has joined #tripleo | 05:05 | |
| *** ramishra has joined #tripleo | 05:06 | |
| *** rpodolyaka1 has quit IRC | 05:07 | |
| *** akuznetsov has quit IRC | 05:08 | |
| *** Rakesh5 has joined #tripleo | 05:09 | |
| *** rpodolyaka1 has joined #tripleo | 05:11 | |
| *** killer_prince has quit IRC | 05:14 | |
| *** akuznetsov has joined #tripleo | 05:19 | |
| StevenK | Crumbs, that's an enormous backlog | 05:25 |
| *** CaptTofu has joined #tripleo | 05:30 | |
| tchaypo | backlog? | 05:30 |
| lifeless | tchaypo: I'm guessing at the CI queue | 05:30 |
| tchaypo | oh right. | 05:31 |
| tchaypo | hah. i imagine it is. | 05:31 |
| StevenK | check-tripleo [8 hour history of changes in pipeline] (74) | 05:32 |
| StevenK | That we're only running 3 jobs at the moment is a worry | 05:32 |
| tchaypo | oh myyyyy | 05:33 |
| *** CaptTofu has quit IRC | 05:35 | |
| lifeless | cap of 35 nodes | 05:36 |
| lifeless | which is 7 jobs | 05:36 |
| lifeless | and a bunch stalled deleting that I'm poking at | 05:36 |
| *** akuznetsov has quit IRC | 05:46 | |
| tchaypo | StevenK: what's your preferred method for creating an ubuntu mirror? | 05:49 |
| StevenK | tchaypo: apt-mirror | 05:50 |
| StevenK | tchaypo: Depending on what you're mirroring, the churn is ~3G/day | 05:50 |
| tchaypo | the fact that https://help.ubuntu.com/community/Rsyncmirror references "the new feisty" leads me to believe it's not entirely current | 05:50 |
| StevenK | Hahaha | 05:50 |
| * tchaypo tries to remember the ubuntu releases | 05:51 | |
| tchaypo | i remember joking about feisty at an LCA, but can't remember if it was 07 or 08 | 05:52 |
| StevenK | Warty, Hoary, Breezy, Dapper, Edgy, Feisty, Gutsy, Hardy, Intrepid, Jaunty, Karmic, Lucid, Maverick, Natty, Oneiric, Precise, Quantal, Raring, Saucy, Trusty | 05:52 |
| StevenK | (From memory) | 05:52 |
| clarkb | hardy was the best | 05:53 |
| StevenK | Dapper 4 eva | 05:53 |
| StevenK | Dapper was the first release I contributed to | 05:54 |
| StevenK | tchaypo: I can pastebin a config for apt-mirror if you wish | 05:55 |
| StevenK | There's also an element for it in tie | 05:55 |
| tchaypo | i never object to things that save me thinking | 05:55 |
| StevenK | "Could not submit your paste because your paste contains spam." | 05:56 |
| StevenK | LIES | 05:56 |
| StevenK | tchaypo: http://paste.openstack.org/show/74713/ | 05:56 |
| clarkb | I just remember using hardy and thinking I didn't need to update immediately on the next release, which was a first for me | 05:57 |
| greghaynes | looks spammy to me | 05:57 |
| StevenK | greghaynes: Yeah, I'm certain the URLs contain all sorts of horrible malware. :-D | 05:58 |
| clarkb | and yet http://paste.openstack.org/show/74712/ gets through | 05:58 |
| greghaynes | Its the au. that gives it away | 05:58 |
| StevenK | clarkb: I joined Canonical during the Gutsy release cycle, I didn't stay running Hardy for long | 05:59 |
| * clarkb can't wait for trusty and non segfaulty vim | 05:59 | |
| StevenK | clarkb: Upgrade now? | 06:00 |
| StevenK | The final beta is out, the churn between now and release is going to be small | 06:00 |
| clarkb | StevenK: last I checked xubuntu had a couple nasty issues | 06:00 |
| * clarkb looks again | 06:00 | |
| clarkb | beta2 looks much better I might do that tomorrow | 06:02 |
| StevenK | It takes my machines roughly 3 minutes to grab all the .debs for an upgrade. <3 local mirror | 06:03 |
| tchaypo | non-segfaulty vim? | 06:04 |
| StevenK | tchaypo: One of the yaml files in -infra/config causes vim to segv | 06:05 |
| tchaypo | nice | 06:06 |
| tchaypo | which one? i wanna try! | 06:06 |
| *** rpodolyaka1 has quit IRC | 06:07 | |
| clarkb | tchaypo: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 06:10 |
| StevenK | % vim modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 06:12 |
| StevenK | Vim: Caught deadly signal SEGV | 06:12 |
| clarkb | though it was recently updated and may not cause segfaults anymore | 06:12 |
| clarkb | oh good its still deadly | 06:12 |
| StevenK | Yes, it has not lost its deadlyness | 06:12 |
| tchaypo | first of all that's an erb not a yaml | 06:13 |
| tchaypo | and second, it loads for me | 06:14 |
| StevenK | clarkb: vim -u /dev/null modules/openstack_project/templates/nodepool/nodepool.yaml.erb does not segv, but then you have nothing helpful set. | 06:14 |
| StevenK | tchaypo: Are you running Saucy? | 06:14 |
| tchaypo | lsb_release says 14.04 Trusty Tahr | 06:15 |
| StevenK | tchaypo: That's why -- vim on that file works on Trusty, SEGV's on Saucy | 06:15 |
| tchaypo | oh, so you're not even on the trusty beta? | 06:16 |
| StevenK | Not yet | 06:16 |
| * tchaypo relocates indoors | 06:16 | |
| StevenK | I will probably upgrade my laptop this week | 06:16 |
| lifeless | I've just added trusty to my mirror | 06:18 |
| lifeless | of course universe sources hit the checksum race. FAIL. | 06:18 |
| StevenK | Heh, I've had trusty on my mirror since it hit au.archive | 06:19 |
| *** e0ne has joined #tripleo | 06:23 | |
| tchaypo | okay, i have bandersnatch runnin in a cronjb, exciting. | 06:34 |
| tchaypo | now for apt-mirror | 06:34 |
| tchaypo | according to http://apt-mirror.github.io/ this shouldn't take long | 06:35 |
| *** rpodolyaka1 has joined #tripleo | 06:35 | |
| *** e0ne has quit IRC | 06:37 | |
| *** rdopieralski has joined #tripleo | 06:39 | |
| tchaypo | whee, 168.9Gb to download | 06:43 |
| * tchaypo puts that in the background and carries on working | 06:43 | |
| *** rpodolyaka1 has quit IRC | 06:45 | |
| *** nosnos has quit IRC | 06:45 | |
| *** nosnos has joined #tripleo | 06:46 | |
| *** jprovazn has joined #tripleo | 06:55 | |
| *** mrunge has joined #tripleo | 06:56 | |
| *** matsuhashi has quit IRC | 06:58 | |
| *** matsuhashi has joined #tripleo | 06:58 | |
| tchaypo | I'm staring at https://git.openstack.org/cgit/openstack/tripleo-incubator/tree/scripts/devtest_overcloud.sh#n140 | 06:59 |
| tchaypo | i can see that we're waiting for nova hypervisor-stats to show the right number of nodes available | 06:59 |
| tchaypo | what I can't understand is why we would ever expect to end up with the right number of nodes | 06:59 |
| *** jtomasek has joined #tripleo | 07:01 | |
| tchaypo | ah. I'm guessing this must be setup-baremetal at the end of _undercloud.sh | 07:01 |
| *** jistr has joined #tripleo | 07:02 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Making overcloud init work https://review.openstack.org/83340 | 07:02 |
| *** matsuhashi has quit IRC | 07:03 | |
| *** giulivo has joined #tripleo | 07:05 | |
| *** matsuhashi has joined #tripleo | 07:06 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Making overcloud init work https://review.openstack.org/83340 | 07:09 |
| *** bauzas has joined #tripleo | 07:14 | |
| GheRivero | morning all | 07:15 |
| *** jcoufal has joined #tripleo | 07:16 | |
| *** CaptTofu has joined #tripleo | 07:19 | |
| *** CaptTofu has quit IRC | 07:23 | |
| rpodolyaka | morning | 07:27 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 07:29 |
| *** xuhaiwei has joined #tripleo | 07:32 | |
| *** marun is now known as maru_afk | 07:36 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 07:37 |
| *** ramishra has quit IRC | 07:38 | |
| *** hashar has joined #tripleo | 07:44 | |
| *** eghobo has quit IRC | 07:45 | |
| lifeless | wheeeooo upgrades to trusty are fast with a local mirror | 07:46 |
| GheRivero | is trusty released? | 07:47 |
| lifeless | nah | 07:49 |
| lifeless | but samba4.1 is in it | 07:49 |
| lifeless | gnarr lots of | nova-compute | ci-overcloud-novacompute7-3my4s5pay2om | nova | enabled | down | 2014-04-01T06:56:05.000000 | - | | 07:52 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Allow to set seed node cpus and memory https://review.openstack.org/84078 | 07:53 |
| *** athomas has joined #tripleo | 07:54 | |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Allow to use a cirros image as the end user image https://review.openstack.org/83347 | 08:01 |
| *** lazy_prince has joined #tripleo | 08:02 | |
| *** lazy_prince is now known as killer_prince | 08:02 | |
| *** gcha has joined #tripleo | 08:03 | |
| *** matsuhashi has quit IRC | 08:03 | |
| *** matsuhas_ has joined #tripleo | 08:05 | |
| *** lucasagomes has joined #tripleo | 08:13 | |
| *** CaptTofu has joined #tripleo | 08:20 | |
| *** killer_prince has quit IRC | 08:23 | |
| lifeless | wow | 08:24 |
| lifeless | I think somethings really unhappy here | 08:24 |
| *** derekh has joined #tripleo | 08:24 | |
| *** CaptTofu has quit IRC | 08:24 | |
| lifeless | derekh: so we had CI for a few hours | 08:24 |
| lifeless | derekh: now nova-compute is hung on the hypervisors | 08:24 |
| lifeless | derekh: and by hung, I mean service stop doesn't stop it :) | 08:25 |
| derekh | lifeless: one all of the compute nodes? | 08:25 |
| derekh | * on all of the compute nodes | 08:26 |
| derekh | brb | 08:26 |
| lifeless | nova service-list :( | 08:26 |
| Ng | morning | 08:27 |
| xuhaiwei | I have built the mirror, but how to use it? set a environment variable? | 08:30 |
| lifeless | xuhaiwei: yes, see the pypi element README.md | 08:31 |
| xuhaiwei | I can't understand it well, should I export PYPI_MIRROR_URL= ~/.cache/image-create/pypi/mirror | 08:32 |
| lifeless | if thats where your mirror is, you just need to include the pypi element | 08:32 |
| lifeless | e.g. export DIB_COMMON_ELEMENTS="pypi stackuser" | 08:32 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Allow to pass uncompress option to boot-seed-vm https://review.openstack.org/82772 | 08:33 |
| *** matsuhas_ has quit IRC | 08:33 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Update documentation. https://review.openstack.org/82741 | 08:35 |
| *** andreaf has joined #tripleo | 08:37 | |
| *** ramishra has joined #tripleo | 08:38 | |
| *** matsuhashi has joined #tripleo | 08:38 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Adding overcloud keystone client https://review.openstack.org/84379 | 08:39 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Overcloud initialization https://review.openstack.org/83340 | 08:39 |
| *** martyntaylor has left #tripleo | 08:39 | |
| derekh | lifeless: bad version of nova maybe? | 08:40 |
| lifeless | derekh: I'm hoping not :) | 08:40 |
| lifeless | http://paste.openstack.org/show/74727/ | 08:40 |
| lifeless | but its not context switching | 08:41 |
| lifeless | derekh: oh that reminds me | 08:41 |
| lifeless | derekh: log files, we should grab /mnt/state/var/log as well | 08:42 |
| lifeless | I was going to whip up a patch but the day got away from me | 08:42 |
| lifeless | wow | 08:42 |
| lifeless | VmPeak: 1807004 kB | 08:42 |
| lifeless | wtf is nova-compute /doing/ | 08:42 |
| *** ramishra has quit IRC | 08:43 | |
| xuhaiwei | I find this code in devtest_ramdisk.sh: DIB_COMMON_ELEMENTS=${DIB_COMMON_ELEMENTS:-'stackuser'}, when I run this shell, I should export DIB_COMMON_ELEMENTS="pypi stackuser", but when I run other shells I should export DIB_COMMON_ELEMENTS again if different element is required?? | 08:43 |
| lifeless | If I understand your question correctly. Yes. | 08:43 |
| xuhaiwei | OK, thanks | 08:44 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Add elements to user image via TEST_IMAGE_DIB_EXTRA_ARGS https://review.openstack.org/82739 | 08:45 |
| lifeless | SpamapS: I've put in the review what I want to see | 08:50 |
| lifeless | derekh: hahaha tried to reboot the node... it hung | 08:51 |
| lifeless | derekh: there's something fundamentally wrong here | 08:51 |
| lifeless | 08:51 | |
| lifeless | 1 root 20 0 48828 24m 1436 R 100 0.0 119:07.00 init | 08:51 |
| lifeless | pipe(0x7fff5fddb900) = -1 EMFILE (Too many open files) | 08:52 |
| lifeless | thats from init | 08:52 |
| lifeless | ahahahahahahahahahahahahahahahahahaha | 08:53 |
| lifeless | KEYBUK | 08:53 |
| *** jp_at_hp has joined #tripleo | 08:53 | |
| *** d0ugal_ has quit IRC | 08:53 | |
| lifeless | no upstart changes since oct 9 | 08:53 |
| *** d0ugal_ has joined #tripleo | 08:53 | |
| *** d0ugal_ is now known as d0ugal | 08:54 | |
| *** d0ugal has quit IRC | 08:54 | |
| *** d0ugal has joined #tripleo | 08:54 | |
| *** d0ugal has quit IRC | 08:54 | |
| *** d0ugal has joined #tripleo | 08:54 | |
| Ng | lifeless: ugh, that's awful | 08:55 |
| derekh | lifeless: I'm not sure what you mean | 08:55 |
| lifeless | bug https://bugs.launchpad.net/tripleo/+bug/1300663 | 08:56 |
| uvirtbot | Launchpad bug 1300663 in tripleo "upstart using 100% CPU" [Critical,Triaged] | 08:56 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates gettextutils module from oslo-incubator https://review.openstack.org/84171 | 08:56 |
| * Ng is minded to land 84049 purely on the grounds that any repo called "-incubator" shouldn't have to worry too much about layerings and dependencies | 08:56 | |
| openstackgerrit | A change was merged to openstack/tuskar: Updates test module from oslo-incubator https://review.openstack.org/84172 | 08:56 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates local module from oslo-incubator https://review.openstack.org/84173 | 08:57 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates timeutils module from oslo-incubator https://review.openstack.org/84174 | 08:58 |
| derekh | lifeless: so init has 100's of /dev/ptmx open, are you thinking that the problem | 08:58 |
| *** athomas has quit IRC | 08:59 | |
| lifeless | derekh: 1023 fd's open for init | 08:59 |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 09:00 |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| derekh | # ps -ef | grep flock | wc | 09:00 |
| derekh | 999 8992 65940 | 09:00 |
| lifeless | oh hoh ho dhcp-all-interfaces is spawning per plugged vm port ? | 09:01 |
| lifeless | and not cleaning up properly? | 09:02 |
| lifeless | derekh: this could be a symptom rather than a cause | 09:02 |
| derekh | lifeless: ok | 09:02 |
| lifeless | e..g the <defunct> | 09:02 |
| lifeless | why haven't they been reaped | 09:02 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates importutils module from oslo-incubator https://review.openstack.org/84175 | 09:03 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates jsonutils module from oslo-incubator https://review.openstack.org/84176 | 09:04 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates excutils module from oslo-incubator https://review.openstack.org/84177 | 09:04 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates fileutils module from oslo-incubator https://review.openstack.org/84178 | 09:04 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates log module from oslo-incubator https://review.openstack.org/84179 | 09:04 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates lockutils module from oslo-incubator https://review.openstack.org/84184 | 09:04 |
| openstackgerrit | A change was merged to openstack/tuskar: Updates fixture module from oslo-incubator https://review.openstack.org/84185 | 09:04 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Adding overcloud keystone client https://review.openstack.org/84379 | 09:05 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Overcloud initialization https://review.openstack.org/83340 | 09:05 |
| derekh | lifeless: the first of them (on compute4), was started 1 hour before the compute node stopped checking in with controller, the 999th was started around the same time compute hit problems | 09:05 |
| lifeless | derekh: yeah | 09:05 |
| derekh | still need to find cause though | 09:06 |
| lifeless | see my comment 3 on the bug | 09:06 |
| lifeless | Ng: what happens after start on starting network-interface - when does the service *stop* ? | 09:06 |
| lifeless | Ng: or does exec say 'run till completion' ? | 09:06 |
| Ng | lifeless: the upstart config we have for dhcp-all doesn't specify a stopping condition, so yeah, it's just saying "launch this and let it end when it ends" | 09:07 |
| lifeless | Ng: so a few thoughts | 09:10 |
| lifeless | Ng: this is meant to be skipping these interfaces | 09:12 |
| lifeless | Ng: but clearly its not | 09:12 |
| lifeless | Ng: ahha, I think I know why | 09:12 |
| lifeless | Ng: when you exec and flock | 09:12 |
| *** ramishra has joined #tripleo | 09:13 | |
| lifeless | Ng: you drop the specifier for the same interface | 09:13 |
| lifeless | Ng: so it goes from 'tap foo' to 'check them all' | 09:13 |
| lifeless | Ng: because INTERFACE is set to ${1:-} | 09:13 |
| Ng | lifeless: good spot | 09:13 |
| lifeless | secondly | 09:14 |
| lifeless | you FLOCK before inspecting the interface | 09:14 |
| lifeless | let me work up a patch | 09:14 |
| lifeless | :q | 09:14 |
| lifeless | :q | 09:14 |
| Ng | I don't flock anything, dan added that ;) | 09:14 |
| Ng | but it's entirely probably that I reviewed it and didn't spot the loss of $1 | 09:14 |
| openstackgerrit | jan grant proposed a change to openstack/diskimage-builder: Hard-link multiple identical files. https://review.openstack.org/84384 | 09:15 |
| Ng | a "$@" on the end of the flock command ought to be sufficient to fix the loss of $1, but I was just looking at this now and thinking that some of the inspection should be happening first too | 09:15 |
| Ng | no point flocking and re-execing if we're just going to immediately exit | 09:15 |
| jang | lifeless: as promised last night: turned my hand-cranked script into an element, I think. Note this is currently untested in that form, trying it now. | 09:16 |
| openstackgerrit | lifeless proposed a change to openstack/diskimage-builder: Document a little the concerns for operators. https://review.openstack.org/84385 | 09:17 |
| *** ramishra has quit IRC | 09:17 | |
| *** jcoufal has quit IRC | 09:18 | |
| *** tserong has quit IRC | 09:18 | |
| *** tserong has joined #tripleo | 09:19 | |
| *** tserong has joined #tripleo | 09:19 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 09:19 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Make it possible to pass real hardware in. https://review.openstack.org/83188 | 09:19 |
| openstackgerrit | lifeless proposed a change to openstack/diskimage-builder: Fix resource exhaustion with upstart. https://review.openstack.org/84386 | 09:20 |
| *** ramishra has joined #tripleo | 09:24 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add more tools. https://review.openstack.org/84049 | 09:24 |
| Ng | lifeless: -1 on the resource exhaustion change I'm afraid | 09:24 |
| *** martyntaylor has joined #tripleo | 09:26 | |
| jang | agh | 09:28 |
| jang | I was running devtest _somewhere_ | 09:28 |
| jang | I've lost the window. | 09:29 |
| Ng | always run devtest in screen :) | 09:29 |
| lifeless | Ng: do you hate freedom ? | 09:30 |
| jang | heh | 09:31 |
| Ng | lifeless: passionately | 09:31 |
| StevenK | Of course he does, he runs OS X | 09:31 |
| jang | Is there a screen setting to not define a window bracket, but rather to let my scrollback buffer work> | 09:32 |
| jang | ah yes. "Trashcan full? No big of a deal, just buy another." | 09:32 |
| lifeless | jang: screen *is* my scrollback buffer:) | 09:34 |
| lifeless | jang: some folk think I may be odd. | 09:34 |
| openstackgerrit | lifeless proposed a change to openstack/diskimage-builder: Fix resource exhaustion with upstart. https://review.openstack.org/84386 | 09:34 |
| lifeless | Ng: it is of course untested. | 09:35 |
| lifeless | thats next | 09:35 |
| jang | I know you can use ^A - something (? esc?) to scroll up. I was wondering if there was a method more in keeping with what the various xterm-alikes use. | 09:35 |
| lifeless | ^A[ | 09:36 |
| lifeless | at least for me | 09:36 |
| lifeless | maybe tmux has mouse integration | 09:36 |
| rdopieralski | I always do ^A [ to enter the copy-paste mode | 09:36 |
| lifeless | that would be nice | 09:36 |
| Ng | [ and Esc are equivalent afair, it's just going into the copy mode | 09:36 |
| lifeless | someone should write a terminal multiplexer gui | 09:36 |
| jang | Right. I just prefer kde's (!) point-and-drool keyboard controls for most things. And yet I still use vi. | 09:37 |
| rdopieralski | Ng: actually esc is ^[ | 09:37 |
| lifeless | ok, trying a reboot of this machine | 09:38 |
| Ng | rdopieralski: I'm saying in terms of screen bindings. ^a [ and ^a Esc are bound to the same feature | 09:38 |
| rdopieralski | Ng: interesting | 09:38 |
| Ng | rdopieralski: but I admit I wasn't very clear about that :) | 09:39 |
| *** jistr has quit IRC | 09:39 | |
| *** gcha has quit IRC | 09:40 | |
| Ng | lifeless: once more I am stamping your freedom into the ground with my bead-blasted aluminium jackboot :/ | 09:43 |
| lifeless | hhahahhahahahahahahahahah | 09:44 |
| derekh | Ng: isn't that one ok, because the interface name is passed into enable_interface? | 09:45 |
| openstackgerrit | lifeless proposed a change to openstack/diskimage-builder: Fix resource exhaustion with upstart. https://review.openstack.org/84386 | 09:45 |
| lifeless | derekh: $0 will be wrong | 09:46 |
| lifeless | derekh: I think | 09:46 |
| derekh | lifeless: no it isn't | 09:46 |
| derekh | lifeless: I checked befor I added a +2 | 09:46 |
| lifeless | ah | 09:46 |
| lifeless | well I pushed a new variant anyhow that is perhaps easier to reason about | 09:46 |
| *** morganfainberg is now known as morganfainberg_Z | 09:46 | |
| Ng | hmm, derekh is right, but I still think this new rev is better | 09:46 |
| Ng | sneaking $@ through several layers is not exactly clear | 09:47 |
| derekh | lifeless: yup, it looks easier to the eye now | 09:47 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Fix resource exhaustion with upstart. https://review.openstack.org/84386 | 09:50 |
| *** athomas has joined #tripleo | 09:50 | |
| *** jcoufal has joined #tripleo | 09:50 | |
| openstackgerrit | jan grant proposed a change to openstack/diskimage-builder: Hard-link multiple identical files. https://review.openstack.org/84384 | 09:51 |
| lifeless | derekh: i'm inclined to fix in-situ given the headaches we've had | 09:52 |
| derekh | lifeless: sounds reasonable | 09:52 |
| lifeless | derekh: we also need to manually stable br-ctlplane to workaround the dhclient running on eth2 issue | 09:52 |
| lifeless | derekh: but lets get one machine copacetic and go from there | 09:52 |
| lifeless | ifarkas: https://review.openstack.org/#/c/81200/ - https://bugs.launchpad.net/tripleo/+bug/1300458 | 09:54 |
| uvirtbot | Launchpad bug 1300458 in tripleo "incorrect ram weighting on overclouds" [Critical,Triaged] | 09:54 |
| lifeless | lsmola_: ^ FYI too | 09:54 |
| lsmola_ | lifeless: what is the problem here? | 09:56 |
| *** athomas has quit IRC | 09:56 | |
| lifeless | lsmola_: the negative weight should only be applied to bare metal clouds | 09:56 |
| lifeless | lsmola_: otherwise it makes vm clouds stack all the VMs onto one host | 09:57 |
| lifeless | lsmola_: which is rather bad for performance! | 09:57 |
| ifarkas | lifeless, right, that makes sense | 09:58 |
| lsmola_ | lifeless: ah, we are using the same nova element for overcloud and overcloud, uh | 09:58 |
| lifeless | yes | 09:58 |
| lifeless | if you look down in the config | 09:58 |
| lifeless | there are bits that are trigged by baremetal | 09:58 |
| lsmola_ | lifeless: yes, we need to make condition there, sorry i didn't realized | 09:58 |
| lsmola_ | lifeless: also list of used filters will be different | 09:58 |
| lifeless | just need to either move the condition down, or make it have a similar guard | 09:58 |
| lifeless | lsmola_: exactly | 09:58 |
| openstackgerrit | A change was merged to openstack/tripleo-heat-templates: Drop dnsmasq_range from the undercloud source. https://review.openstack.org/83130 | 09:59 |
| *** athomas has joined #tripleo | 09:59 | |
| *** jistr has joined #tripleo | 09:59 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Drop dnsmasq_range from the seeds config.json. https://review.openstack.org/83128 | 09:59 |
| lifeless | bah, eni typo | 10:00 |
| lifeless | this should come up now | 10:00 |
| lifeless | separately | 10:01 |
| lifeless | not what you want to see | 10:01 |
| lifeless | Errors were encountered while processing: | 10:01 |
| lifeless | grub-efi-amd64 | 10:01 |
| lifeless | grub-efi-amd64-signed | 10:01 |
| lifeless | shim-signed | 10:01 |
| lifeless | Error in function: | 10:01 |
| lifeless | A fatal error occurred | 10:01 |
| lsmola_ | lifeless: so you meant putting it under [baremetal] ? | 10:01 |
| lifeless | lsmola_: well no, I mean in the {{# guarded section | 10:01 |
| lifeless | or a similarly guarded section | 10:02 |
| lifeless | it has to be in [DEFAULT] doesn't it? | 10:02 |
| lsmola_ | lifeless: yes | 10:02 |
| lsmola_ | lifeless: ok, should I fix that? | 10:02 |
| lifeless | please | 10:02 |
| lsmola_ | lifeless: so this should be fine? {{#nova.baremetal}} | 10:03 |
| ifarkas | lsmola_, lifeless, btw: https://review.openstack.org/#/c/84131/ | 10:03 |
| *** e0ne has joined #tripleo | 10:03 | |
| lifeless | ifarkas: yeah | 10:06 |
| lifeless | so if we configure it via heat, cool | 10:07 |
| lifeless | but lets unbreak things first | 10:07 |
| lsmola_ | lifeless: ok I will put there quick {{#nova.baremetal}} guard | 10:07 |
| ifarkas | lifeless, ack | 10:08 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Install libffi-dev in the glance element https://review.openstack.org/80337 | 10:08 |
| *** CaptTofu has joined #tripleo | 10:09 | |
| lifeless | blink | 10:09 |
| lifeless | -GRUB_CMDLINE_LINUX_DEFAULT="" | 10:09 |
| lifeless | +GRUB_CMDLINE_LINUX_DEFAULT="nomdmonddf nomdmonisw" | 10:09 |
| lifeless | EWTF | 10:09 |
| tchaypo | nom d mon nom | 10:13 |
| *** e0ne has quit IRC | 10:13 | |
| *** CaptTofu has quit IRC | 10:13 | |
| *** matsuhashi has quit IRC | 10:14 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Adding baremetal guard to RamWeigher https://review.openstack.org/84397 | 10:16 |
| *** akuznetsov has joined #tripleo | 10:16 | |
| lsmola_ | lifeless: https://review.openstack.org/#/c/84397/1 | 10:18 |
| lsmola_ | lifeless: quick fix, making it configurable via heat will solve rest of the issues | 10:18 |
| lsmola_ | ifarkas: ^ | 10:18 |
| lifeless | lsmola_: one minor tweak | 10:19 |
| lifeless | ok yay that hypervisor checked in eventually | 10:19 |
| jp_at_hp | lifeless: on https://review.openstack.org/#/c/83188 - I still hold the opinion that the domumentation around what the nodes file should be is incomplete... | 10:20 |
| lifeless | jp_at_hp: yes, I understand now | 10:20 |
| *** akuznetsov has quit IRC | 10:20 | |
| lsmola_ | lifeless: oh :-) | 10:20 |
| lifeless | jp_at_hp: I haven't had time to put an example in ni devtest_setup.sh, but I agree thats valuable to have. | 10:20 |
| jp_at_hp | I think it can be a separate patch,k to improve that whole section documentation wise... | 10:20 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Adding baremetal guard to RamWeigher https://review.openstack.org/84397 | 10:21 |
| ifarkas | lsmola_, one typo :-) | 10:22 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Adding baremetal guard to RamWeigher https://review.openstack.org/84397 | 10:23 |
| lsmola_ | ifarkas: lifeless fixed | 10:23 |
| *** yamahata has quit IRC | 10:24 | |
| *** e0ne has joined #tripleo | 10:24 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 10:26 |
| *** e0ne_ has joined #tripleo | 10:30 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 10:30 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 10:31 |
| *** e0ne has quit IRC | 10:33 | |
| *** hashar has quit IRC | 10:33 | |
| *** akuznetsov has joined #tripleo | 10:34 | |
| derekh | lifeless: did that go ok? want me to update the other nodes? | 10:39 |
| lifeless | derekh: I'm just writing a script to do that | 10:39 |
| derekh | lifeless: ok | 10:40 |
| *** nosnos has quit IRC | 10:40 | |
| *** markmc has joined #tripleo | 10:45 | |
| *** CaptTofu has joined #tripleo | 10:46 | |
| *** pblaho has joined #tripleo | 10:48 | |
| *** gcha has joined #tripleo | 10:49 | |
| lifeless | derekh: so see /root/recovert | 10:51 |
| lifeless | derekh: once they all power on and are network reachable, running force-o-c-c should get us all sorted modulo cleaning up the state on any hung instances | 10:52 |
| *** CaptTofu has quit IRC | 10:54 | |
| derekh | lifeless: lgtm, we don't need to do anything on the controller? | 10:56 |
| lifeless | derekh: nova list --all-tenants | awk '$6=="ERROR" { print $2 }' | xargs -n1 nova reset-state | 10:57 |
| lifeless | derekh: after its all back up | 10:57 |
| lifeless | but - I think thats sorted already | 10:57 |
| derekh | lifeless: ok | 10:58 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/diskimage-builder: Add utility to mount and unmount images https://review.openstack.org/76509 | 10:59 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/diskimage-builder: WIP - Add script to replay elements in an existing image https://review.openstack.org/80267 | 11:01 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/diskimage-builder: WIP - enable elements to be re-playable https://review.openstack.org/81494 | 11:01 |
| StevenK | Add utility to mount and unmount images -- didn't we have those already in dib? And that makes me sad, since it's like two lines to do so anyway | 11:02 |
| jp_at_hp | lifeless: on https://review.openstack.org/#/c/83770, I want to understand your comment on patch set 7 better. I thought that the ssh details were only needed for the virtual power manager,, is that wrong? | 11:02 |
| openstackgerrit | jan grant proposed a change to openstack/diskimage-builder: Hard-link multiple identical files. https://review.openstack.org/84384 | 11:03 |
| openstackgerrit | Jan Provaznik proposed a change to openstack/tripleo-heat-templates: Add mysql innodb buffer pool size https://review.openstack.org/84405 | 11:03 |
| lifeless | jp_at_hp: I may have tied myself up in knots | 11:05 |
| lifeless | jp_at_hp: let me see | 11:05 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Removing unused testadata https://review.openstack.org/84406 | 11:11 |
| lifeless | jp_at_hp: try that on for size | 11:14 |
| *** slagle has joined #tripleo | 11:14 | |
| xuhaiwei | when running devtest_seed.sh, I got an error, the log says:"Storing debug log for failure in /root/.pip/pip.log", but the /root/.pip/pip.log does not exist | 11:15 |
| lifeless | derekh: I count 10 hypervisors up | 11:16 |
| lifeless | derekh: and active slaves | 11:17 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Adding baremetal guard to RamWeigher https://review.openstack.org/84397 | 11:18 |
| derekh | lifeless: sweet | 11:18 |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/tuskar-ui: Make run_tests.sh work with errexit https://review.openstack.org/84409 | 11:19 |
| jang | lifeless: https://review.openstack.org/#/c/84384/ looks to be working now (hardlink aggregation). It about halved the size of /opt on my seed. Testing it across the board now. | 11:20 |
| jp_at_hp | thanks lifeless - that's kinda what I was expecting, but it's good to get it so clearly. | 11:20 |
| lifeless | jang: cool. Is it a net win perf wise ? | 11:21 |
| *** slagle_ has joined #tripleo | 11:21 | |
| *** slagle has quit IRC | 11:21 | |
| derekh | lifeless: I can't ssh to the te-broker floating IP, gonna see if its running on compute | 11:21 |
| *** e0ne_ has quit IRC | 11:22 | |
| lifeless | derekh: its not | 11:23 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 11:23 |
| derekh | lifeless: ya, its gone, not even defined in libvirt | 11:23 |
| jang | lifeless: measuring now; it'll take me a while to get a number of runs that amount to any kind of statistical difference. | 11:24 |
| lifeless | derekh: thats odd :) | 11:24 |
| lifeless | ` | 11:24 |
| jang | for a dev cycle, it might be close. But for deploying finished images, where you don't care about the CI time as much, it might be a winner. | 11:24 |
| lifeless | derekh: I hope i haven't just wedged it:) - I'm going to tag and run | 11:25 |
| lifeless | derekh: I have one suggestion - if you rebuild the broker you can use neutron create-port to create a port with 192.168.1.1 in advance and supply that on the boot line | 11:25 |
| lifeless | derekh: rather than a net-id | 11:25 |
| derekh | lifeless: ok, will do | 11:25 |
| lifeless | derekh: then we don't need to wait for infra | 11:25 |
| lifeless | --net nic-id=asdad IIRC | 11:26 |
| derekh | lifeless: ahh, its gone to ERROR state now, will rebuild it | 11:26 |
| lifeless | me -> sleep | 11:26 |
| Ng | night lifeless | 11:26 |
| jang | night | 11:27 |
| *** hashar has joined #tripleo | 11:28 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Adding overcloud keystone client https://review.openstack.org/84379 | 11:31 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Overcloud initialization https://review.openstack.org/83340 | 11:31 |
| *** lucasagomes is now known as lucas-hungry | 11:34 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Make run_tests.sh work with errexit https://review.openstack.org/84409 | 11:36 |
| *** kiall_ is now known as Kiall | 11:36 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Hide non working scaling https://review.openstack.org/84413 | 11:37 |
| derekh | Ng: Hmm, that fix doesn't seem to have worked, still got a shed load of dhcp-all-interfaces.sh instances running | 11:41 |
| *** weshay has joined #tripleo | 11:41 | |
| Ng | derekh: huh. where is this actually happening? | 11:42 |
| derekh | Ng: I'm looking on ci-overcloud-novacompute4-dqiq7436leuh | 11:42 |
| derekh | Ng: so this is the upstart config http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/dhcp-all-interfaces/install.d/dhcp-all-interfaces.conf | 11:44 |
| derekh | Ng: so that would start dhcp-all-interfaces.sh on all interfaces , each time a single interface comes up | 11:45 |
| Ng | ugh | 11:45 |
| derekh | e.g. | 11:46 |
| derekh | -rw-r----- 1 root root 276 Apr 1 11:31 dhcp-all-interfaces-qbr00dd03df-ca.log | 11:46 |
| derekh | and loads others like it | 11:46 |
| Ng | agreed | 11:46 |
| derekh | so we need it to ignore the instance interfaces somehow | 11:48 |
| *** xuhaiwei has quit IRC | 11:48 | |
| Ng | derekh: ideally at the upstart level, so we're not pointlessly spawning the script for each one of these interfaces | 11:50 |
| Ng | just poking at an instance interface to see if it has any easy hints we can detect | 11:50 |
| derekh | Ng: yup, ok | 11:50 |
| lifeless | derekh: lots of instances, or a growing backlog ? | 11:50 |
| lifeless | derekh: I'd expect lots of instances to have run, but to all exit very quickly. | 11:50 |
| * lifeless failed at sleep | 11:51 | |
| *** d0ugal has quit IRC | 11:51 | |
| lifeless | derekh: the /root/recovert scripts should help mass-apply whatever you and ng come up with | 11:51 |
| derekh | lifeless: the flock could doesn't seem to be changing holding steady at 502 for about 10 minutes so far | 11:51 |
| *** d0ugal has joined #tripleo | 11:51 | |
| derekh | lifeless: yup | 11:52 |
| derekh | lifeless: *flock count | 11:52 |
| lifeless | oh any flock count means we're still trying to enable now | 11:52 |
| Ng | I am a bit surprised that flockers aren't dying off as each one goes through the pointless enumeration exercise | 11:52 |
| lifeless | assuming the fix *did* get copied off | 11:52 |
| derekh | lifeless: we have the updated script | 11:53 |
| lifeless | ok, I must leave you to it | 11:53 |
| * lifeless tries again | 11:54 | |
| derekh | lifeless: night | 11:54 |
| *** morazi has joined #tripleo | 11:55 | |
| *** ramishra has quit IRC | 11:56 | |
| *** mrunge has quit IRC | 11:59 | |
| Ng | derekh: any objections if I murder the flock of flocks? I'd like to get things calmed down and test a modification to the upstart job | 11:59 |
| derekh | Ng: go for it | 11:59 |
| derekh | Ng: ok, so I think the change to dhcp-all-interfaces.sh is wrong | 12:02 |
| Ng | derekh: oh? | 12:02 |
| derekh | we're calling it with no args in the upstart script | 12:02 |
| *** e0ne has joined #tripleo | 12:02 | |
| derekh | so ARGS="/usr/local/sbin/dhcp-all-interfaces.sh" | 12:02 |
| Ng | derekh: yes, that is the modification to the upstart job I wanted to test, adding $INTERFACE | 12:03 |
| Ng | this whole thing is awful, it's still even called *all-interfaces* ;) | 12:03 |
| *** e0ne has quit IRC | 12:04 | |
| *** e0ne has joined #tripleo | 12:04 | |
| derekh | Ng: when its respawned in enable_interface for a specifc interface it gets recalled on all interfaces | 12:05 |
| derekh | as those were the original prams | 12:05 |
| Ng | derekh: agreed, it's basically never ever operating in single-interface mode ever | 12:05 |
| Ng | at all | 12:05 |
| derekh | Ng: what your trying should fix this case but dhcp-all-interfaces.sh should also be fixed for the case where it gets no param (or changed not to support that) | 12:06 |
| derekh | Ng: yup | 12:06 |
| derekh | Ng: will be back in a few minutes | 12:07 |
| Ng | k | 12:07 |
| Ng | hmm, now I'm not sure if I subtly broke the upstart job or not. I just brought up a random manual TAP interface to try and get upstart to fire off the script, but I see nothing | 12:07 |
| Ng | also not sure if upstart is so wedged that it can't do anything anymore | 12:08 |
| Ng | it doesn't seem to be reaping the dead flock | 12:08 |
| *** CaptTofu has joined #tripleo | 12:08 | |
| *** e0ne has quit IRC | 12:09 | |
| Ng | yeah I think upstart is toast, "initctl list" isn't even returning | 12:10 |
| Ng | going by the original purpose of this script, we really shouldn't let it keep getting triggered indefinitely, it's supposed to get us DHCP'd on all physical interfaces at boot, because we don't know which one is the one we want | 12:11 |
| Ng | adding more and more chicanery to keep us tip-toeing around virtual interfaces that appear later, and avoiding flooding init daemons to death, all seems entirely pointless, so I'm thinking about useful ways to detect that our work is done, and stop running anymore | 12:13 |
| *** CaptTofu has quit IRC | 12:16 | |
| Ng | derekh: so I'm thinking a two-prong approach here. first, remove the ability for the script to detect all interfaces and just require that an interface name be passed in. I don't think we're ever calling it other than from upstart/udev with (at least the intention ;) of passing in an interface name | 12:19 |
| *** e0ne has joined #tripleo | 12:19 | |
| Ng | derekh: maybe three-prong, because with prong one complete, we should probably rename this element and everything inside it ;) | 12:20 |
| Ng | derekh: but the final prong would be to try and figure out a smart way to detect that we don't need to be doing this anymore (i.e. the machine has booted and is talking to the world) | 12:20 |
| *** jprovazn has quit IRC | 12:21 | |
| Ng | I'm not sure the final prong is terribly reasonable, sadly, but we can at least attempt to do better at racing to exit | 12:21 |
| openstackgerrit | Andrea Rosa proposed a change to openstack/tripleo-image-elements: Allow settings for Nova Scheduler https://review.openstack.org/84131 | 12:29 |
| *** rlandy has joined #tripleo | 12:34 | |
| *** lucas-hungry is now known as lucasagomes | 12:36 | |
| *** CaptTofu has joined #tripleo | 12:37 | |
| *** ramishra has joined #tripleo | 12:37 | |
| *** jdob has joined #tripleo | 12:40 | |
| *** jprovazn has joined #tripleo | 12:45 | |
| *** geerdest has joined #tripleo | 12:46 | |
| *** lblanchard has joined #tripleo | 12:49 | |
| derekh | Ng: back, removing the ability to detect all interfaces seems reasonable, to get us over this hump we should be ok to just pass the interface name around correctly, and we can put the rest of the changes through gerrit? | 12:50 |
| derekh | looks like it already tests if a interface is real | 12:56 |
| derekh | Inspecting interface: qbr12932ae8-f5...Device has generated MAC, skipping. | 12:57 |
| *** dprince has joined #tripleo | 12:57 | |
| *** killer_prince2 has quit IRC | 12:58 | |
| SpamapS | generated MAC really means locally administered bit right? | 12:58 |
| SpamapS | Seems like we need to actually check that it is a NIC, and not a bridge/tap/etc. | 12:59 |
| derekh | SpamapS: yup, that what we want | 13:01 |
| *** sballe has joined #tripleo | 13:02 | |
| *** derekh changes topic to "FIREDRILL ci-overcloud down | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 13:03 | |
| *** ccrouch has quit IRC | 13:04 | |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-heat-templates: WIP: Switch overcloud to software-config https://review.openstack.org/81666 | 13:05 |
| *** pblaho has quit IRC | 13:08 | |
| SpamapS | hm | 13:11 |
| SpamapS | bridges at least don't have tun_flags in /sys/devices/*/net/$devname | 13:12 |
| *** ccrouch has joined #tripleo | 13:13 | |
| SpamapS | derekh: I wonder if the 'flags' files in /sys/devices/*/net/* have the information we need | 13:17 |
| *** noslzzp has joined #tripleo | 13:17 | |
| SpamapS | http://paste.openstack.org/show/74745 | 13:17 |
| SpamapS | definite patterns there | 13:17 |
| * derekh goes searching for docs | 13:20 | |
| SpamapS | derekh: I'm digging through the kernel source at the moment :-P | 13:21 |
| derekh | SpamapS: ok | 13:21 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Loading template paraemters from tuskar-api https://review.openstack.org/84427 | 13:22 |
| *** CaptTofu has quit IRC | 13:23 | |
| SpamapS | derekh: note that we're ~10 minutes from screaming hungry beasts needing their morning feeding.. so if you don't hear from me soon.. carry on ;) | 13:24 |
| derekh | SpamapS: ok np | 13:24 |
| *** akuznetsov has quit IRC | 13:26 | |
| SpamapS | derekh: ok so the flags are defined in /usr/include/linux/if.h | 13:27 |
| SpamapS | derekh: and it looks like IFF_EBRIDGE is a _private_ flag. :-/ | 13:28 |
| SpamapS | meaning not in sysfs | 13:28 |
| derekh | :-( | 13:28 |
| SpamapS | derekh: so we'll probably be reduced to filtering things out by digging in ovs | 13:28 |
| SpamapS | or whitelisting classes | 13:29 |
| SpamapS | like ethX, emX, vnetX | 13:29 |
| SpamapS | but that seems not awesome either | 13:29 |
| *** akuznetsov has joined #tripleo | 13:29 | |
| derekh | ya would be nice if we could avoid it | 13:31 |
| dprince | derekh: what is the problem you are solving here? | 13:31 |
| derekh | btw we're about to get CLOUDOUTRAGED, if people think its annoying I can turn it off, just giving it a go | 13:32 |
| derekh | dprince: https://etherpad.openstack.org/p/cloud-outage | 13:32 |
| *** CLOUDOUTAGE has joined #tripleo | 13:32 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 13:32 |
| *** CLOUDOUTAGE has quit IRC | 13:32 | |
| SpamapS | oh heh I get those in my email already | 13:32 |
| SpamapS | anyway, time to go feed the beasts | 13:32 |
| derekh | k | 13:33 |
| SpamapS | derekh: I did notice that /sys/devices/*/net/xxx/bridge might be useful in finding bridges to filter out | 13:33 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Loading template parameters from tuskar-api https://review.openstack.org/84427 | 13:33 |
| SpamapS | derekh: clint@clint-HP:~$ cat /sys/devices/virtual/net/virbr0/bridge/bridge_id | 13:33 |
| SpamapS | 8000.fe540009a2f1 | 13:33 |
| derekh | SpamapS: ok | 13:34 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Loading template parameters from tuskar-api https://review.openstack.org/84427 | 13:35 |
| derekh | Ng: are you still doing anything with the upstart script, if not I might just bouch compute4 with $INTERFACE added, its not exaclty what we want but I think might be good enough to get us passed this problem in overcloud | 13:35 |
| derekh | *bounce | 13:35 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Loading template parameters from tuskar-api https://review.openstack.org/84427 | 13:36 |
| *** julim has joined #tripleo | 13:38 | |
| *** mrunge has joined #tripleo | 13:38 | |
| dprince | SpamapS/derekh: The mac_addr_type bit should differentiate between real NICs vs. others right? (i.e. locally managed) | 13:39 |
| SpamapS | dprince: except that you just changed some things to have not locally managed macs.. ;) | 13:39 |
| SpamapS | but I guess those things are things we want dhcp'd.. | 13:40 |
| dprince | SpamapS: oh, well why didn't you say something. | 13:40 |
| SpamapS | dprince: either way, that's not _really_ the right clue. | 13:41 |
| SpamapS | also I wonder if even filtering bridges is right.. since once the bridge is up we do want the dhclient on the bridge not the iface, right? | 13:41 |
| derekh | I've changed the unstart script on novacompute4 to say | 13:42 |
| dprince | SpamapS: dhcp-all-interfaces should not touch bridges. It is for bootstrapping only | 13:42 |
| derekh | exec /usr/local/sbin/dhcp-all-interfaces.sh $INTERFACE | 13:42 |
| dprince | SpamapS: IMO it is a violation of its purpose... | 13:42 |
| derekh | going to reboot it to see if it works while we find a long term solution | 13:42 |
| *** sballe has quit IRC | 13:43 | |
| *** Rakesh5 has quit IRC | 13:44 | |
| SpamapS | dprince: that is a fair point. | 13:47 |
| SpamapS | dprince: given that, then yes, if /sys/devices/virtual/net/$interface/bridge/bridge_id exists, then we should not add it | 13:48 |
| SpamapS | and I suspect tap interfaces are fairly easy to detect | 13:48 |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Fix test caching https://review.openstack.org/84436 | 13:51 |
| dprince | SpamapS: Actually, I think excluding anything in just /sys/devices/virtual/net/ would do fine. | 13:52 |
| derekh | SpamapS: dprince: ya, I was about to say, the virt interfaces are in /sys/devices/virtual http://paste.openstack.org/show/74750/ | 13:54 |
| dprince | derekh/SpamapS: who is doing this? would you like me to test/throw up a patch? | 13:54 |
| derekh | dprince: go for it | 13:54 |
| dprince | derekh: okay. So it was just br-ctlplane that is causing us failures then? | 13:55 |
| Ng | derekh: sorry, got dragged away for lunch | 13:56 |
| derekh | dprince: no the origional problem was that dhcp-all-interfaces.sh was being recec'd with no params and also we were not passing an interface name into it from the upstart config, so they just kept looping over all interfaces | 13:58 |
| derekh | *reexec'ed | 13:58 |
| dprince | derekh: okay. | 13:58 |
| dprince | SpamapS: re, the "original problem", could Ubuntu use udev rule's to avoid that? e.i. instead of calling the script for 'all interfaces' could we just do it one interface at a time like we do on Fedora? | 13:59 |
| derekh | dprince: we were the ones calling it for all interfaces, not passing any param in http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/dhcp-all-interfaces/install.d/dhcp-all-interfaces.conf?id=f10e61457977a22f531154bac66a532d4684f346 | 14:01 |
| derekh | upstart was calling once for each interface | 14:02 |
| Ng | I'm refactoring the element to stop ever trying to do every interface, and to then have a more appropriate name | 14:02 |
| dprince | derekh: Well, there were several iterations. I was trying to re-use what was already there (for Debian) | 14:02 |
| *** CLOUDOUTAGE has joined #tripleo | 14:03 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 14:03 |
| *** CLOUDOUTAGE has quit IRC | 14:03 | |
| derekh | dprince: yup | 14:03 |
| dprince | derekh: They were calling it multiple times (for each interface added)... but each time it would inspect all interfaces. | 14:03 |
| dprince | derekh: and I'm now questioning that since I added support to do just a single interface at a time... | 14:03 |
| dprince | SpamapS: can we do this instead: 'exec /usr/local/sbin/dhcp-all-interfaces.sh $INTERFACE' | 14:04 |
| Ng | that's what derekh has done to the live system, aiui | 14:05 |
| Ng | and that certainly should work | 14:05 |
| derekh | Ng: dprince yup, testing it at the moment | 14:05 |
| dprince | derekh: gotcha. I buy that. Now that dhcp-all-interfaces.sh supports adding interfaces one-at-a-time I think it is cleaner that way | 14:06 |
| derekh | dprince: agreed | 14:06 |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Refactor dhcp-all-interfaces to restore sanity. https://review.openstack.org/84442 | 14:07 |
| Ng | This is not a pretty diff :( | 14:07 |
| *** yassine has joined #tripleo | 14:09 | |
| lxsli | Eyes please? https://review.openstack.org/#/c/83596/ | 14:10 |
| *** bauzas has quit IRC | 14:13 | |
| dprince | Ng: left you some comments. The main thing is I think we need to use /sys/devices/virtual/net and exclude devices there. | 14:17 |
| dprince | Ng: also several nits about various things too. I might suggest that you decouple the renaming from the functional changes too | 14:18 |
| dprince | Ng: from an element standpoint (top level) it will still effectively 'DHCP ALL INTERFACES' (the real ones) | 14:18 |
| dprince | Ng: noting that I already sort of wanted to change it too... but resisted because it seemed reasonable as is | 14:19 |
| derekh | ok, compute4 back up with the change to pass $INTERFACE into dhcp-all-interfaces, looks better (for defunct flock processes) but still not correct, nova having problems creating networks | 14:22 |
| dprince | derekh: is this an undercloud/overcloud issue? Or an issue inside the test environment itself? | 14:24 |
| derekh | dprince: an issue on ci-overcloud compute nodes | 14:24 |
| derekh | dprince: the node I'm looking at is ci-overcloud-novacompute4-dqiq7436leuh if you want to take a look | 14:25 |
| bnemec | jprovazn: https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest_variables.sh#L79 | 14:26 |
| derekh | we now have over 1000 dhcp-all-interfaces-*.log log files, one for each devices being created (real and virtual) | 14:26 |
| *** lazy_prince has joined #tripleo | 14:26 | |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Rename dhcp-all-interfaces and fix it on Ubuntu. https://review.openstack.org/84442 | 14:27 |
| *** lazy_prince is now known as killer_prince | 14:27 | |
| Ng | dprince: I agreed with your comments on the commit message, but I'm not convinced about the other stuff yet | 14:27 |
| Ng | derekh: ugh, that's still a pretty crazy amount of work going on for no real reason | 14:27 |
| derekh | Ng: yup | 14:28 |
| *** shardy is now known as shardy_afk | 14:28 | |
| jprovazn | bnemec, thanks, I use these vars too, for some reason I though that there was also a check "if fedora then set qpid" | 14:28 |
| Ng | derekh: did you have any genius ideas for ways we can detect that things are working and the udev/upstart jobs can be disabled? | 14:29 |
| bnemec | jprovazn: Yeah, below that there is an if fedora then use selinux-permissive check that would be similar though. | 14:29 |
| jprovazn | bnemec, you are right | 14:29 |
| dprince | Ng: if os-collect-config has executed dhcp-all-interfaces has done it's job and can go away IMO | 14:30 |
| Ng | interesting | 14:30 |
| dprince | Ng: its only real purpose is for bootstrapping, That said, the udev rules (with systemd) will only exec this script once per interface. | 14:31 |
| *** sballe has joined #tripleo | 14:31 | |
| Ng | that could work. we'd need to move the element to t-i-e | 14:31 |
| Ng | dprince: sure, but we are generating a *lot* of interfaces on some machines | 14:31 |
| Ng | [15:26:09] derekh: we now have over 1000 dhcp-all-interfaces-*.log log files | 14:31 |
| Ng | I don't understand why we are making a thousand network interfaces, but I'm sure there is some kind of horrible neutron-y explanation ;) | 14:31 |
| dprince | Ng: Can we put an extra conditional in the upstart script itself? I added this to the systemd script to safeguard things on Fedora: ConditionPathExists=!/etc/sysconfig/network-scripts/ifcfg-%I | 14:32 |
| derekh | I was tempted to see what happens if I add "rm -f /etc/init/dhcp-all-interfaces.conf" to the end of dhcp-all-interfaces.sh but that would be a short term get this up and running fix | 14:32 |
| dprince | Ng: i.e, don't run this init script if networking is already configured... | 14:32 |
| dprince | derekh/Ng: it sounds to me like upstart just needs to be tamed here... | 14:33 |
| derekh | we're now upto 1482 log files, I guess because zuul keeps retrying VM's that failed to start | 14:33 |
| Ng | it's not just upstart, you guys have one file per if which is pretty cheap to test | 14:33 |
| *** killer_prince has quit IRC | 14:34 | |
| *** CLOUDOUTAGE has joined #tripleo | 14:34 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 14:34 |
| *** CLOUDOUTAGE has quit IRC | 14:34 | |
| *** killer_prince has joined #tripleo | 14:35 | |
| jang | all, how much is 192.0.2.0/24 baked into the devtest config? If I want to pick another subnet across the board, is this still a mainly-manual process, or is it parameterised? | 14:35 |
| *** jprovazn is now known as jprovazn_afk | 14:35 | |
| dprince | jang: https://review.openstack.org/#/c/82327/ | 14:35 |
| derekh | So is it possible (only thinking out loud now) that the networking-interface upstart job is failing all virtual interfaces being brought up because dhcp-all-interfaces doesn't drop a config file | 14:35 |
| derekh | root@ci-overcloud-novacompute4-dqiq7436leuh:/var/log/upstart# ls -l *qbr8edfc22d-f6* | 14:35 |
| derekh | -rw-r----- 1 root root 76 Apr 1 14:27 dhcp-all-interfaces-qbr8edfc22d-f6.log | 14:35 |
| derekh | -rw-r----- 1 root root 49 Apr 1 14:28 network-interface-qbr8edfc22d-f6.log | 14:35 |
| derekh | root@ci-overcloud-novacompute4-dqiq7436leuh:/var/log/upstart# cat dhcp-all-interfaces-qbr8edfc22d-f6.log | 14:35 |
| derekh | Inspecting interface: qbr8edfc22d-f6...Device has generated MAC, skipping. | 14:35 |
| derekh | root@ci-overcloud-novacompute4-dqiq7436leuh:/var/log/upstart# cat network-interface-qbr8edfc22d-f6.log | 14:35 |
| derekh | ifdown: interface qbr8edfc22d-f6 not configured | 14:35 |
| jang | bonzer dprince ! | 14:36 |
| dprince | jang: the fact that we hard code it should go away (hopefully soon) | 14:36 |
| *** CaptTofu has joined #tripleo | 14:36 | |
| Ng | derekh: hmm. they shouldn't have a static config though, right? neutron ought to be configuring them dynamically | 14:38 |
| Ng | (I'm guessing) | 14:38 |
| Ng | ifup/ifdown rely on eni | 14:39 |
| *** bauzas has joined #tripleo | 14:39 | |
| dprince | Ng: I would very much like us to be using static configs (aka. eni): https://review.openstack.org/#/c/69918/ | 14:40 |
| dprince | Ng: to me it is the normal thing to do :) | 14:40 |
| derekh | Ng: yup they shouldn't, | 14:40 |
| *** CaptTofu has quit IRC | 14:40 | |
| Ng | derekh: with the $INTERFACE added to the upstart job, is the networking still broken? | 14:43 |
| Ng | if that works, let's at least land that and then have a wider (list?) discussion about how we refactor this stuff to be simple and robust | 14:44 |
| dprince | Ng: was going to push just that but bailed when I saw your branch... so are you going to push just this fix too? | 14:45 |
| Ng | dprince: if it works, I'm happy to push a change. | 14:45 |
| derekh | Ng: with $INTERFACE added we have gotten rid of our respawn problem but neutron/nova are having problems with the net devices its creating, I've just reboot it again to try and remove the upstart job after it first runs to see what happens | 14:45 |
| dprince | Ng: actually, I think it would be nice to exclude virtual devices too since that seems to be an immediate problem as well | 14:46 |
| derekh | Ng: once its back if we still have a problem we should jump on the box and see if we can figure out what else is wrong | 14:46 |
| Ng | derekh: if we remove it after the first run we might not generate a dhcp config for the physical device that needs it | 14:47 |
| *** rdopieralski has quit IRC | 14:47 | |
| *** spzala has joined #tripleo | 14:47 | |
| *** rpodolyaka has quit IRC | 14:48 | |
| derekh | Ng: good point..so that was a bad idea | 14:48 |
| Ng | we could avoid spitting out so many logs by not echoing so much, but that seems like a secondary concern | 14:48 |
| Ng | the problem with removing the job is knowing when it's safe to do. so far the only idea there that makes sense is a successful occ run, IMHO | 14:49 |
| Ng | and network-interface.conf will still run, since that's an upstream upstart thing, not one of ours, afair, so we will likely get lots of logs anyway | 14:50 |
| *** rpodolyaka has joined #tripleo | 14:51 | |
| openstackgerrit | Alexis Lee proposed a change to openstack/os-apply-config: Add DO_NOT_CREATE option https://review.openstack.org/82038 | 14:51 |
| jang | dprince: wrt that eni patch - I've just seen a rebooted overcloud control (props on the name change incidentally, long overdue) come up without shifting its ip address from eth0 to br-ex. Should I expect to see code in there to ensure that that'll work properly with the patch? | 14:54 |
| *** slagle_ is now known as slagle | 14:55 | |
| dprince | jang: that is what this branch does. Making reboots work was one of the primary goals | 14:56 |
| dprince | jang: the ability to reboot a node (IMO) is a fundamental case we don't yet support :( | 14:56 |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Fix dhcp-all-interfaces on Ubuntu/Debian. https://review.openstack.org/84466 | 14:58 |
| dprince | derekh: so you removed the dhcp-all-interfaces.conf script on ci-overcloud-novacompute4-dqiq7436leuh.novalocal ? | 14:59 |
| Ng | derekh: if we can get it working with just the $INTERFACE addition, 84466 is just that | 14:59 |
| derekh | dprince: yup, I did | 14:59 |
| derekh | Ng: that didn't work but I'm not sure if the problem was unrelated | 14:59 |
| Ng | derekh: ok | 14:59 |
| Ng | I need to disappear for 20-30 mins, will be happy to assist further when I return :) | 15:00 |
| derekh | Ng: maybe try it again on compute5 can see what the problem is | 15:00 |
| derekh | Ng: ok | 15:00 |
| *** untriaged-bot has joined #tripleo | 15:01 | |
| untriaged-bot | Untriaged bugs so far: | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1300693 | 15:01 |
| uvirtbot | Launchpad bug 1300693 in tripleo "devtest.sh run leaves 3 default security groups" [Undecided,New] | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 15:01 |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 15:01 |
| *** untriaged-bot has quit IRC | 15:01 | |
| *** matty_dubs|gone is now known as matty_dubs | 15:02 | |
| *** TravT has joined #tripleo | 15:04 | |
| *** CLOUDOUTAGE has joined #tripleo | 15:05 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 15:05 |
| *** CLOUDOUTAGE has quit IRC | 15:05 | |
| derekh | we also got lots of these tracbacks on the nova-comute log which may be a new reason nova isn't able to start an instance http://paste.openstack.org/show/74757/ | 15:06 |
| dprince | derekh: oh, that looks not so good | 15:08 |
| bnemec | I think lifeless was looking at that yesterday too. | 15:08 |
| bnemec | He posted a paste with the same traceback. | 15:08 |
| openstackgerrit | gerry-drudy proposed a change to openstack/tripleo-heat-templates: Add SwiftSortingMethod parameter https://review.openstack.org/84470 | 15:10 |
| derekh | bnemec: any idea if he mentioned what the problem might be? | 15:11 |
| dprince | derekh: seems to be related to the recent neutron event change https://review.openstack.org/#/c/81120/ | 15:12 |
| bnemec | derekh: Not that I can see. He just posted a link to http://paste.openstack.org/show/74700/ but skimming the scrollback I don't see any followup. | 15:13 |
| derekh | dprince: ok, so maybe we always had that tracback and its not effecting anything | 15:14 |
| derekh | dprince: checking | 15:14 |
| *** ramishra has quit IRC | 15:16 | |
| derekh | dprince: we were not getting that trace back in successful CI runs | 15:17 |
| *** jcoufal has quit IRC | 15:18 | |
| dprince | derekh: well, it could be a race where neutron wins sometimes right? | 15:19 |
| *** hashar_ has joined #tripleo | 15:19 | |
| derekh | dprince: ok | 15:20 |
| *** hashar has quit IRC | 15:22 | |
| *** CaptTofu has joined #tripleo | 15:22 | |
| *** hashar_ has quit IRC | 15:26 | |
| *** ifarkas has quit IRC | 15:26 | |
| derekh | Apr 1 15:37:25 ci-overcloud-novacompute4-dqiq7436leuh kernel: [ 2467.029161] kvm [16383]: vcpu0 unhandled wrmsr: 0x682 data 0 | 15:29 |
| derekh | lots of these aswell | 15:30 |
| derekh | googling, probably a non issue | 15:30 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Load undercloud images with -d (delete duplicate) https://review.openstack.org/84478 | 15:32 |
| *** hashar has joined #tripleo | 15:33 | |
| *** CLOUDOUTAGE has joined #tripleo | 15:36 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 15:36 |
| *** CLOUDOUTAGE has quit IRC | 15:36 | |
| *** yassine has quit IRC | 15:40 | |
| dprince | derekh: could we just add the $INTERFACE fix to the upstart script in the testenv worker images and then respin them? | 15:42 |
| * dprince likes a clean slate sometimes | 15:42 | |
| derekh | dprince: yes, we could but until we get the ci-overcloud up and running it wont help, I haven't even gotten as far as testing the testenv's to see if they have a problem | 15:43 |
| derekh | dprince: but I got no objection | 15:44 |
| * dprince has to run out... will be back online in a bit... | 15:46 | |
| derekh | also got lots of these | 15:46 |
| *** dprince has quit IRC | 15:46 | |
| derekh | 2014-04-01 15:45:58.938+0000: 8234: warning : virAuditSend:135 : Failed to send audit message virt=kvm op=stop reason=destroyed vm="instance-00003f52" uuid=610fc898-5cfb-4512-a4ad-3802dd962a75 vm-pid=-1: Operation not permitted | 15:46 |
| *** mrunge has quit IRC | 15:47 | |
| *** jcoufal has joined #tripleo | 15:48 | |
| *** UtahDave has joined #tripleo | 15:48 | |
| *** akrivoka has quit IRC | 15:49 | |
| *** UtahDave has left #tripleo | 15:50 | |
| *** gcha has quit IRC | 15:52 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 15:56 |
| *** matty_dubs is now known as matty_dubs|lunch | 15:57 | |
| *** akrivoka has joined #tripleo | 15:57 | |
| * Ng returns later than anticipated | 15:58 | |
| derekh | Ng: I meant to try just that $INTERFACE change on compute5 while your were gone but got distracted, will do it now | 16:00 |
| Ng | k | 16:00 |
| *** hashar has quit IRC | 16:00 | |
| *** hashar has joined #tripleo | 16:01 | |
| *** hashar has quit IRC | 16:02 | |
| derekh | Ng: done just changed line to "exec /usr/local/sbin/dhcp-all-interfaces.sh $INTERFACE" and rebooted | 16:03 |
| *** ccorrigan has quit IRC | 16:03 | |
| Ng | derekh: fingers crossed :) | 16:04 |
| *** eghobo has joined #tripleo | 16:04 | |
| *** sballe_ has joined #tripleo | 16:05 | |
| *** sballe has quit IRC | 16:05 | |
| *** BadCub01 has quit IRC | 16:05 | |
| *** cwolferh has joined #tripleo | 16:05 | |
| *** CLOUDOUTAGE has joined #tripleo | 16:07 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 16:07 |
| *** CLOUDOUTAGE has quit IRC | 16:07 | |
| *** e0ne has quit IRC | 16:09 | |
| Shrews | does it say something about my mental health that my mind insists on reading that as CLOUDOUTRAGE? | 16:12 |
| derekh | Shrews: It may reflect something about your mental health but its not just you, its been commented on before | 16:13 |
| Shrews | ah, *phew*. good to hear | 16:13 |
| *** morazi has quit IRC | 16:15 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 16:16 |
| *** jistr has quit IRC | 16:17 | |
| *** morazi has joined #tripleo | 16:18 | |
| derekh | Ng: compute5 back up wanna look at anything befor I run "os-collect-config --force --one -v"? | 16:18 |
| Ng | derekh: peeking now | 16:19 |
| Ng | derekh: ok, I think everything seems sane so far | 16:23 |
| derekh | Ng: ok, running o-c-c on compute5 | 16:24 |
| *** CaptTofu has quit IRC | 16:24 | |
| derekh | Ng: its finished, instances should start to get scheduled on in now | 16:25 |
| Ng | k | 16:25 |
| derekh | same log entry in nova-compute.log http://paste.openstack.org/show/74757/ | 16:29 |
| *** maru_afk is now known as marun | 16:32 | |
| SpamapS | Ng: regarding thousands of interfaces, you need a tap interface to attach to each VM | 16:34 |
| SpamapS | derekh: anything I can do to help with the outtage? | 16:34 |
| *** sballe_ has quit IRC | 16:34 | |
| Ng | SpamapS: yeah, I was just curious why we were spinning up quite so many in such a short time | 16:34 |
| derekh | Ng: I thinks its because, instance were failing to spawn (I'm guessing because of http://paste.openstack.org/show/74757/ ) then zull just keeps retrying them, so we got lots of tap interfaces | 16:35 |
| Ng | ah right | 16:36 |
| Ng | that would make some sense I guess :) | 16:36 |
| derekh | SpamapS: ya, please you will hopefully have better luck then I did, I been keeping some highlight of what we did here https://etherpad.openstack.org/p/cloud-outage | 16:36 |
| derekh | SpamapS: I've gotta run in a few minutes so if you have any questions about whats happened so far befor I go then fire them out | 16:37 |
| *** CLOUDOUTAGE has joined #tripleo | 16:38 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 16:38 |
| *** CLOUDOUTAGE has quit IRC | 16:38 | |
| derekh | I've been more focused on trying to get the ci-overcloud runing in the short term (by trying fixes in place), leaving long term solution to gerrit process | 16:39 |
| derekh | I'll leave the CLOUDOUTAGE bot running but if ye need to suppress it you can just comment out the "ircmessage:" on the etherpad | 16:40 |
| *** dprince has joined #tripleo | 16:41 | |
| *** athomas has quit IRC | 16:41 | |
| SpamapS | Ng: I think the right answer is to not print anything out in the network interface jobs | 16:42 |
| SpamapS | Ng: no printing == no log file | 16:42 |
| *** rpodolyaka1 has joined #tripleo | 16:42 | |
| SpamapS | Ng: another option is to turn off console logging. | 16:42 |
| Ng | SpamapS: agreed, we don't really need to be echoing from dhcp-all-interfaces | 16:42 |
| Ng | but network-interface.conf isn't ours, so we should think carefully before modifying it | 16:43 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Add unit test for cache-url https://review.openstack.org/84493 | 16:43 |
| SpamapS | Ng: also we could use 'logger' instead to send them to /var/log/syslog | 16:43 |
| SpamapS | Ng: network-interface will only be echoing things if 'ifup $INTERFACE' echoes things | 16:43 |
| SpamapS | is that what is happening? | 16:43 |
| Ng | SpamapS: yeah, all the squillions of tap devices have no eni entry, so ifup will whine | 16:44 |
| Ng | but none of this is what is breaking us atm | 16:44 |
| derekh | ok, gotta run | 16:44 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Add unit test for cache-url https://review.openstack.org/84493 | 16:44 |
| *** derekh has quit IRC | 16:44 | |
| *** rpodolyaka2 has joined #tripleo | 16:45 | |
| SpamapS | Ng: hmmmmmm | 16:45 |
| SpamapS | Ng: we only --allow auto | 16:45 |
| SpamapS | Ng: so ifup should be a noop for them | 16:45 |
| SpamapS | somebody.. please... get derekh an irc bouncer | 16:45 |
| Ng | SpamapS: oh, I'm wrong, it's not ifup, it's ifdown | 16:46 |
| Ng | so when the VM terminates, you get a single line log file with something like "ifdown: interface qbr716cf0b0-18 not configured" | 16:46 |
| *** rpodolyaka1 has quit IRC | 16:47 | |
| Ng | I'll file an upstream bug about that | 16:47 |
| Ng | but we need to figure out wtf is wrong with CI :) | 16:47 |
| *** rpodolyaka2 has quit IRC | 16:49 | |
| Ng | 2014-04-01 16:47:54.756 6949 TRACE nova.compute.manager [instance: fe53754d-5cd0-4982-8369-a909e0a2dd29] Unauthorized: Unknown auth strategy | 16:49 |
| Ng | wat | 16:49 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Add unit test for cache-url https://review.openstack.org/84493 | 16:49 |
| openstackgerrit | gerry-drudy proposed a change to openstack/tripleo-heat-templates: Add SwiftSortingMethod parameter https://review.openstack.org/84470 | 16:51 |
| *** mrunge has joined #tripleo | 16:56 | |
| *** BadCub01 has joined #tripleo | 16:57 | |
| *** matty_dubs|lunch is now known as matty_dubs | 16:57 | |
| SpamapS | Ng: yeah that's a bug for sure. | 16:59 |
| *** stevehuang has joined #tripleo | 16:59 | |
| SpamapS | Ng: check to make sure cfn is working (os-collect-config --print cfn) | 16:59 |
| Ng | SpamapS: looks ok, I see metadata with passwords and stuff | 17:00 |
| openstackgerrit | Andrea Rosa proposed a change to openstack/tripleo-incubator: Adding single quote to a grep command https://review.openstack.org/84495 | 17:01 |
| *** arosen has quit IRC | 17:02 | |
| *** mrunge has quit IRC | 17:02 | |
| Ng | SpamapS: I'm a bit stumped here, and I'm about to get called for dinner - can I hand off to you? | 17:03 |
| *** jcoufal has quit IRC | 17:04 | |
| SpamapS | Ng: yeah, which box? | 17:04 |
| Ng | (ci-overcloud-novacompute5 is where we've been poking around. afaics it's configured properly and things have been restarted, but it's just super unhappy) | 17:04 |
| SpamapS | wait what's even the outtage? I see nodepools running.. ?? | 17:05 |
| * SpamapS reads the etherpad | 17:05 | |
| *** CaptTofu has joined #tripleo | 17:05 | |
| Ng | SpamapS: hrm, they weren't working before. I did just do a desparate occ --force --one-time about 2 minutes ago | 17:06 |
| *** CLOUDOUTAGE has joined #tripleo | 17:09 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 17:09 |
| *** CLOUDOUTAGE has quit IRC | 17:09 | |
| SpamapS | Ng: ifdown | 17:10 |
| *** UtahDave has joined #tripleo | 17:10 | |
| SpamapS | doh | 17:10 |
| SpamapS | n/m | 17:10 |
| SpamapS | Ng: sorry, copy/paste/scrollback/stupid fail ;) | 17:11 |
| *** UtahDave has left #tripleo | 17:11 | |
| SpamapS | Ng: ah yeah I see most are in ERROR | 17:11 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Add support for "signals" to os-refresh-config https://review.openstack.org/83614 | 17:12 |
| *** slagle has quit IRC | 17:13 | |
| *** slagle has joined #tripleo | 17:15 | |
| SpamapS | heh so we have ACTIVE vms on 4 and 5 | 17:20 |
| *** morazi_ has joined #tripleo | 17:28 | |
| SpamapS | whoa.. | 17:29 |
| SpamapS | upstart seems overwhelmed on ci-overcloud-novacompute6 | 17:29 |
| SpamapS | with all the logging I think | 17:29 |
| SpamapS | root 1 91.1 0.0 37772 13384 ? Rs 11:02 359:42 /sbin/init | 17:30 |
| SpamapS | Ng: hah! fixed in trusty. ;) | 17:30 |
| *** morganfainberg_Z is now known as morganfainberg | 17:31 | |
| *** morazi has quit IRC | 17:32 | |
| *** CaptTofu has quit IRC | 17:32 | |
| *** CaptTofu has joined #tripleo | 17:32 | |
| *** morazi_ is now known as morazi | 17:33 | |
| *** CaptTofu has quit IRC | 17:37 | |
| *** CLOUDOUTAGE has joined #tripleo | 17:40 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 17:40 |
| *** CLOUDOUTAGE has quit IRC | 17:40 | |
| *** TravT has quit IRC | 17:40 | |
| *** e0ne has joined #tripleo | 17:40 | |
| *** TravT has joined #tripleo | 17:44 | |
| *** rpodolyaka1 has joined #tripleo | 17:46 | |
| *** lucasagomes is now known as lucas-afk | 17:49 | |
| *** rpodolyaka1 has quit IRC | 17:50 | |
| *** jprovazn_afk is now known as jprovazn | 17:52 | |
| *** rpodolyaka1 has joined #tripleo | 17:58 | |
| clarkb | hello #tripleo. Is the firedrill topic for ci-overcloud related to nodepool getting "ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" from tripleo cloud? | 18:02 |
| SpamapS | clarkb: our compute nodes are having issues | 18:03 |
| SpamapS | clarkb: but I wouldn't expect 500 from the controller | 18:03 |
| *** julim has quit IRC | 18:04 | |
| *** jang1 has joined #tripleo | 18:05 | |
| *** julim has joined #tripleo | 18:06 | |
| *** derekh has joined #tripleo | 18:07 | |
| *** CLOUDOUTAGE has joined #tripleo | 18:11 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 18:11 |
| *** CLOUDOUTAGE has quit IRC | 18:11 | |
| SpamapS | and now we seem to have lost NovaCompute6 | 18:11 |
| derekh | 2014-04-01T17:29:49 <SpamapS> upstart seems overwhelmed on ci-overcloud-novacompute6 | 18:11 |
| SpamapS | derekh: yeah, so I unblocked upstart by raising max open files for pid 1 | 18:12 |
| SpamapS | derekh: but then networking went dead | 18:12 |
| derekh | SpamapS: this was the origional error, init hit its fd limit | 18:12 |
| SpamapS | derekh: I'm poking in via console now. | 18:12 |
| SpamapS | derekh: ah ok.. so that was already known | 18:12 |
| derekh | SpamapS: ps -ef | grep -i flock | wc | 18:12 |
| SpamapS | derekh: that's a definite bug in upstart | 18:12 |
| SpamapS | it shouldn't be responding to EMFILE with trying forever.. it should be just logging on its main log "can't log for XXXX" | 18:12 |
| derekh | out dhcp-all-interfaces script was respawning itself as we we're passing in a interface | 18:13 |
| derekh | *our | 18:13 |
| derekh | SpamapS: https://bugs.launchpad.net/tripleo/+bug/1300663 | 18:13 |
| uvirtbot | Launchpad bug 1300663 in tripleo "upstart using 100% CPU" [Critical,Triaged] | 18:13 |
| SpamapS | derekh: ok | 18:14 |
| SpamapS | derekh: I set 'console none' on network-interface on NovaCompute6 | 18:15 |
| derekh | *not passing in a interface | 18:15 |
| SpamapS | derekh: the logging is fixed in trusty btw (14.04) | 18:15 |
| derekh | SpamapS: ok | 18:15 |
| SpamapS | doh | 18:15 |
| SpamapS | kernel paniced | 18:15 |
| SpamapS | something killed init | 18:15 |
| * SpamapS reboots | 18:15 | |
| derekh | so with the updated dhcp-all-interfaces on both compute4 and 5 zuul is now starting instances, but there quickly go into DELETE state I'm guess because of this error http://paste.openstack.org/show/74757/ | 18:17 |
| derekh | SpamapS: ^ thats were I was when I left it | 18:17 |
| derekh | sorry ERROR state then DELETE | 18:18 |
| SpamapS | derekh: we have some ACTIVE's | 18:18 |
| SpamapS | but just a few | 18:18 |
| derekh | SpamapS: but they don't stay active for line, their then deleted once they go to ERROR | 18:20 |
| derekh | hmm one seems to be staying running | 18:20 |
| *** panda has quit IRC | 18:21 | |
| *** panda has joined #tripleo | 18:21 | |
| SpamapS | ahh | 18:22 |
| *** CaptTofu has joined #tripleo | 18:23 | |
| * derekh wonders if we have the old ci-overcloud images anywhere to give us some breathing room | 18:23 | |
| *** jang1 has quit IRC | 18:25 | |
| derekh | forget that idea, their gone | 18:25 |
| SpamapS | well.. | 18:26 |
| SpamapS | also.. | 18:26 |
| SpamapS | we should like, make new stuff work :) | 18:26 |
| *** rpodolyaka1 has quit IRC | 18:28 | |
| SpamapS | ok I see instances booting on NovaCompute6 | 18:29 |
| dprince | if we build a new image with the $INTERFACE fix for dhcp-all-interfaces, For the other error I wonder if making the vif_plugging_timeout larger would help: https://review.openstack.org/#/c/81224/2/elements/nova/os-apply-config/etc/nova/nova.conf | 18:29 |
| derekh | SpamapS: yes, but ideally not on our running cloud, having a rollback strategy wouldn't do any harm | 18:29 |
| SpamapS | so I think we should disable logging of network-interface anyway.. | 18:30 |
| derekh | dprince: no, that timeout was made small because we're expecting it to timeout | 18:30 |
| SpamapS | that is just never a good idea on a nova compute host | 18:30 |
| derekh | dprince: because neutron doesn't have the creds to send the notify to nova | 18:31 |
| dprince | derekh: ack | 18:31 |
| derekh | SpamapS: dprince going to restart neutron-openvswitch-agent on the controller, it hasn't loged anythin since yesterday and its usually so chatty | 18:32 |
| derekh | SpamapS: dprince its last log entry was | 18:32 |
| derekh | 2014-03-31 06:42:59.489 25453 CRITICAL neutron [-] Trying to re-send() an already-triggered event. | 18:32 |
| SpamapS | interesting | 18:33 |
| dprince | derekh: sounds promising | 18:33 |
| derekh | 2014-04-01 18:35:39.920 2479 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: 2014-04-01T18:35:39Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated) | 18:33 |
| * dprince thinks good monitoring would help w/ all this | 18:33 | |
| SpamapS | dprince: and HA ;) | 18:34 |
| SpamapS | and CI gating of image replacement | 18:34 |
| SpamapS | all things we hold dear | 18:34 |
| SpamapS | that we're not doing. :-P | 18:34 |
| dprince | SpamapS: I actually think I would take monitoring first, but sure. All these things | 18:34 |
| * dprince likes to see | 18:34 | |
| SpamapS | we have crap monitoring already | 18:34 |
| SpamapS | If we had HA, we wouldn't have been compelled to replace the whole ci overcloud. | 18:35 |
| SpamapS | if we had CI gating of images, when we were ready to update it, we'd have found this problem | 18:35 |
| dprince | SpamapS: yeah, well what if your HA eats its face off | 18:36 |
| SpamapS | dprince: ???? | 18:36 |
| SpamapS | I don't really understand.. but o-k. :) | 18:36 |
| dprince | i.e. it dies because someone deployed wrong or something | 18:36 |
| SpamapS | so there really is no HA w/o monitoring | 18:37 |
| SpamapS | so yes you'd have to have monitoring before HA.. but my point is, we had monitoring, just crappy monitoring, which alerted us to "cloud not working" | 18:37 |
| SpamapS | we responded with our only possible fix for a hardware issue: redeploy cloud | 18:37 |
| SpamapS | had we been using actual CD, we'd have deployed images that were already known to work | 18:38 |
| SpamapS | thats what I'm frustrated by: that we don't have CD.. we have hope and prayer mostly | 18:38 |
| dprince | I think the problem in this case may have been it took a bit to notice the errors. | 18:38 |
| SpamapS | no | 18:38 |
| SpamapS | the errors were the death of the machine | 18:39 |
| *** e0ne has quit IRC | 18:39 | |
| SpamapS | knowing it was dead 8 hours earlier would have had us in this situation 8 hours earlier. | 18:39 |
| dprince | So who actually deployed the new images then? And when? | 18:39 |
| SpamapS | lifeless IIRC.. and its what I would do too btw. | 18:39 |
| SpamapS | with no backups.. not much else to do. | 18:39 |
| dprince | not trying to blame. just understand the timing here | 18:40 |
| *** e0ne has joined #tripleo | 18:40 | |
| *** e0ne has quit IRC | 18:40 | |
| SpamapS | derekh: so is neutron failing on controller now? | 18:40 |
| *** rpodolyaka1 has joined #tripleo | 18:40 | |
| *** e0ne has joined #tripleo | 18:40 | |
| SpamapS | 2014-04-01 18:43:14.555 10089 TRACE neutron.notifiers.nova BadRequest: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) | 18:41 |
| SpamapS | thats what neutron-server says | 18:41 |
| derekh | SpamapS: log file isn't logging anythin but process is running, but I didn't notice something | 18:41 |
| derekh | 2014-04-01 18:42:25.093 7737 TRACE nova.api.openstack NeutronClientException: 409-{u'NeutronError': {u'message': u"Quota exceeded for resources: ['floatingip']", u'type': u'OverQuota', u'detail': u''}} | 18:41 |
| SpamapS | oh! | 18:41 |
| SpamapS | well that would explain why they're being deleted | 18:41 |
| *** CLOUDOUTAGE has joined #tripleo | 18:42 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 18:42 |
| *** CLOUDOUTAGE has quit IRC | 18:42 | |
| Ng | doh! | 18:42 |
| derekh | | floating_ips | 100 | | 18:42 |
| derekh | quota seems high enough | 18:42 |
| SpamapS | derekh: quotas are likeing being misreported | 18:43 |
| SpamapS | floatingip-list shows 64 assigned to that tenant | 18:43 |
| derekh | hmm "neutron floatingip-list" shows a list of 63 floating IP's only 2 seem to be in use | 18:44 |
| SpamapS | you are charged if you own them | 18:44 |
| derekh | gonna floatingip-delete one of them | 18:45 |
| SpamapS | +----------+----------------------------------+ | 18:45 |
| SpamapS | | count(*) | tenant_id | | 18:45 |
| SpamapS | +----------+----------------------------------+ | 18:45 |
| SpamapS | | 1 | 94df8fe8a9b44a6498ac0c4a05238e7c | | 18:45 |
| SpamapS | | 60 | e01e473a9250498883955b80966a1e58 | | 18:45 |
| SpamapS | +----------+----------------------------------+ | 18:45 |
| SpamapS | I see quotas of 60 for that tenant id | 18:46 |
| SpamapS | which should still be enough | 18:46 |
| derekh | SpamapS: ok, we have progress | 18:47 |
| derekh | after deleting a floating IP | 18:47 |
| derekh | one of the instance got a floatingip (first time today) | 18:48 |
| derekh | and I can ping it | 18:48 |
| derekh | 64 bytes from 138.35.77.49: icmp_seq=1 ttl=44 time=264 ms | 18:48 |
| derekh | SpamapS: ^ | 18:48 |
| SpamapS | derekh: sounds good | 18:48 |
| SpamapS | derekh: so maybe nodepool doesn't know about the ips it already has | 18:48 |
| derekh | SpamapS: and zuul just popped a job off its pool http://status.openstack.org/zuul/ | 18:48 |
| *** newell_ has joined #tripleo | 18:49 | |
| derekh | SpamapS: yup, in all the downage it must have lost track | 18:49 |
| derekh | SpamapS: gonna delete the rest | 18:49 |
| SpamapS | tripleo ci-overcloud: setting the standard for downtime since 2014 | 18:49 |
| SpamapS | derekh: so I didn't do any other "fixing" on NovaCompute6 .. just disabled console logging for network-interface ... | 18:52 |
| derekh | SpamapS: ok, so now we have 2 things todo (hopefully) | 18:53 |
| derekh | 1. add $INTERFACE into the upstart config for dhcp-all-interfaces on all compute nodes | 18:53 |
| SpamapS | ok that seems like a straight forward bug fix | 18:54 |
| SpamapS | not sure how we missed that | 18:54 |
| derekh | 2. zull is now running jobs but they are all going to fail because our gear broker is dead, we need a new broker and it must have the IP tripleo-bm-test=192.168.1.1 | 18:54 |
| SpamapS | dprince: btw, an upstart job that is 'start on starting network-interface' is basically identical to a udev rule | 18:55 |
| SpamapS | am not at all averse to just using a udev rule only. | 18:55 |
| tchaypo | Morninge | 18:56 |
| derekh | I'll be sticking around for the meeting but gotta go then so wont be able to apply those fixes, | 18:56 |
| tchaypo | Or rather, late night! (IMHO morning begins at sunrise) | 18:56 |
| SpamapS | ugh meetings | 18:57 |
| derekh | brb | 18:57 |
| *** jcoufal has joined #tripleo | 18:59 | |
| *** akrivoka has quit IRC | 19:00 | |
| *** akrivoka has joined #tripleo | 19:01 | |
| *** jistr has joined #tripleo | 19:01 | |
| *** jcoufal has quit IRC | 19:03 | |
| *** jcoufal has joined #tripleo | 19:03 | |
| marios | are we meeting tonight? | 19:04 |
| rpodolyaka1 | we are | 19:04 |
| *** tzumainn has joined #tripleo | 19:05 | |
| marios | rpodolyaka1: k tx | 19:05 |
| rpodolyaka1 | np | 19:05 |
| *** e0ne has quit IRC | 19:06 | |
| *** e0ne has joined #tripleo | 19:07 | |
| *** rlandy has quit IRC | 19:07 | |
| *** rlandy has joined #tripleo | 19:07 | |
| *** blamar has joined #tripleo | 19:08 | |
| *** eguz has joined #tripleo | 19:11 | |
| marios | rpodolyaka1: should be now right? | 19:11 |
| * marios time change confused | 19:12 | |
| rpodolyaka1 | marios: it should. SpamapS has been waiting for lifeless to respond, but he's probably offline | 19:12 |
| marios | rpodolyaka1: k thanks bud | 19:12 |
| *** CLOUDOUTAGE has joined #tripleo | 19:13 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 19:13 |
| *** CLOUDOUTAGE has quit IRC | 19:13 | |
| rpodolyaka1 | marios: np :) don't go anywhere, we are about to start :) | 19:13 |
| *** eghobo has quit IRC | 19:15 | |
| lifeless | derekh: I'll get C's breakfast - can we do a graceful handover after that? I see that most of the hypervisors are still down? | 19:24 |
| *** akuznetsov has quit IRC | 19:28 | |
| *** jp_at_hp has quit IRC | 19:33 | |
| *** jp_at_hp has joined #tripleo | 19:40 | |
| *** CLOUDOUTAGE has joined #tripleo | 19:44 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 19:44 |
| *** CLOUDOUTAGE has quit IRC | 19:44 | |
| *** TravT has quit IRC | 19:45 | |
| *** hashar has joined #tripleo | 19:46 | |
| * tchaypo notices the marconi/macaroni thread | 19:46 | |
| *** newell_ has quit IRC | 19:46 | |
| tchaypo | it's so cute when the rest of the world finally joins the party a day late | 19:47 |
| *** newell_ has joined #tripleo | 19:47 | |
| *** e0ne_ has joined #tripleo | 19:51 | |
| *** e0ne has quit IRC | 19:51 | |
| *** e0ne_ has quit IRC | 19:53 | |
| *** TravT has joined #tripleo | 19:54 | |
| *** e0ne has joined #tripleo | 19:54 | |
| *** sdake_ is now known as sdake__ | 19:54 | |
| *** yamahata has joined #tripleo | 19:55 | |
| *** CaptTofu has quit IRC | 19:55 | |
| *** CaptTofu has joined #tripleo | 19:56 | |
| *** lsmola has joined #tripleo | 20:00 | |
| slagle | lsmola: so, is there going to be any working version of tuskar that uses just icehouse of all the dependent projects? | 20:00 |
| *** CaptTofu has quit IRC | 20:00 | |
| jistr | lsmola: what features of heat did you mean? | 20:00 |
| slagle | lsmola: that would be a reason to create a stable icehouse branch for tuskar | 20:00 |
| *** akuznetsov has joined #tripleo | 20:01 | |
| *** notmyname has joined #tripleo | 20:01 | |
| notmyname | derekh: this is the right channel in which to talk? | 20:02 |
| notmyname | derekh: to you or lifeless? | 20:02 |
| jistr | yeah i think what slagle says is correct. We're far from perfect with Tuskar, but it does deploy something :) i think it might be good to have a stable icehouse branch | 20:02 |
| jprovazn | SpamapS, "<SpamapS> next steps for HA is to have Heat inform nodes when they're about to be rebooted or deleted." - I don't understand why this is needed - for some graceful shutdown? | 20:02 |
| lifeless | notmyname: hi | 20:02 |
| jcoufal | lsmola: I think we need a stable release of icehouse | 20:02 |
| greghaynes | SpamapS: Not sure if you saw my message yesterday - was having an issue with the software-config patch in that the db-password cfn property was inside of a deployments array, causing os-collect-config --key db-password to fail | 20:02 |
| lsmola | slagle, yes for icehouse we have alsmost prepared a stable version of tuskar and tuskar-ui | 20:02 |
| derekh | lifeless: I gotta run, are you happy enough you know what needs to happen | 20:02 |
| lifeless | notmyname: sure, or #openstack-dev | 20:02 |
| jcoufal | slagle: there should be | 20:02 |
| lifeless | derekh: I believe I do - mass deploy the add of $INTERFACE and fixup the broker | 20:02 |
| derekh | lifeless: yup, | 20:02 |
| lifeless | derekh: is there a review up for that fix? Or should I add one? | 20:02 |
| SpamapS | greghaynes: you have an old os-collect-config then. It will be exploded into a json per deployment | 20:03 |
| jcoufal | slagle: still fixing few stuff, but we will need one | 20:03 |
| slagle | lsmola: jcoufal jistr : ok, then i think it makes sense to create icehouse branches for tuskar, that will continue to work with icehouse branches from other projects | 20:03 |
| greghaynes | aha | 20:03 |
| jistr | slagle: +1 | 20:03 |
| *** rbrady has quit IRC | 20:03 | |
| jcoufal | slagle: +1 | 20:03 |
| derekh | lifeless: that was a short term fix Ng SpamapS and dprince were talking about what the long term fix should look link, | 20:03 |
| jcoufal | tha make sense | 20:03 |
| lsmola | slagle, ok, as it should be packed as tech preview, we will need that | 20:03 |
| derekh | Ng: may have that exact fix up | 20:03 |
| jcoufal | lsmola: that's not accurate | 20:04 |
| lifeless | derekh: I think thats the one we can and should commit through | 20:04 |
| lifeless | *though* | 20:04 |
| derekh | lifeless: ok | 20:04 |
| slagle | lsmola: yea, i believe expectations have been set :) no one is expecting perfection | 20:04 |
| tchaypo | matty_dubs: your doge link was acceptable | 20:04 |
| derekh | notmyname: yes, I'll let lifeless fill you in, I gotta run, will follow up tomorrow if anything is needed | 20:04 |
| matty_dubs | tchaypo: Yay! :) | 20:05 |
| notmyname | lifeless: derekh just told me in the meeting that swift broke things for you. I hadn't seen your email until I search the list history | 20:05 |
| lsmola | slagle, well we have working creating and deleting of stack, that should be enough :-) | 20:05 |
| derekh | ok bye all | 20:05 |
| lifeless | notmyname: ah - did you find it? | 20:05 |
| *** derekh has quit IRC | 20:05 | |
| lifeless | notmyname: I don't think (and didn't intend to suggest) that swift did anything wrong | 20:05 |
| notmyname | lifeless: I did, but there isnt' a lot of detail there | 20:05 |
| notmyname | lifeless: subject "[openstack-dev] [TripleO][CI] all overcloud jobs failing" | 20:06 |
| notmyname | lifeless: I'm wondering if it was this commit https://github.com/openstack/swift/commit/c6cebb6e621a245c9c2d5bff0df59689b0140373 | 20:07 |
| lifeless | notmyname: yes that was it | 20:07 |
| notmyname | lifeless: swift patch or email? | 20:07 |
| lifeless | notmyname: it changes the mode from world readable to user readable only | 20:07 |
| lifeless | notmyname: both :) | 20:07 |
| *** CaptTofu has joined #tripleo | 20:08 | |
| lsmola | slagle, we can chat about more details tomorrow, I need to go now :-) | 20:09 |
| lsmola | good night everybody | 20:09 |
| *** rpodolyaka1 has quit IRC | 20:09 | |
| slagle | lsmola: night | 20:10 |
| jprovazn | SpamapS, to follow next HA steps you mentioned: when a resource plugin which tells server it's being rebooted/deleted is done, what are next steps? you can do then a graceful shutdown - migrate services somewhere else. But this all sounds like a different way than having all services in HA mode - then you don't have to care about graceful shutdown. Or am I missing something? | 20:12 |
| *** akuznetsov has quit IRC | 20:13 | |
| greghaynes | :) I had basically the same question almost typed out in my irc buffer | 20:13 |
| SpamapS | hah | 20:13 |
| jprovazn | :) great | 20:13 |
| SpamapS | so the reason this is needed is for the update case | 20:14 |
| greghaynes | Theres also the bit about deferring to the services on state of the cluster / keeping quorum during upgrade | 20:14 |
| SpamapS | it is as much about making Heat wait for a signal back that the node is o-k to take down as it is about quiescing. | 20:14 |
| *** CaptTofu has quit IRC | 20:14 | |
| openstackgerrit | lifeless proposed a change to openstack/diskimage-builder: Fix dhcp-all-interfaces upstart job https://review.openstack.org/84539 | 20:14 |
| lifeless | SpamapS: slagle: ^ | 20:14 |
| SpamapS | for nova-compute, this is evacuate.. | 20:14 |
| *** CLOUDOUTAGE has joined #tripleo | 20:14 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 20:14 |
| *** CLOUDOUTAGE has quit IRC | 20:14 | |
| *** CaptTofu has joined #tripleo | 20:15 | |
| SpamapS | for rabbit and galera, we want to delay any downtime if it would break quorum. | 20:15 |
| tchaypo | that's marvellously spammy - and in weechat, it's extremely colorful too | 20:15 |
| * tchaypo likes | 20:15 | |
| *** e0ne has quit IRC | 20:16 | |
| greghaynes | I really love that the username for CLOUDOUTAGE ends up being cloudouta | 20:17 |
| *** e0ne has joined #tripleo | 20:17 | |
| notmyname | lifeless: so I take it from the tripelo patch that you're runnign swift as root? | 20:18 |
| jprovazn | SpamapS, I see, though for controller nodes it seems to me that you need all services in HA mode at first to have them in cluster, then you can take care of no breaking quorum | 20:18 |
| SpamapS | notmyname: we're running the ring creation tools as root | 20:18 |
| notmyname | SpamapS: ok. why? (not attacking, just curious) | 20:18 |
| lifeless | notmyname: we run the rest of swift as swift | 20:18 |
| greghaynes | SpamapS: So the resource you were describing would be what is responsible for asking service if its ok to remove that node from the cluster, and then reporting back to heat wheather or not to progress? | 20:19 |
| SpamapS | notmyname: https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/swift/os-refresh-config/configure.d/73-swift runs as root | 20:19 |
| lifeless | notmyname: but config changes need root access if only to be able to sudo to the right users :) | 20:19 |
| jprovazn | SpamapS, IOW adding HA support for heat-engine, neutron would be prerequisities to run >1 controller node | 20:19 |
| *** CaptTofu has quit IRC | 20:19 | |
| SpamapS | jprovazn: yes, but all things in HA mode are already in the review pipeline, right? | 20:19 |
| SpamapS | jprovazn: heat-engine 'just works' | 20:19 |
| greghaynes | https://etherpad.openstack.org/p/tripleo-icehouse-ha-production-configuration is relevant etherpad | 20:19 |
| SpamapS | jprovazn: neutron-l3-agent is the only one that won't really be HA IIRC | 20:19 |
| SpamapS | greghaynes: correct | 20:20 |
| jprovazn | SpamapS, heat-engine is merged - lifless was not sure about it's stability (when we were talking about it in sunnyvalle), but I would give it a shot | 20:20 |
| SpamapS | heat-engine works multi-node | 20:21 |
| lifeless | (heat-engine) as long as you don't reboot it | 20:21 |
| SpamapS | lifeless: that's a MASSIVE architectural problem in Heat. | 20:21 |
| SpamapS | we're not solving that. | 20:21 |
| SpamapS | We need to, but not in our HA work. | 20:21 |
| jprovazn | SpamapS, then another thing is adding backend for cinder (gluster or ceph) | 20:21 |
| lifeless | SpamapS: Oh, I knwo :) | 20:22 |
| SpamapS | jprovazn: yes, not in the next step category though. | 20:22 |
| lifeless | well soon | 20:22 |
| SpamapS | In the important steps, yes. | 20:22 |
| lifeless | lets get HA control plane bedded down | 20:22 |
| lifeless | then have a discussion about what the next step should be | 20:22 |
| SpamapS | But cinder will keep working as-is right now for volumes that are not on downed nodes, will it not? | 20:22 |
| lifeless | rather than saying yes or no right now | 20:22 |
| SpamapS | right cinder control plane will be fine is what I'm trying ot say | 20:23 |
| SpamapS | to | 20:23 |
| jprovazn | SpamapS, ok, I see what you mean then, I just don't considered most of services done :) | 20:23 |
| jprovazn | (in queue) | 20:23 |
| notmyname | lifeless: just to be explicit, you aren't waiting on swift for anything, right? | 20:23 |
| lifeless | notmyname: nope - I would have screamed loudly if we thought that :) | 20:23 |
| notmyname | lifeless: ok, thanks | 20:24 |
| *** notmyname has left #tripleo | 20:24 | |
| lifeless | notmyname: what you can do, if you're interested, is run 'check experimental' on your patches | 20:24 |
| greghaynes | Im curious how the control flow for the new heat resource will go, does heat pick a node, resource asks node if its ok, repeat until its ok or fail? Or is there plans for asking down to the app level on which node would be best to update at a given point? | 20:24 |
| SpamapS | greghaynes: in our current templates, Heat is going to pick all the nodes, and try to rebuild them all at once. I am a little worried about that actually. :-P | 20:25 |
| greghaynes | oh joy! | 20:25 |
| greghaynes | WCPGW | 20:25 |
| *** julim has quit IRC | 20:25 | |
| greghaynes | I think the let heat pick a node at random and beg it to become ok to upgrade in the near future is suboptimal but might be a good mvp... | 20:27 |
| *** eguz has quit IRC | 20:27 | |
| *** eghobo has joined #tripleo | 20:28 | |
| greghaynes | but this is my super naive at heat idea :) | 20:28 |
| SpamapS | greghaynes: there is a rolling upgrade story already in Heat.. | 20:30 |
| greghaynes | ah | 20:31 |
| SpamapS | but we have issues using heat's scaling groups | 20:31 |
| SpamapS | I think we'll get there with Heat. | 20:31 |
| SpamapS | but .. so much to do.. :-P | 20:31 |
| ccrouch | (3:23:11 PM) jprovazn: SpamapS, ok, I see what you mean then, I just don't considered most of services done :) | 20:31 |
| ccrouch | so which ones do you think are still pending? | 20:32 |
| SpamapS | I am not keeping track of that :-/ | 20:32 |
| SpamapS | oy.. nor am I tracking food intake.. such hungry.. very food .. so eat.. | 20:32 |
| jprovazn | SpamapS, ccrouch: well, some "smaller" tasks from here: https://etherpad.openstack.org/p/tripleo-icehouse-ha-production-configuration line 66 | 20:33 |
| lifeless | SpamapS: go eat something then | 20:33 |
| greghaynes | much lunch | 20:33 |
| lifeless | SpamapS: you're no use to anyone with low brain energy | 20:34 |
| lifeless | wow signalling heat waitcondition completion is slow sometimes - 20seconds | 20:34 |
| greghaynes | jprovazn: Yep, I think we also tried to make trello cards for all the not done things in there | 20:34 |
| lifeless | ok 10 hypervisors up | 20:36 |
| jprovazn | greghaynes, well, it might be that 2) and 5) are part of an existing card | 20:36 |
| jprovazn | greghaynes, others should be covered | 20:36 |
| lifeless | resetting state on ERRORd VMs | 20:36 |
| *** jcoufal has quit IRC | 20:37 | |
| greghaynes | jprovazn: By others should be covered you mean they all have trello cards? | 20:37 |
| *** jistr has quit IRC | 20:37 | |
| jprovazn | greghaynes, I mean they are part of an existing trello card | 20:37 |
| greghaynes | ok. So do you know of any todos for it that you dont see in trello? | 20:38 |
| * greghaynes puts note in etherpad that these should all be in trello | 20:38 | |
| jprovazn | greghaynes, lifeless: this card is not clear to me - https://trello.com/c/DaIs1zxb/82-neutron-ha-redundant-environment - it mentions 2 different approaches: | 20:39 |
| jprovazn | l3 agent isn't currently able to run active-active for a network, so we either need to migrate networks on router quiesceing or run active-passive neutron agents (which as agents have uuids is possibly non-trivial) | 20:39 |
| lifeless | jprovazn: right, the card is the problem not the solution | 20:40 |
| lifeless | jprovazn: There is a third approach | 20:40 |
| jprovazn | lifeless, yes | 20:40 |
| jprovazn | the new patch | 20:40 |
| lifeless | jprovazn: yeah | 20:40 |
| jprovazn | lifeless, so I would prefer a/a or a/p | 20:41 |
| jprovazn | but not network migrations - it seems like waste of time | 20:41 |
| lifeless | jprovazn: all the options are a/a or a/p :P | 20:41 |
| lifeless | jprovazn: network migrations are a/p | 20:41 |
| jprovazn | lifeless, but graceful, right? | 20:41 |
| lifeless | jprovazn: yes, unless we're migrating because of a failure | 20:41 |
| lifeless | jprovazn: in which case they restore service | 20:41 |
| jprovazn | lifeless, so I understand it that we would do some special script which migrates networks when the resource plugin (mentioned by SpamapS before) is done | 20:42 |
| lifeless | jprovazn: or when pacemaker is failing a node over | 20:44 |
| *** CLOUDOUTAGE has joined #tripleo | 20:46 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 20:46 |
| *** CLOUDOUTAGE has quit IRC | 20:46 | |
| jprovazn | lifeless, hm, I would expect that pacemaker (+maybe drdb if needed) would do this for us - IOW with pacemaker we wouldn't have to do any migration script | 20:46 |
| lifeless | jprovazn: with pacemaker we need to manually override the uuid of the agent | 20:46 |
| lifeless | jprovazn: have you looked at the a/a status? having that would be most cool .... | 20:47 |
| *** CaptTofu has joined #tripleo | 20:47 | |
| jprovazn | lifeless, and this is the point where my neutron knowledge ends up (we need to manually override the uuid of the agent) | 20:47 |
| lifeless | jprovazn: ok so | 20:47 |
| lifeless | jprovazn: each neutron agent has a uui | 20:47 |
| jprovazn | lifeless, I think it's still nto merged | 20:47 |
| jprovazn | (the patch) | 20:47 |
| *** CaptTofu has quit IRC | 20:48 | |
| *** CaptTofu_ has joined #tripleo | 20:48 | |
| lifeless | jprovazn: if you run 3 l3 agents | 20:48 |
| lifeless | and create three networks | 20:48 |
| lifeless | they will end up one per agent (probably) | 20:48 |
| lifeless | so networks are partitioned over agents | 20:48 |
| lifeless | if an agent fails | 20:49 |
| lifeless | and you want to start it up elsewhere | 20:49 |
| lifeless | you have to override the agent id (basically hostname on the rabbit bus) | 20:49 |
| lifeless | or it won't run the networks its meant to - their router will remain down | 20:49 |
| *** sballe has joined #tripleo | 20:49 | |
| *** CaptTofu_ has quit IRC | 20:49 | |
| jprovazn | lifeless, I see - though this should not be so difficult, RHOS folks already do this with a/p with pacemaker setup | 20:50 |
| lifeless | jprovazn: righto - so we'd need to import their logic | 20:50 |
| lifeless | jprovazn: the nice thing about the migration approach though is that we get some natural redundancy because its a/a | 20:51 |
| lifeless | its just not a/a for any *single* network | 20:51 |
| dprince | lifeless: hi, when you deploy a new overcloud where do you run devtest_overcloud from? | 20:51 |
| lifeless | dprince: me @ home? me on the HP undercloud box for cd-overcloud? me on the HP undercloud box for ci-overcloud ? | 20:52 |
| dprince | lifeless: was asking some questions earlier (to understand things... not question them) | 20:52 |
| lifeless | jprovazn: so markmc said us running the unmerged patch might be a good thing to help it merge | 20:53 |
| dprince | lifeless: on the HP cloud specifically, just trying to understand how we might keep our old images around... just in case | 20:53 |
| markmc | lifeless, markmcclain ? | 20:53 |
| lifeless | dprince: deploying ci-overcloud? | 20:53 |
| jprovazn | lifeless, yes - we can at least test it | 20:53 |
| lifeless | bah | 20:53 |
| *** CaptTofu_ has joined #tripleo | 20:53 | |
| lifeless | jprovazn: markmcclain :P | 20:53 |
| lifeless | markmc: EFAIL :( | 20:53 |
| dprince | lifeless: Sure CI. that is a good start I think | 20:54 |
| markmc | lifeless, yes, markmcclain did a tragically bad job of choosing an IRC nick | 20:54 |
| lifeless | dprince: well point is the answer is different :P | 20:54 |
| lifeless | markmc: and a name, right? | 20:54 |
| lifeless | dprince: anyhow, I normally sit in /opt/stack/tripleo-image-elements/tripleo-cd and run ./deploy-ci-overcloud | 20:54 |
| markmc | lifeless, it's actually quite a nice name :) | 20:54 |
| dprince | lifeless: On the undercloud box? | 20:55 |
| lifeless | dprince: yes, ssh to heat-admin@cd-undercloud.tripleo.org; sudo su -; source undercloud creds | 20:55 |
| dprince | lifeless: gotcha. On the RH side I usually do everything from the seed host. | 20:56 |
| *** jdob has quit IRC | 20:57 | |
| lifeless | dprince: I find doing stuff from baremetal is faster :) | 20:57 |
| dprince | lifeless: And at this point I'm always keeping images at this point though. In fact I typically apply fixes by editing images so as not to break things. | 20:57 |
| dprince | lifeless: I said seed host (it is baremetal) | 20:57 |
| lifeless | dprince: oh right | 20:58 |
| lifeless | dprince: so yeah, the seed host has no tools etc on it on the hp cloud | 20:58 |
| *** blamar has quit IRC | 20:58 | |
| lifeless | dprince: whereas the undercloud comes pre loaded with a bunch of stuff | 20:58 |
| lifeless | dprince: but we should document these differences | 20:58 |
| dprince | lifeless: just wondering if we might adopt a common place to backup images. | 20:58 |
| lifeless | (or standardise) | 20:58 |
| dprince | lifeless: because we use load-images -d (and that doesn't keep things around in Glance) | 20:59 |
| lifeless | dprince: keep em in glance? we could change our scripts .. | 20:59 |
| dprince | lifeless: or... perhaps we need a new glance strategy | 20:59 |
| dprince | lifeless: either way is fine, we'll need to purge Glance from time to time... | 20:59 |
| lifeless | --keep-5 ? | 20:59 |
| lifeless | --keep=5 ? I meant | 20:59 |
| dprince | lifeless: but having them backed up, renamed is going to ward off frustrations I think | 21:00 |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 21:00 |
| *** blamar has joined #tripleo | 21:00 | |
| dprince | lifeless: okay, I'd be fine with a --keep option (or something). Will file a ticket on it and ponder it because I think this would be super useful and perhaps take the pressure off sometimes. | 21:01 |
| lifeless | ack | 21:01 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Add unit test for cache-url https://review.openstack.org/84493 | 21:02 |
| jprovazn | lifeless, with network migrations strategy for l3 agent, in case of failure of one of nodes - networks provided by l3 agent on this node will not be available until heat spawns new instance of the node (I suppose that any l3-agent state would be restored from /mnt/state)? | 21:03 |
| lifeless | jprovazn: no the agent is stateless | 21:03 |
| lifeless | jprovazn: with the migration strategy, we'd have pacemaker detect node down and trigger a rebalance which would divide the networks assigned to that agent to the other agents | 21:03 |
| jprovazn | lifeless, ok, so until new instance with same uuid boots up | 21:03 |
| lifeless | no | 21:03 |
| lifeless | gimme a sec | 21:04 |
| lifeless | trying to get CI back as a priority - 10 minutes? | 21:04 |
| jprovazn | sure | 21:04 |
| *** blamar has quit IRC | 21:04 | |
| jprovazn | lifeless, ok, I think I get it from your last comment | 21:06 |
| *** blamar has joined #tripleo | 21:06 | |
| lifeless | oh wow, nova --nic net-id and port combined gets the wrong order | 21:06 |
| lifeless | gnar | 21:06 |
| *** CaptTofu_ has quit IRC | 21:07 | |
| *** CaptTofu has joined #tripleo | 21:08 | |
| *** lblanchard has quit IRC | 21:10 | |
| meena | I have tempest included in the overcloud not-compute. I see tempest in two locations 1. under /opt/stack/venvs/ which contains only bin, local, share, lib and include directores. 2. under /opt/stack/tempest directories, which contains all the test source config files. When I run run_tempest from /opt/stack/venvs/tempest, does it pick up the tempest.conf from /opt/stack/tempests ? | 21:12 |
| *** CaptTofu has quit IRC | 21:12 | |
| lifeless | meena: no | 21:12 |
| lifeless | tempest.conf will be written by o-a-c to /etc/tempest/ | 21:13 |
| jp_at_hp | lifeless: Can I get eyes on an early change please? https://review.openstack.org/84549 I want to see if there is anything that would be fundamentally unacceptable in it, or any other things you want to suggest. | 21:13 |
| *** blamar has quit IRC | 21:13 | |
| lifeless | jp_at_hp: queued up | 21:13 |
| *** blamar has joined #tripleo | 21:14 | |
| *** dprince has quit IRC | 21:14 | |
| jp_at_hp | lifeless: ta, I'm heading offline now, but will look forward to comments first thing my morning. It is supposed to be moving towards https://etherpad.openstack.org/p/tripleo-incubator-rationalise-ui and https://etherpad.openstack.org/p/tripleo-devtest.sh-refactoring-blueprint. Thanks again. | 21:15 |
| *** CLOUDOUTAGE has joined #tripleo | 21:17 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 21:17 |
| *** CLOUDOUTAGE has quit IRC | 21:17 | |
| *** rpodolyaka1 has joined #tripleo | 21:17 | |
| *** blamar has quit IRC | 21:18 | |
| *** blamar has joined #tripleo | 21:18 | |
| *** e0ne has quit IRC | 21:18 | |
| *** jp_at_hp has quit IRC | 21:19 | |
| *** jprovazn has quit IRC | 21:20 | |
| *** blamar_ has joined #tripleo | 21:20 | |
| *** rpodolyaka1 has quit IRC | 21:22 | |
| *** blamar has quit IRC | 21:23 | |
| meena | lifeless: how do I run tempest from /opt/stack/venvs/tempest? | 21:25 |
| *** blamar has joined #tripleo | 21:25 | |
| *** blamar_ has quit IRC | 21:25 | |
| lifeless | meena: /opt/stack/venvs/tempest/bin/tempest, I presume | 21:26 |
| lifeless | andreaf: you were working on tempest in CI - which strategy are you pursuing | 21:26 |
| lifeless | ? | 21:26 |
| openstackgerrit | Trent Geerdes proposed a change to openstack/tripleo-incubator: Adding horizon element to undercloud extras https://review.openstack.org/84189 | 21:29 |
| *** markmc has quit IRC | 21:32 | |
| SpamapS | ok now my brain is back in working order | 21:35 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Prevent network interface logging w/ ovs agent https://review.openstack.org/84561 | 21:41 |
| *** fungi has joined #tripleo | 21:41 | |
| lifeless | oh for the lova of.. | 21:42 |
| lifeless | https://review.openstack.org/#/c/84208/ | 21:43 |
| lifeless | thats why I'm having trouble | 21:43 |
| jeblair | lifeless: fyi ram quota accounting seen by nodepool seems wrong (not sure if that's expected behavior at this point in the firedrill) | 21:43 |
| lifeless | jeblair: thanks | 21:43 |
| SpamapS | perserved | 21:43 |
| lifeless | jeblair: it probably is, as nova doesn't like havnig hypervisors hang | 21:44 |
| jeblair | picky picky | 21:44 |
| SpamapS | lifeless: so I see more up now, but 6,0,1,9 are all down? | 21:46 |
| *** hashar has quit IRC | 21:46 | |
| *** CLOUDOUTAGE has joined #tripleo | 21:48 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 21:48 |
| *** CLOUDOUTAGE has quit IRC | 21:48 | |
| *** e0ne has joined #tripleo | 21:49 | |
| *** matty_dubs is now known as matty_dubs|gone | 21:51 | |
| * fungi loves the bot | 21:51 | |
| *** CaptTofu has joined #tripleo | 21:51 | |
| *** e0ne_ has joined #tripleo | 21:51 | |
| *** e0ne has quit IRC | 21:51 | |
| SpamapS | lifeless: so is 84208 causing 2014-04-01 21:57:41.763 10126 WARNING nova.virt.libvirt.driver [req-14ef5f5c-a3dd-4de1-8520-1b3d3fa32437 d5af62d2183d431796d74c5bb119ec9f e01e473a9250498883955b80966a1e58] Timeout waiting for vif plugging callback for instance 56242d03-1aee-4673-bcc0-7a5df1073930 | 21:52 |
| SpamapS | lifeless: and if so, should we deploy a new nova-compute image? | 21:53 |
| *** dguerri_ has joined #tripleo | 21:53 | |
| lifeless | thats a 30 second delay on booting everything | 21:53 |
| *** sdake_ has joined #tripleo | 21:53 | |
| *** sdake_ has quit IRC | 21:53 | |
| *** sdake_ has joined #tripleo | 21:53 | |
| *** mtaylor has joined #tripleo | 21:53 | |
| lifeless | its because of the unfixed callback thing from neutron | 21:53 |
| *** sdake_ is now known as sdake_1 | 21:53 | |
| *** jpeeler1 has joined #tripleo | 21:53 | |
| *** dguerri has quit IRC | 21:54 | |
| *** jpeeler has quit IRC | 21:54 | |
| *** openstackgerrit has quit IRC | 21:54 | |
| *** jtomasek has quit IRC | 21:54 | |
| *** mordred has quit IRC | 21:54 | |
| *** sdake has quit IRC | 21:54 | |
| *** jtomasek has joined #tripleo | 21:55 | |
| SpamapS | lifeless: ok, anything I can do? | 21:55 |
| lifeless | I'm just trying to monkey this neutron usage fix in | 21:55 |
| lifeless | hopefully I can just do it in the control plane | 21:55 |
| lifeless | but I'm not sure of that | 21:55 |
| *** e0ne_ has quit IRC | 21:56 | |
| SpamapS | ah so deleting all the floatingips didn't fix it? | 21:56 |
| lifeless | SpamapS: not a floatingip issue | 22:01 |
| lifeless | SpamapS: I needed to get te-broker rebuilt with ip 192.168.1.1 | 22:01 |
| SpamapS | lifeless: mmm k | 22:01 |
| lifeless | SpamapS: but when I passed that in we hit the bug that the review above I listed fixes | 22:01 |
| lifeless | so eth0 was on 192.168.1.1 | 22:01 |
| lifeless | and eth1 was on 10.x | 22:01 |
| lifeless | -> no metadata | 22:02 |
| lifeless | -> fail | 22:02 |
| SpamapS | AH | 22:04 |
| *** weshay has quit IRC | 22:04 | |
| SpamapS | oh what tangled webs | 22:05 |
| *** morazi has quit IRC | 22:05 | |
| *** hashar has joined #tripleo | 22:11 | |
| *** sballe has quit IRC | 22:15 | |
| *** rpodolyaka1 has joined #tripleo | 22:18 | |
| *** CLOUDOUTAGE has joined #tripleo | 22:19 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 22:19 |
| *** CLOUDOUTAGE has quit IRC | 22:19 | |
| *** jp_at_hp has joined #tripleo | 22:20 | |
| *** rpodolyaka1 has quit IRC | 22:22 | |
| *** al has quit IRC | 22:22 | |
| *** giulivo has quit IRC | 22:27 | |
| jb11211 | Is there any work being done to get the tuskar-ui into tripleo? | 22:28 |
| *** giulivo has joined #tripleo | 22:29 | |
| *** jp_at_hp has quit IRC | 22:31 | |
| *** lsmola has quit IRC | 22:31 | |
| lifeless | jb11211: what do you mean ? | 22:34 |
| *** CaptTofu has quit IRC | 22:34 | |
| jb11211 | with a bit of work (tuskar has crappy python dependencies) I can get tuskar and horizon installed on the undercloud host. Are there plans to add tuskar-ui as an element which can also be added | 22:35 |
| *** CaptTofu has joined #tripleo | 22:35 | |
| jb11211 | to the undercloud image | 22:35 |
| lifeless | jb11211: certainly, that would be good | 22:35 |
| *** TravT has quit IRC | 22:36 | |
| jb11211 | lifeless: are you aware of any work that's already been done to this end or does it need to be started from scratch? | 22:38 |
| lifeless | jb11211: I'm not aware of any outstanding patch sets - but check the tripleo-image-elements review queue | 22:39 |
| *** CaptTofu has quit IRC | 22:39 | |
| *** lucas-afk has quit IRC | 22:41 | |
| jb11211 | I'm also thinking trying to add support for integrating keystone with LDAP for the under and over cloud images | 22:42 |
| jb11211 | it would need being able to add/change lines on the /etc/keystone/keystone.conf files. should I try and add that to the image elements or the heat templates? | 22:43 |
| lifeless | config changes typically require template changes in tripleo-image-elements and heat-template changes in tripleo-heat-templates | 22:45 |
| lifeless | the former cover the serialisation the latter the modelling | 22:45 |
| lifeless | current status of ci - broker wasn't handing out te's properly / effectively - assuming te's have hung, am redeploying | 22:49 |
| *** CLOUDOUTAGE has joined #tripleo | 22:50 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 22:50 |
| *** CLOUDOUTAGE has quit IRC | 22:50 | |
| *** e0ne has joined #tripleo | 22:52 | |
| lifeless | ./win 51 | 22:53 |
| *** al has joined #tripleo | 22:53 | |
| *** e0ne has quit IRC | 22:57 | |
| ccrouch | (5:38:58 PM) jb11211: lifeless: are you aware of any work that's already been done to this end or does it need to be started from scratch? | 23:01 |
| ccrouch | there has been some work done in this area under the "instack" umbrella | 23:01 |
| ccrouch | https://github.com/agroup/instack | 23:02 |
| ccrouch | and | 23:02 |
| ccrouch | https://github.com/agroup/instack-undercloud | 23:02 |
| lifeless | ccrouch: huh | 23:02 |
| lifeless | I didn't know about that | 23:02 |
| ccrouch | well you knew about instack right? | 23:03 |
| ccrouch | slagle: was meant to have talked about that at sunnyvale | 23:03 |
| ccrouch | most of the stuff in instack-undercloud is just scripting to get the right packaged version of tripleo and tuskar installed | 23:04 |
| *** michchap has quit IRC | 23:05 | |
| *** decede has quit IRC | 23:05 | |
| ccrouch | there is a very basic tuskar element, just there for expediency. The right solution is tuskar related elements in t-i-e, as jb11211 was mentioning | 23:05 |
| *** decede has joined #tripleo | 23:05 | |
| ccrouch | we just haven't had the bandwidth to do that yet | 23:05 |
| ccrouch | there is also a scripts to initialize the overcloud, which will go away when that feature is in tuskar (os-cloud-config ?) | 23:07 |
| lifeless | ccrouch: yes - but why not use the current scripts in incubator - so we don't have 3 copies of them ? | 23:08 |
| *** yamahata has quit IRC | 23:08 | |
| lifeless | ccrouch: incubator -> os-cloud-config is the plan | 23:08 |
| lifeless | ccrouch: if instack has a copy of them, we'll end up with a harder to track migration | 23:08 |
| lifeless | or I worry that we will anyway | 23:08 |
| * SpamapS has no idea what instack is | 23:09 | |
| ccrouch | lifeless: we use all the incubator scripts | 23:10 |
| ccrouch | e.g. init-keystone | 23:10 |
| ccrouch | setup-endpoints | 23:10 |
| ccrouch | setup-neutron | 23:10 |
| lifeless | ccrouch: ok; I must have misinterpreted what you said :) | 23:10 |
| ccrouch | no worries :-) | 23:11 |
| ccrouch | we've tried hard not to reinvent stuff which was already done in incubator. As you say, useless duplication when those are available and being tested | 23:12 |
| lifeless | cool cool | 23:14 |
| lifeless | I read 12:07 < ccrouch> there is also a scripts to initialize the overcloud, which will go away when that feature is in tuskar (os-cloud-config ?) | 23:14 |
| lifeless | as meaning you had stuff that would be in os-cloud-config | 23:14 |
| ccrouch | SpamapS: apologies. sounds like slagle was too shy in sunnyvale, though i thought he'd mentioned this on a dev thread at some point | 23:14 |
| ccrouch | anyways instack is basically just a way to install t-i-e on an actual running machine, versus an image | 23:14 |
| SpamapS | ccrouch: I sat with slagle the whole week. We were just trying to fix CI. ;) | 23:14 |
| ccrouch | doh :-) | 23:15 |
| ccrouch | well i'm sure he'd be happy to chat more about it | 23:15 |
| ccrouch | lifeless: ah ok, no just using the same stuff as incubator, which will eventually be replaced by os-cloud-config | 23:17 |
| SpamapS | ccrouch: we've talked about it before. I've never much liked the idea, as I think it will bleed upward into the elements as overly complex install scripts. However, I understand why it is desired. | 23:18 |
| *** rpodolyaka1 has joined #tripleo | 23:18 | |
| *** CLOUDOUTAGE has joined #tripleo | 23:21 | |
| CLOUDOUTAGE | lifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle -- ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage | 23:21 |
| *** CLOUDOUTAGE has quit IRC | 23:21 | |
| *** rpodolyaka1 has quit IRC | 23:21 | |
| *** michchap has joined #tripleo | 23:21 | |
| ccrouch | SpamapS: so far so good i think. Besides supporting the package based install of elements, which is orthogonal, I dont think we've had to change the elements to support this specifically. though slagle can keep me honest there | 23:22 |
| lifeless | SpamapS: I think 'run-once' stuff is ok, its the 'evolve it' stuff I worry about bleeding | 23:23 |
| *** derekh has joined #tripleo | 23:23 | |
| lifeless | derekh: HI | 23:24 |
| lifeless | derekh: te's weren't responding | 23:24 |
| derekh | lifeless: hi ya | 23:24 |
| derekh | lifeless: oh, any idea why? | 23:25 |
| lifeless | derekh: so I think there's a bug in the te side agent or something, anyhow redeploying atm, 6 ACTIVE 4 spawning | 23:25 |
| lifeless | derekh: no | 23:25 |
| derekh | lifeless: ok | 23:25 |
| lifeless | derekh: but I imagine killing the broker and then making a new one could well tickle it | 23:25 |
| derekh | lifeless: ok, sounds like that something we've done before when broker used to fall over but maybe its a slightly different case | 23:26 |
| SpamapS | lifeless: one thing that I think we've learned, is that it is really hard to keep image build times fast | 23:27 |
| SpamapS | lifeless: still necessary, but when they slip.. users get impatient and want to do things in place. | 23:27 |
| lifeless | SpamapS: yeah | 23:28 |
| lifeless | ok te's registering now | 23:28 |
| lifeless | well machines atm | 23:28 |
| lifeless | ok te's registered | 23:28 |
| lifeless | derekh: I think the te's weren't registered with the broker, I was misreading the status output | 23:29 |
| lifeless | yes - https://jenkins07.openstack.org/job/check-tripleo-undercloud-precise/373/console is away | 23:29 |
| derekh | woot | 23:30 |
| lifeless | lunch then some reviews | 23:31 |
| lifeless | greghaynes: what do you need? | 23:31 |
| greghaynes | all knowledge of heat via instant osmisis | 23:35 |
| *** stevehuang has quit IRC | 23:36 | |
| greghaynes | I did just push two patches, one which depends on a third, that could use reviewing | 23:36 |
| greghaynes | https://review.openstack.org/#/c/83675/ is really the one that needs reviewing | 23:38 |
| slagle | lifeless: instack is just the "clever" name i gave the scripted undercloud installer we talked about in sunnyvale | 23:40 |
| lifeless | slagle: ack | 23:40 |
| slagle | i didn't spend enough time coming up with a cool name that starts with M*. | 23:40 |
| lifeless | mystack? | 23:40 |
| lifeless | :P | 23:40 |
| slagle | could be :) | 23:40 |
| slagle | jb11211: but, no, we don't yet have elements for tuskar | 23:41 |
| SpamapS | slagle: Matzo | 23:41 |
| lifeless | ok 40 testenvironmets up and running | 23:41 |
| lifeless | SpamapS: Balls! | 23:41 |
| SpamapS | ftw | 23:41 |
| greghaynes | lifeless: I could probably use some advice on what to poke at next - I was thinking id get the software-config patch working for me (I think I have a mirror-related issue with it) and then try to get merge.py able to do CONTROLSCALE > 1... not sure if thats best route to victory though? | 23:43 |
| greghaynes | Alternate idea was trying to figure out enough heat to work on the resource SpamapS was mentioning earlier | 23:45 |
| greghaynes | hrm, I think openstack gerrit bot is MIA | 23:46 |
| SpamapS | greghaynes: Testing the software-config patch and getting CONTROLSCALE > 1 would be very helpful | 23:46 |
| greghaynes | sweet | 23:48 |
| *** hashar has quit IRC | 23:50 | |
| *** e0ne has joined #tripleo | 23:52 | |
| *** xuhaiwei has joined #tripleo | 23:54 | |
| *** e0ne has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!