| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Add gerrit support to source-repositories. https://review.openstack.org/63021 | 00:00 |
|---|---|---|
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Log unsupported source repository types. https://review.openstack.org/63022 | 00:00 |
| Ng | lifeless: yeah that one should be pretty easy. I was going to take a crack at it last week, but didn't get to it | 00:02 |
| Ng | r63021 lets you do: | 00:02 |
| Ng | declare -x DIB_REPOLOCATION_nova="https://review.openstack.org/openstack/nova" | 00:03 |
| Ng | declare -x DIB_REPOREF_nova="refs/changes/12/59312/17" | 00:03 |
| Ng | declare -x DIB_REPOTYPE_nova="gerrit" | 00:03 |
| Ng | and get a devtest.sh run with all of the nova rebuild patches (unless they've been rebased during the day and the refspec needs updating, natch ;) | 00:03 |
| Ng | and with that, the sleeping calls | 00:08 |
| lifeless | cool | 00:09 |
| lifeless | ♥ | 00:09 |
| *** vipul is now known as vipul-away | 00:14 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Replace references to Quantum with Neutron in docs https://review.openstack.org/63026 | 00:20 |
| *** cd-undercloud has joined #tripleo | 00:34 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 00:34 |
| *** cd-undercloud has quit IRC | 00:34 | |
| *** boris-42 has quit IRC | 00:35 | |
| *** AaronGr is now known as AaronGr_afk | 00:40 | |
| devananda | btw, i found a work-around for the "rebooting seed borks neutron" problem, which seems to still be affecting me (as of build yesterday morning) | 00:41 |
| devananda | delete the ports, subnets, and nets from neutron. then call setup-neutron again, then delete and re-add my baremetal node, wait ~2 minutes, and I can "heat stack-create ... undercloud" again | 00:42 |
| pleia2 | oh good, that's much better than "don't reboot seed" | 00:44 |
| devananda | indeed | 00:44 |
| *** jayg is now known as jayg|g0n3 | 00:47 | |
| *** kui has quit IRC | 00:56 | |
| *** CaptTofu has quit IRC | 00:56 | |
| *** CaptTofu has joined #tripleo | 00:56 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Heat patchset has hit trunk. https://review.openstack.org/63034 | 00:56 |
| lifeless | SpamapS: mordred: did you see http://www.slideshare.net/daniel_bilar/acoustic-20131218 ? | 00:57 |
| *** ccrouch has joined #tripleo | 01:08 | |
| *** cd-undercloud has joined #tripleo | 01:09 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 01:09 |
| *** cd-undercloud has quit IRC | 01:09 | |
| lifeless | SpamapS: care to approve my fix for ^ ? | 01:10 |
| *** rwsu has quit IRC | 01:18 | |
| *** rwsu has joined #tripleo | 01:19 | |
| pleia2 | lifeless: as I go through some of these scripts I'm noticing a fair number of missing license headers on scripts in tripleo-incubator/scripts (particularly glaring in our devtest*.sh scripts), what's our policy here? | 01:31 |
| lifeless | pleia2: LALALALALALALALALALALALA | 01:31 |
| pleia2 | well it's now logged! | 01:31 |
| lifeless | eavesdrop go away! | 01:31 |
| lifeless | pleia2: seriously? We should add em. | 01:32 |
| pleia2 | lifeless: I might just swoop through and do them all in one commit then | 01:32 |
| pleia2 | should only take 20 minutes or so to grep through git to grab all copyright holders (mostly hp and rh anyway) | 01:32 |
| lifeless | pleia2: meh, do it in late jan | 01:32 |
| pleia2 | hah, ok | 01:33 |
| lifeless | pleia2: 20m should get boot-seed-vm fixed ;) | 01:33 |
| lifeless | and thats /way/ more urgent | 01:33 |
| pleia2 | ok | 01:33 |
| *** nosnos has joined #tripleo | 01:38 | |
| *** cd-undercloud has joined #tripleo | 01:43 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 01:43 |
| *** cd-undercloud has quit IRC | 01:43 | |
| *** itooon has quit IRC | 01:45 | |
| *** julim has quit IRC | 01:49 | |
| *** jcooley_ has quit IRC | 01:55 | |
| *** itooon has joined #tripleo | 02:04 | |
| *** kui has joined #tripleo | 02:06 | |
| *** rongze has joined #tripleo | 02:12 | |
| *** cd-undercloud has joined #tripleo | 02:18 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 02:18 |
| *** cd-undercloud has quit IRC | 02:18 | |
| *** jcooley_ has joined #tripleo | 02:27 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Consolidate under and overcloud passwd scripts https://review.openstack.org/63044 | 02:28 |
| *** nosnos_ has joined #tripleo | 02:30 | |
| pleia2 | I didn't break getopt this time, I promise | 02:30 |
| pleia2 | hopefully | 02:30 |
| pleia2 | wait, I did | 02:31 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Consolidate under and overcloud passwd scripts https://review.openstack.org/63044 | 02:32 |
| *** nosnos has quit IRC | 02:33 | |
| lifeless | pleia2: your comment is confusing me https://review.openstack.org/#/c/63044/2/scripts/devtest_overcloud.sh | 02:35 |
| lifeless | pleia2: if it defaults to undercloud, how is -c OVERCLOUD not needed? | 02:36 |
| pleia2 | oh nice, I commented in the wrong file | 02:36 |
| *** jcooley_ has quit IRC | 02:37 | |
| pleia2 | it may be EOD time :) | 02:37 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-incubator: Use Image IDs to make image based updates work https://review.openstack.org/62982 | 02:44 |
| SpamapS | pleia2: IMO before you add them we need a bash hacking equivalent. | 02:45 |
| SpamapS | pleia2: (before you add license headers, sorry) | 02:45 |
| lifeless | SpamapS: how is that conected? | 02:49 |
| asalkeld | lifeless, would you mind resending that sql with a new gpg key (I had to make a new one 5E28FB23) | 02:49 |
| lifeless | asalkeld: LOL | 02:49 |
| asalkeld | (angus@salkeld.id.au) | 02:49 |
| asalkeld | I make that key you used 2 years ago | 02:50 |
| asalkeld | and lost it :( | 02:50 |
| asalkeld | trying to educate my self on pgp | 02:50 |
| asalkeld | bit of a pain | 02:50 |
| lifeless | asalkeld: have you pushed it up to subkeys.pgp.net ? | 02:53 |
| *** cd-undercloud has joined #tripleo | 02:53 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 02:53 |
| *** cd-undercloud has quit IRC | 02:53 | |
| asalkeld | I did to another server, syncing that one no too | 02:54 |
| lifeless | got it | 02:55 |
| lifeless | is 1D94 75AF 326A ADB2 78C3 879E ACBC 3F31 5E28 FB23 your fingerprint ? | 02:55 |
| SpamapS | lifeless: we need a bash checker that makes sure we don't backslide on license headers, etc. | 02:56 |
| clarkb | run hacking on it and only have it run that one rule >_> | 02:56 |
| lifeless | SpamapS: we don't need one to fix a glaring bug | 02:56 |
| lifeless | asalkeld: ^ | 02:57 |
| asalkeld | sorry dude, getting there | 02:57 |
| SpamapS | lifeless: There are other benefits. It can just run 'bash -n' too. Anyhoo, I just prefer to fix the actual bug before attacking the symptoms. :-P Not going to enforce it or anything. | 02:57 |
| asalkeld | lifeless, what command do you want me to run? | 02:59 |
| lifeless | asalkeld: gpg2 --fingerprint angus@salkeld.id.au | 02:59 |
| lifeless | should come back with just one thing | 02:59 |
| lifeless | match up the primary key fingerprint | 02:59 |
| asalkeld | Key fingerprint = 1D94 75AF 326A ADB2 78C3 879E ACBC 3F31 5E28 FB23 | 02:59 |
| asalkeld | looks the same | 03:00 |
| asalkeld | sorry lifeless, never got into gpg before | 03:00 |
| lifeless | sent | 03:02 |
| *** vkozhukalov has joined #tripleo | 03:06 | |
| asalkeld | lifeless, groan - I tried to get smart an pratise backing up / restoring and f'd up - sigh | 03:08 |
| lifeless | asalkeld: orly | 03:09 |
| asalkeld | where is the gun | 03:09 |
| lifeless | asalkeld: you *don't* say? | 03:09 |
| asalkeld | I think I am a test of usablity, this is failing | 03:09 |
| asalkeld | --export-ownertrust | 03:10 |
| asalkeld | who would have thought | 03:10 |
| *** jergerber has joined #tripleo | 03:27 | |
| *** cd-undercloud has joined #tripleo | 03:28 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 03:28 |
| *** cd-undercloud has quit IRC | 03:28 | |
| *** jergerber has quit IRC | 03:30 | |
| *** jergerber has joined #tripleo | 03:37 | |
| *** CaptTofu has quit IRC | 03:40 | |
| *** cd-undercloud has joined #tripleo | 03:41 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 03:41 |
| *** cd-undercloud has quit IRC | 03:41 | |
| *** CaptTofu has joined #tripleo | 03:45 | |
| *** ccrouch has quit IRC | 03:50 | |
| *** cd-undercloud has joined #tripleo | 03:50 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 03:50 |
| *** cd-undercloud has quit IRC | 03:50 | |
| *** julim has joined #tripleo | 03:54 | |
| *** julim has quit IRC | 03:57 | |
| *** vipul-away is now known as vipul | 04:01 | |
| *** vipul has joined #tripleo | 04:02 | |
| *** yamahata__ has quit IRC | 04:12 | |
| *** yamahata__ has joined #tripleo | 04:13 | |
| *** yamahata__ has quit IRC | 04:13 | |
| *** yamahata_ has joined #tripleo | 04:13 | |
| *** AaronGr_afk is now known as AaronGr_Zzz | 04:14 | |
| *** olaph has quit IRC | 04:24 | |
| *** jcooley_ has joined #tripleo | 04:39 | |
| asalkeld | lifeless, I sent you an email with my new and working key - when you have a moment | 04:41 |
| *** boris-42 has joined #tripleo | 04:43 | |
| *** rongze has quit IRC | 04:44 | |
| *** akuznetsov has joined #tripleo | 04:54 | |
| *** CaptTofu has quit IRC | 04:56 | |
| lifeless | asalkeld: 8417 81C4 3BE3 7E25 9E6C ED7B 5AC3 6012 F166 35A8 is the fingerprint ? | 05:04 |
| asalkeld | yip | 05:04 |
| *** nosnos_ has quit IRC | 05:06 | |
| lifeless | try that | 05:06 |
| *** nosnos has joined #tripleo | 05:07 | |
| *** rushiagr has joined #tripleo | 05:08 | |
| *** itooon has quit IRC | 05:11 | |
| *** rushiagr2 has joined #tripleo | 05:12 | |
| *** rushiagr has quit IRC | 05:16 | |
| *** CaptTofu has joined #tripleo | 05:16 | |
| *** rongze has joined #tripleo | 05:18 | |
| *** rongze has quit IRC | 05:23 | |
| *** sdake2 has quit IRC | 05:28 | |
| *** rpodolyaka1 has joined #tripleo | 05:31 | |
| *** olaph has joined #tripleo | 05:33 | |
| *** rushiagr2 is now known as rushiagr | 05:40 | |
| asalkeld | lifeless, if I change your sql dump from: | 05:58 |
| asalkeld | ENGINE=InnoDB AUTO_INCREMENT=1166 DEFAULT CHARSET=latin1; | 05:58 |
| asalkeld | to | 05:58 |
| asalkeld | ENGINE=InnoDB AUTO_INCREMENT=1166 DEFAULT CHARSET=utf8; | 05:58 |
| asalkeld | it works | 05:58 |
| asalkeld | seems like the latin1 charset is causing some trouble | 05:59 |
| asalkeld | https://github.com/openstack/heat/blob/master/heat/db/sqlalchemy/migrate_repo/versions/031_stack_lock.py#L31 | 06:00 |
| *** jergerber has quit IRC | 06:02 | |
| *** rongze has joined #tripleo | 06:20 | |
| *** rongze has quit IRC | 06:25 | |
| *** rongze has joined #tripleo | 06:27 | |
| *** rongze has quit IRC | 06:28 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Use Image IDs to make image based updates work https://review.openstack.org/62982 | 06:31 |
| lifeless | asalkeld: not sure why it's latin1 | 06:34 |
| lifeless | asalkeld: did something change in prior migrations? | 06:34 |
| asalkeld | no but it seems the varchar thinks it is utf8 | 06:34 |
| asalkeld | then the foreign key looks for a latin1 | 06:35 |
| asalkeld | and can't find it | 06:35 |
| asalkeld | lifeless, do you know if those comments do anything? | 06:36 |
| asalkeld | /*!40101 SET NAMES utf8 */ | 06:36 |
| asalkeld | /*!40101 SET character_set_client = utf8 */; | 06:36 |
| *** rushiagr has quit IRC | 06:36 | |
| asalkeld | it *might* be setting the default varchar to utf8 | 06:36 |
| *** CaptTofu has quit IRC | 06:36 | |
| *** rongze has joined #tripleo | 06:37 | |
| *** CaptTofu has joined #tripleo | 06:37 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack/tripleo-incubator: configure-vm: refactor template formatting https://review.openstack.org/63057 | 06:39 |
| *** rushiagr has joined #tripleo | 06:44 | |
| rpodolyaka1 | morning tripleo | 06:47 |
| lifeless | rpodolyaka1: morning | 06:48 |
| lifeless | rpodolyaka1: I believe jog0 has been hacking the stack | 06:48 |
| rpodolyaka1 | lifeless: hey! cool! | 06:48 |
| rpodolyaka1 | I had a kind of unpleasant merge conflict to solve yesterday (rebuild() was rewritten to use unified objects models instead of instance dicts) | 06:49 |
| rpodolyaka1 | so I'm still working on API samples | 06:49 |
| lifeless | hah, nice | 06:50 |
| rpodolyaka1 | but API samples must be the last thing on the list to remove WIP status from that patch | 06:52 |
| *** vkozhukalov has quit IRC | 06:52 | |
| *** jamezpolley has quit IRC | 06:53 | |
| *** akuznetsov has quit IRC | 06:59 | |
| *** jamezpolley has joined #tripleo | 07:00 | |
| *** mrunge has joined #tripleo | 07:01 | |
| lifeless | SpamapS: do you want to approve https://review.openstack.org/#/c/62976/ ? | 07:09 |
| lifeless | SpamapS: it's what I meant when I pinged you about it; it should unbreak tripleo-cd | 07:09 |
| *** rpodolyaka1 has quit IRC | 07:13 | |
| SpamapS | lifeless: approving. We need to open a bug, I think, that states that we have no automated way to assert those bits though | 07:14 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Don't do one-time setup when updating a cloud. https://review.openstack.org/62976 | 07:15 |
| *** rushiagr has quit IRC | 07:22 | |
| *** rushiagr has joined #tripleo | 07:22 | |
| marios | morning gents - am trying to revive my devtest setup so i can test the removal of workaround on https://bugs.launchpad.net/tripleo/+bug/1254555 - seed comes up but fails on waitin for keystone. real reason is problem with the power driver on seed (ssh --> var/log/messages: http://fpaste.org/63121/13874377/ ) - any clues ? :) | 07:23 |
| uvirtbot | Launchpad bug 1254555 in neutron "tenant does not see network that is routable from tenant-visible network until neutron-server is restarted" [Low,Confirmed] | 07:23 |
| SpamapS | marios: interesting | 07:29 |
| marios | SpamapS: will try a few more things, worst case I will open the review (i tested the neutron fix on devstack and can't recreate the bug, i.e. restarting q-svc the networks remain in place/visible) and ask people to test it; we can at least start progress and I can fix my setup in parallel | 07:32 |
| *** vipul is now known as vipul-away | 07:34 | |
| SpamapS | marios: I just built myself a new devtest today from scratch on Ubuntu and that script completed fine | 07:38 |
| marios | SpamapS: ok thanks, good to know at least its something here/fedora specific. should it matter if i invoke devtest as 'root' vs as 'stack' | 07:39 |
| SpamapS | marios: devtest should, I believe, sudo when it needs to | 07:40 |
| SpamapS | which is mostly deep within diskimage-builder I think | 07:40 |
| marios | ok, tx. how do you run it? do you create a 'stack' user ? | 07:41 |
| SpamapS | marios: I run it as me | 07:42 |
| marios | hehe, k tx | 07:42 |
| SpamapS | marios: but I have not had the automated thing ever finish.. I usually do a few steps by hand after a failure | 07:42 |
| marios | SpamapS: ah, ok - well, fwiw i've also only ever been able to get a complete setup 'by hand' in the past. its just been a while and i thought the script was 'there' (almost ;) ) | 07:43 |
| SpamapS | marios: I wish it was. | 07:45 |
| SpamapS | marios: Ng has been working on that, IIRC | 07:45 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Set the correct path to os-collect-config https://review.openstack.org/62890 | 07:47 |
| *** boris-42 has quit IRC | 07:48 | |
| SpamapS | doh | 07:49 |
| *** rongze has quit IRC | 07:49 | |
| SpamapS | why would you want to explicitly set the path of a command like os-collect-config anyway | 07:49 |
| *** rongze has joined #tripleo | 07:49 | |
| * SpamapS shakes head.. systemd ye have confused us all | 07:49 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Do not install unneeded python-zmq package https://review.openstack.org/62893 | 07:50 |
| SpamapS | lifeless: I think we should just alter those tables back to utf8 | 07:50 |
| SpamapS | lifeless: and I'll modify the mysql element to assert utf8 as the default charset | 07:50 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Add quotes around $DISTRO https://review.openstack.org/62906 | 07:52 |
| *** rushiagr is now known as rushiagr_away | 07:55 | |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Configure os-collect-config on seed properly https://review.openstack.org/63065 | 08:07 |
| *** rushiagr_away is now known as rushiagr | 08:12 | |
| lifeless | SpamapS: cool | 08:14 |
| SpamapS | lifeless: testing modified mysql element on fedora ATM | 08:14 |
| SpamapS | the fedora cloud images.. | 08:14 |
| SpamapS | install a lot of updates.. :-P | 08:14 |
| *** jcoufal has joined #tripleo | 08:19 | |
| lifeless | SpamapS: F20 was just released | 08:20 |
| lifeless | SpamapS: should have to install? | 08:20 |
| SpamapS | Perhaps we need to update dib.. dunno | 08:21 |
| SpamapS | might explain why mirrors are timing out tho | 08:21 |
| *** akuznetsov has joined #tripleo | 08:22 | |
| *** vkozhukalov has joined #tripleo | 08:29 | |
| *** rdopieralski has joined #tripleo | 08:35 | |
| *** jtomasek has joined #tripleo | 08:36 | |
| *** jtomasek has quit IRC | 08:41 | |
| *** pblaho has joined #tripleo | 08:45 | |
| *** jtomasek has joined #tripleo | 08:50 | |
| *** jtomasek has quit IRC | 08:51 | |
| *** jtomasek has joined #tripleo | 08:51 | |
| *** marun has quit IRC | 08:53 | |
| *** jcooley_ has quit IRC | 08:54 | |
| *** ifarkas has joined #tripleo | 09:01 | |
| *** jtomasek has quit IRC | 09:06 | |
| *** jtomasek has joined #tripleo | 09:07 | |
| *** jtomasek has quit IRC | 09:09 | |
| *** jtomasek has joined #tripleo | 09:09 | |
| *** jtomasek has quit IRC | 09:12 | |
| *** jtomasek has joined #tripleo | 09:12 | |
| *** athomas has joined #tripleo | 09:15 | |
| *** lucasagomes has joined #tripleo | 09:24 | |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Fix building mysql element on non-upstart systems https://review.openstack.org/63077 | 09:24 |
| *** akuznetsov has quit IRC | 09:26 | |
| Ng | morning | 09:27 |
| lifeless | wait,what? | 09:31 |
| SpamapS | lifeless: that latest patch? | 09:33 |
| SpamapS | I'm not sure how anything has worked.. :-P | 09:33 |
| SpamapS | ./boot-stack/install.d/01-boot-stack:install-packages mysql-server python-mysqldb | 09:37 |
| SpamapS | ah | 09:37 |
| SpamapS | we're not using the mysql element | 09:37 |
| *** jcooley_ has joined #tripleo | 09:38 | |
| SpamapS | ok thats enough revelations for the day | 09:39 |
| lifeless | SpamapS: we perhaps should :) | 09:43 |
| lifeless | SpamapS: no, I was wait, whatting ng's statement of morning | 09:43 |
| Ng | it totally is morning! | 09:46 |
| erwan_ta- | lo lifeless | 09:49 |
| erwan_ta- | lifeless: I was wondering if the bug you had was by trying to boot locally or not | 09:49 |
| erwan_ta- | lifeless: because it seems that you can work in a full pxe system even after installation | 09:50 |
| *** boris-42 has joined #tripleo | 09:52 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Add ovsbridge option to configure-vm https://review.openstack.org/62939 | 09:54 |
| lifeless | erwan_ta-: no, we don't try to boot locally | 09:54 |
| lifeless | not intentionally anyhow | 09:55 |
| erwan_ta- | lifeless: ok | 09:55 |
| erwan_ta- | lifeless: are you able to estimate when this bug occured : i.e before or after the pxelinux stuff, etc.. | 09:56 |
| erwan_ta- | in what part of the boot sequence (which is long) this is happening | 09:56 |
| erwan_ta- | and what release of pxelinux | 09:56 |
| *** jtomasek has quit IRC | 09:56 | |
| *** jtomasek has joined #tripleo | 09:57 | |
| lifeless | erwan_ta-: no, no, and the one installed on ubuntu | 10:04 |
| lifeless | I will poke around tomorrow, or Ng might be able to answer in more detail :) | 10:04 |
| Ng | what what | 10:05 |
| *** jcooley_ has quit IRC | 10:09 | |
| erwan_ta- | lifeless: ideally, making a moving of the booting process would be so usefull | 10:09 |
| erwan_ta- | movie | 10:09 |
| *** rongze has quit IRC | 10:11 | |
| *** derekh has joined #tripleo | 10:15 | |
| lifeless | erwan_ta-: we don't have anyone on the team even in the same city as the servers :) | 10:17 |
| *** jcooley_ has joined #tripleo | 10:19 | |
| *** rongze has joined #tripleo | 10:23 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Replace references to Quantum with Neutron in docs https://review.openstack.org/63026 | 10:25 |
| erwan_ta- | lifeless: maybe via the ilo ? | 10:25 |
| erwan_ta- | lifeless: recording the desktop showing the ilo booting | 10:25 |
| erwan_ta- | lifeless: if you see this opcode error it means you have the view on the server at this time | 10:26 |
| lifeless | erwan_ta-: I saw that when activating the textcons on the ilo when it looked like it hadn't booted | 10:28 |
| *** martyntaylor has joined #tripleo | 10:28 | |
| lifeless | erwan_ta-: it cleared just after I switched to the textcons, so no idea how long it had been sitting in the textcons buffer | 10:28 |
| lifeless | Ng: what what :) | 10:28 |
| *** rushiagr is now known as rushiagr_away | 10:30 | |
| *** akuznetsov has joined #tripleo | 10:32 | |
| *** kui has quit IRC | 10:35 | |
| derekh | any thoughts on https://review.openstack.org/#/c/62979/1/scripts/devtest_overcloud.sh I see it has a partial fix https://bugs.launchpad.net/neutron/+bug/1254555 | 10:35 |
| uvirtbot | Launchpad bug 1254555 in neutron "tenant does not see network that is routable from tenant-visible network until neutron-server is restarted" [Low,Confirmed] | 10:35 |
| derekh | but new report as being reproduced on bug | 10:35 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Revert "Work around neutron floatingip race condition" https://review.openstack.org/62979 | 10:36 |
| derekh | should we just merge anyways and find out | 10:36 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Heat patchset has hit trunk. https://review.openstack.org/63034 | 10:37 |
| Ng | derekh: we just did ;) | 10:37 |
| derekh | Ng: ahh well, thats the end of that conversation :-) | 10:37 |
| Ng | derekh: fwiw, lifeless did suggest in the commit message that this was to see if it is indeed fixed | 10:38 |
| lifeless | Also in the meeting this morning | 10:38 |
| rpodolyaka | Ng: derekh: this might sound crazy but enikanorov suggested that restarting neutron-server might actually trigger that issue | 10:38 |
| derekh | Yup, but that was before the new report appeared on the bug | 10:38 |
| lifeless | which neither derekh or Ng were at (but the mintes are on the minute site) | 10:38 |
| *** greghaynes has quit IRC | 10:39 | |
| derekh | The new report on the bug is why I brought it up here, comment 8 https://review.openstack.org/#/c/62979/1/scripts/devtest_overcloud.sh | 10:39 |
| derekh | anyways its merged now, so lets just wait and see | 10:40 |
| rpodolyaka | :) | 10:40 |
| *** max_lobur_afk is now known as max_lobur | 10:40 | |
| * derekh must look at the links befor he pastes them | 10:40 | |
| *** greghaynes has joined #tripleo | 10:41 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix building mysql element on non-upstart systems https://review.openstack.org/63077 | 10:47 |
| *** rushiagr_away is now known as rushiagr | 10:47 | |
| *** martyntaylor has quit IRC | 10:48 | |
| *** jcooley_ has quit IRC | 10:49 | |
| *** martyntaylor has joined #tripleo | 10:59 | |
| *** max_lobur is now known as max_lobur_afk | 10:59 | |
| *** nosnos has quit IRC | 11:00 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Add element to modify /etc/apt/sources.list in dib https://review.openstack.org/62594 | 11:01 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Configure os-collect-config on seed properly https://review.openstack.org/63065 | 11:03 |
| *** rpodolyaka has quit IRC | 11:06 | |
| *** rongze has quit IRC | 11:13 | |
| *** martyntaylor has quit IRC | 11:15 | |
| *** pblaho has quit IRC | 11:18 | |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Fix source-repositories to work with gerrit refs. https://review.openstack.org/63021 | 11:22 |
| *** akrivoka has joined #tripleo | 11:26 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix the grub2 config location for non Ubuntu/Debian systems https://review.openstack.org/62885 | 11:27 |
| *** rpodolyaka has joined #tripleo | 11:28 | |
| *** jcooley_ has joined #tripleo | 11:32 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack/tripleo-incubator: configure-vm: refactor template formatting https://review.openstack.org/63057 | 11:32 |
| *** rpodolyaka has quit IRC | 11:33 | |
| *** athomas has quit IRC | 11:39 | |
| *** rushiagr has quit IRC | 11:40 | |
| *** athomas has joined #tripleo | 11:46 | |
| *** rushiagr has joined #tripleo | 11:50 | |
| *** martyntaylor has joined #tripleo | 11:50 | |
| *** rushiagr has quit IRC | 11:55 | |
| *** rushiagr has joined #tripleo | 11:55 | |
| *** CaptTofu has quit IRC | 12:01 | |
| *** CaptTofu has joined #tripleo | 12:01 | |
| *** jcooley_ has quit IRC | 12:04 | |
| *** rongze has joined #tripleo | 12:04 | |
| *** jtomasek has quit IRC | 12:05 | |
| *** jtomasek has joined #tripleo | 12:06 | |
| Ng | we need to get markmc into core :) | 12:19 |
| *** rongze_ has joined #tripleo | 12:24 | |
| * derekh nodes | 12:24 | |
| *** rongze has quit IRC | 12:28 | |
| *** rpodolyaka has joined #tripleo | 12:28 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add ovsbridge option to configure-vm https://review.openstack.org/62939 | 12:35 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: configure-vm: refactor template formatting https://review.openstack.org/63057 | 12:35 |
| *** CaptTofu has quit IRC | 12:37 | |
| *** CaptTofu has joined #tripleo | 12:37 | |
| *** CaptTofu has quit IRC | 12:41 | |
| *** max_lobur_afk is now known as max_lobur | 12:45 | |
| openstackgerrit | A change was merged to openstack/python-tuskarclient: Fixes in do_flavor_delete() method https://review.openstack.org/62841 | 12:46 |
| *** jdob has joined #tripleo | 12:49 | |
| *** akuznetsov has quit IRC | 12:53 | |
| *** pblaho has joined #tripleo | 12:55 | |
| *** jcooley_ has joined #tripleo | 12:57 | |
| *** mrunge has quit IRC | 13:03 | |
| *** tzumainn has quit IRC | 13:05 | |
| *** tzumainn has joined #tripleo | 13:05 | |
| *** rushiagr has quit IRC | 13:09 | |
| *** jcooley_ has quit IRC | 13:27 | |
| *** boris-42 has quit IRC | 13:28 | |
| *** akuznetsov has joined #tripleo | 13:47 | |
| *** dprince has joined #tripleo | 13:50 | |
| *** gothicmindfood has joined #tripleo | 13:53 | |
| *** akuznetsov has quit IRC | 13:58 | |
| *** derekh has quit IRC | 14:00 | |
| *** lucasagomes has quit IRC | 14:06 | |
| *** lucasagomes has joined #tripleo | 14:06 | |
| *** lucasagomes has quit IRC | 14:07 | |
| marios | SpamapS: ping, still around? | 14:08 |
| *** lblanchard has joined #tripleo | 14:11 | |
| *** rushiagr has joined #tripleo | 14:14 | |
| *** jcooley_ has joined #tripleo | 14:18 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Remove management node UI https://review.openstack.org/63128 | 14:22 |
| *** rushiagr has quit IRC | 14:29 | |
| *** rushiagr has joined #tripleo | 14:29 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Remove management node UI https://review.openstack.org/63128 | 14:30 |
| *** julim has joined #tripleo | 14:30 | |
| *** jayg|g0n3 is now known as jayg | 14:41 | |
| *** boris-42 has joined #tripleo | 14:45 | |
| GheRivero | hi all | 14:46 |
| slagle | hi Ghe | 14:46 |
| *** openstack has joined #tripleo | 14:49 | |
| *** jcooley_ has quit IRC | 14:52 | |
| *** matty_dubs|gone is now known as matty_dubs | 14:59 | |
| *** julim has quit IRC | 15:03 | |
| openstackgerrit | Marios Andreou proposed a change to openstack/tripleo-image-elements: Manually add d2to1 to pypi-mirror element as temp fix https://review.openstack.org/63136 | 15:05 |
| *** julim has joined #tripleo | 15:07 | |
| *** blamar has joined #tripleo | 15:08 | |
| *** jergerber has joined #tripleo | 15:11 | |
| *** jtomasek has quit IRC | 15:12 | |
| *** mestery_ has joined #tripleo | 15:12 | |
| *** akrivoka has quit IRC | 15:16 | |
| *** mestery has quit IRC | 15:16 | |
| *** blamar has quit IRC | 15:16 | |
| *** blamar has joined #tripleo | 15:17 | |
| *** akrivoka has joined #tripleo | 15:18 | |
| *** mestery has joined #tripleo | 15:22 | |
| *** mestery_ has quit IRC | 15:25 | |
| *** akrivoka has quit IRC | 15:32 | |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Fix missing space in devstest_overcloud.sh https://review.openstack.org/63149 | 15:34 |
| *** vkozhukalov has quit IRC | 15:40 | |
| *** akrivoka has joined #tripleo | 15:41 | |
| *** rongze has joined #tripleo | 15:42 | |
| *** jcooley_ has joined #tripleo | 15:44 | |
| *** rongze_ has quit IRC | 15:45 | |
| *** gothicmindfood has quit IRC | 15:48 | |
| *** CaptTofu has joined #tripleo | 15:50 | |
| *** CaptTofu has quit IRC | 15:52 | |
| *** CaptTofu has joined #tripleo | 15:53 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Fix missing space in devstest_overcloud.sh https://review.openstack.org/63149 | 15:56 |
| SpamapS | marios: No I wasn't around then, but I have just sat down for the day (0756 for me in Los Angeles). | 15:56 |
| *** CaptTofu has quit IRC | 15:57 | |
| *** julim has quit IRC | 16:01 | |
| *** julim has joined #tripleo | 16:02 | |
| Ng | bah, no derek | 16:12 |
| *** slagle has quit IRC | 16:12 | |
| Ng | dammit | 16:12 |
| Ng | I was about to talk to slagle too | 16:12 |
| SpamapS | Ng: you can talk to me... but.. expect snark. A lot. | 16:13 |
| Ng | SpamapS: I was just going to follow up on the -1s on 63021 about gerrit support in source-repositories not working with offline mode | 16:14 |
| Ng | and I was going to point out that it only actually breaks offline mode if you've hardcoded a gerrit repo/ref into an element | 16:15 |
| Ng | overriding an element with DIB_REPOblah unconditionally disables caching | 16:15 |
| *** rongze has quit IRC | 16:15 | |
| *** jcooley_ has quit IRC | 16:16 | |
| *** rongze has joined #tripleo | 16:16 | |
| Ng | and at the risk of consuming local disk space, I was going to suggest that we just cache the overrides too | 16:16 |
| Ng | it seems to me that it would be particularly useful where you are overriding the repo, because you're probably doing some kind of dev/test iteration and shortening the loops has to be Good™ | 16:17 |
| *** pblaho has quit IRC | 16:18 | |
| *** sdake_ has joined #tripleo | 16:22 | |
| *** sdake_ has joined #tripleo | 16:22 | |
| *** rongze has quit IRC | 16:22 | |
| *** rongze has joined #tripleo | 16:22 | |
| *** tserong has quit IRC | 16:23 | |
| *** rdopieralski has quit IRC | 16:23 | |
| SpamapS | Ng: agreed that we should cache the overrides | 16:25 |
| SpamapS | Ng: perhaps we should also consider a way to do cache GC as well. | 16:26 |
| *** vipul-away is now known as vipul | 16:26 | |
| *** tserong has joined #tripleo | 16:27 | |
| *** jcooley_ has joined #tripleo | 16:33 | |
| *** rushiagr is now known as rushiagr_away | 16:39 | |
| *** jcoufal has quit IRC | 16:42 | |
| *** jcooley_ has quit IRC | 16:52 | |
| *** tzumainn has quit IRC | 16:52 | |
| *** AaronGr_Zzz is now known as AaronGr | 16:54 | |
| *** jtomasek has joined #tripleo | 16:55 | |
| *** jcooley_ has joined #tripleo | 16:55 | |
| *** jcooley_ has quit IRC | 16:58 | |
| *** tzumainn has joined #tripleo | 16:59 | |
| *** martyntaylor has quit IRC | 17:00 | |
| *** jcooley_ has joined #tripleo | 17:04 | |
| *** UtahDave has joined #tripleo | 17:07 | |
| *** jcooley_ has quit IRC | 17:09 | |
| *** UtahDave has quit IRC | 17:10 | |
| *** jcooley_ has joined #tripleo | 17:15 | |
| *** rushiagr_away is now known as rushiagr | 17:15 | |
| *** max_lobur is now known as max_lobur_afk | 17:21 | |
| *** jcooley_ has quit IRC | 17:22 | |
| *** jtomasek has quit IRC | 17:25 | |
| *** GheRivero has quit IRC | 17:28 | |
| *** marun has joined #tripleo | 17:30 | |
| SpamapS | lifeless: when you're around, is there some reason tripleo-cd is stopped? | 17:32 |
| * Ng bails a bit early to go and pick up some dinner | 17:39 | |
| Ng | bbl | 17:39 |
| *** rongze has quit IRC | 17:39 | |
| *** slagle has joined #tripleo | 17:39 | |
| *** rongze has joined #tripleo | 17:40 | |
| *** rpodolyaka1 has joined #tripleo | 17:41 | |
| *** slagle has quit IRC | 17:41 | |
| *** GheRivero has joined #tripleo | 17:41 | |
| *** slagle has joined #tripleo | 17:41 | |
| *** rongze has quit IRC | 17:44 | |
| *** vkozhukalov has joined #tripleo | 17:46 | |
| *** lblanchard has quit IRC | 18:09 | |
| *** akrivoka has quit IRC | 18:15 | |
| *** akuznetsov has joined #tripleo | 18:19 | |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/os-apply-config: Add netdevice value type https://review.openstack.org/63188 | 18:23 |
| *** rlandy has joined #tripleo | 18:23 | |
| *** CaptTofu has joined #tripleo | 18:33 | |
| *** ifarkas has quit IRC | 18:34 | |
| *** rongze has joined #tripleo | 18:40 | |
| *** rushiagr has quit IRC | 18:41 | |
| *** athomas has quit IRC | 18:42 | |
| *** rongze has quit IRC | 18:45 | |
| *** rlandy has quit IRC | 18:53 | |
| lifeless | SpamapS: I imagine it hit upstarts error threshold | 18:54 |
| lifeless | SpamapS: I have not stopped it | 18:54 |
| SpamapS | ok cool restarting | 18:55 |
| SpamapS | why is nova-conductor still logging at DEBUG level? HRM | 18:56 |
| SpamapS | oh | 18:56 |
| SpamapS | because logging.conf says to | 18:56 |
| *** AaronGr is now known as AaronGR_afk | 19:01 | |
| *** rpodolyaka1 has quit IRC | 19:04 | |
| lifeless | so how goes tripleo-cd | 19:13 |
| lifeless | SpamapS: what would be the most useful thing for me to do? | 19:16 |
| lifeless | Today is the last work day of the year for HP NZers | 19:16 |
| lifeless | and I have the 'office' lunch at 1230 - leave here in < 4 hours | 19:17 |
| *** akuznetsov has quit IRC | 19:20 | |
| *** jdob_ has joined #tripleo | 19:21 | |
| *** rpodolyaka1 has joined #tripleo | 19:21 | |
| pleia2 | lifeless: there has been chatter on https://review.openstack.org/#/c/63044/ - any thoughts? I am not thrilled with markmc's idea because I think -p UNDER_ is clunky (worse than -c), but what about james's note? | 19:23 |
| *** lsmola_ has quit IRC | 19:25 | |
| pleia2 | oh look, you're on it :) thank you | 19:25 |
| lifeless | pleia2: I think all the feedback is bikeshedding except the bit about removing the files ;) | 19:25 |
| pleia2 | hehe | 19:25 |
| lifeless | pleia2: I cam because you pinged me | 19:25 |
| pleia2 | ok | 19:25 |
| lifeless | s/cam/am/ | 19:25 |
| devananda | lifeless: re bug tracker thing, have you talked to / are you aware of krotscheck? | 19:27 |
| lifeless | devananda: the person? Yes, I think so. I don't have a bug tracker thing paged in though... | 19:28 |
| SpamapS | lifeless: let's see where tripleo-cd's latest run goes | 19:29 |
| devananda | lifeless: bug tracker thing we talked about >6mo ago | 19:29 |
| *** CaptTofu has quit IRC | 19:29 | |
| SpamapS | | notcompute | fcd29cd0-9247-465b-bec1-1204ffe9596a | state changed | UPDATE_IN_PROGRESS | 2013-12-19T19:17:19Z | | 19:29 |
| SpamapS | | 22244758-754c-40c4-9194-bc4e0ab2981f | overcloud-NovaCompute0-nn7gqrc3pkzv | REBUILD | rebuild_spawning | Running | ctlplane=10.10.16.174 | | 19:29 |
| devananda | lifeless: causes? | 19:30 |
| lifeless | devananda: oh right, yes. | 19:30 |
| lifeless | https://github.com/causes-tracker/designdocs/blob/master/Manifesto.md | 19:30 |
| devananda | yep | 19:30 |
| devananda | lifeless: he's working on storyboard. i'm wondering to what degree these things overlap | 19:31 |
| lifeless | I don't think they really do | 19:31 |
| lifeless | storyboard is an openstack specific task tracker with design points highly optimised for openstacks perceived needs | 19:32 |
| lifeless | causes is a ground up rethink about how one handles user and machine gathered data about software issues | 19:32 |
| devananda | lifeless: you two should talk at some point :) | 19:33 |
| lifeless | sure | 19:33 |
| lifeless | would be happy to | 19:33 |
| SpamapS | root 13765 15.4 0.0 19500 1888 ? D 19:33 0:11 \_ mkfs.ext3 -F -L ephemeral0 /dev/disk/by-path/ip-10.10.16.171:3260-iscsi-iqn-86776649-8996-4115-912c-d4548eb25d3c-lun-1-part1 | 19:34 |
| SpamapS | lifeless: perhaps we're not setting preserve ephemeral yet? :-P | 19:34 |
| lifeless | SpamapS: we're not | 19:35 |
| SpamapS | ok good :) | 19:35 |
| lifeless | SpamapS: until we get the id thing landed we won't be rebuilding servers anyhow | 19:35 |
| lifeless | SpamapS: are we? | 19:35 |
| SpamapS | ID thing landed | 19:35 |
| lifeless | ok | 19:35 |
| lifeless | so in stackrc | 19:35 |
| lifeless | export OVERCLOUD_IMAGE_UPDATE_POLICY=REBUILD_PRESERVE_EPHEMERAL | 19:35 |
| lifeless | and restart tripleo-cd | 19:35 |
| SpamapS | very surprised that Heat landed that before it landed in nova btw :) | 19:36 |
| SpamapS | (glad, but surprised) | 19:36 |
| SpamapS | lifeless: note that we don't have to restart tripleo-cd | 19:37 |
| SpamapS | lifeless: it sources /root/stackrc in each loop iteration | 19:37 |
| SpamapS | lifeless: I think I know why deploys have gone slower actually | 19:39 |
| SpamapS | mkfs.ext3 on the ephemeral | 19:39 |
| *** jdob_ has quit IRC | 19:40 | |
| *** jdob has quit IRC | 19:40 | |
| SpamapS | 537GB worth of ext3 over iscsi | 19:40 |
| *** jdob has joined #tripleo | 19:40 | |
| SpamapS | lifeless: ugh so we're going to run into a new problem now. Updates don't actually wait for wait conditions again. | 19:42 |
| SpamapS | of course, we're just waiting for nova-compute anyway so it will work that way | 19:45 |
| SpamapS | but I think it might fail because we'll only wait 5 minutes for that | 19:45 |
| *** lblanchard has joined #tripleo | 19:46 | |
| SpamapS | lifeless: good news is it looks like rebuild (without preserve ephemeral) does work | 19:46 |
| lifeless | SpamapS: SpamapS 537GB of writes to do mkfs ? | 19:47 |
| lifeless | SpamapS: we should do ext4 mkfs | 19:47 |
| SpamapS | lifeless: 537GB of total partition size | 19:47 |
| SpamapS | I'm sure a small percentage of that to write the FS | 19:47 |
| lifeless | SpamapS: 1500GB of partition size | 19:47 |
| SpamapS | but that is still non-trivial amount of extra data to write | 19:47 |
| lifeless | sorry, 1000GB | 19:48 |
| SpamapS | lifeless: oh hm maybe I was looking at the wrong partition table | 19:48 |
| lifeless | | 13d5adda-96a3-4e9c-9f56-58d8f132997a | baremetal | 98304 | 500 | 1000 | | 24 | 1.0 | True | | 19:48 |
| *** cd-undercloud has joined #tripleo | 19:48 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:48 |
| *** cd-undercloud has quit IRC | 19:48 | |
| lifeless | 500GB /, 1000GB /mnt according to the flavor definition | 19:48 |
| SpamapS | ok so yeah it timed out waiting for nova service-list to show a nova-compute | 19:48 |
| SpamapS | because we have no wait conditions on update | 19:49 |
| lifeless | ok | 19:49 |
| SpamapS | lifeless: the problem is that wait conditions just wait until they're in a state of "COMPLETE" | 19:49 |
| lifeless | I'll work on mkfs.ext4 | 19:49 |
| lifeless | if you tackle the wait conditions | 19:49 |
| SpamapS | lifeless: they reach that state when Count == SignalCount | 19:49 |
| SpamapS | and updates don't reset SignalCount | 19:49 |
| SpamapS | I think we probably need a new type of wait condition that does reset the signal count on update. | 19:50 |
| lifeless | SpamapS: or we need a random uuid per update? | 19:51 |
| lifeless | SpamapS: turn it into a parameter, for now? | 19:51 |
| SpamapS | lifeless: we can't use parameters to rename things, and there is no parameter for waitcondition that would accept a uuid. | 19:51 |
| SpamapS | oh hmm | 19:52 |
| SpamapS | Timeout | 19:52 |
| SpamapS | I think we talked about this | 19:52 |
| lifeless | we did | 19:52 |
| SpamapS | we can tweak Timeout up/down by 1 second and it will work | 19:52 |
| SpamapS | while I land a patch in Heat for an UpdateWaitCondition | 19:53 |
| SpamapS | which would be a starting point for a CanaryUpdateWaitCondition actually. :) | 19:53 |
| SpamapS | lifeless: are you like me where you'd rather not take days off but your family will make you pay attention to them and stuff? :-P | 19:54 |
| *** AaronGR_afk is now known as AaronGR | 19:55 | |
| *** sdake has quit IRC | 19:55 | |
| lifeless | SpamapS: well | 19:55 |
| lifeless | SpamapS: somewhat | 19:55 |
| lifeless | SpamapS: I have LCA papers to prep and things too | 19:56 |
| *** sdake has joined #tripleo | 19:56 | |
| *** sdake has joined #tripleo | 19:56 | |
| SpamapS | lifeless: ahh yes | 19:56 |
| lifeless | so it will be more do whatever I want days rather than 'off', if you get me | 19:56 |
| SpamapS | Yeah true | 19:56 |
| lifeless | wheee | 19:57 |
| lifeless | making a 1T ext3 fs takes a while | 19:57 |
| lifeless | with the default parameters used by nova | 19:57 |
| lifeless | this is on a 6 disk RAID6 array - 8K inode tables | 19:58 |
| lifeless | disk are going batty | 19:58 |
| SpamapS | we should consider XFS anyway ;) | 19:58 |
| lifeless | 30M/second | 19:58 |
| lifeless | utterly random IOs | 19:59 |
| lifeless | write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32768) = 32768 | 20:01 |
| lifeless | lseek(3, 795913125888, SEEK_SET) = 795913125888 | 20:01 |
| lifeless | wtf ext3, wtf | 20:01 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Consolidate under and overcloud passwd scripts https://review.openstack.org/63044 | 20:01 |
| SpamapS | lifeless: ok so timeout wobble won't work | 20:02 |
| SpamapS | lifeless: the state that is checked is in the handle.. | 20:02 |
| lifeless | SpamapS: your hunch was right | 20:02 |
| lifeless | SpamapS: this is writing G's | 20:02 |
| lifeless | 29839 be/4 root 0.00 B 2.34 G 0.00 % 98.06 % mkfs.ext3 -F /dev/lifelesswks-vg/test | 20:02 |
| SpamapS | lifeless: it sat there for many minutes so I knew it wasn't playing Canasta. :) | 20:02 |
| lifeless | thats only a partial run too | 20:02 |
| lifeless | trying with ext4 now | 20:03 |
| SpamapS | lifeless: for my own morbid curiosity can you try XFS too? | 20:03 |
| lifeless | whats xfs support like on (ubuntu, debian, RHEL, Suse) ? | 20:03 |
| SpamapS | AFAIK perfect | 20:03 |
| SpamapS | I've used it for every MySQL server I've built in production for years. | 20:03 |
| SpamapS | last time I had to modprobe it was probably 2003. | 20:03 |
| lifeless | so way upwards of 5G | 20:04 |
| SpamapS | lifeless: it is superior to ext4 for basically every server workload except bazillions of deletes. | 20:04 |
| lifeless | even if we did this locally | 20:04 |
| lifeless | it would be slow | 20:04 |
| lifeless | so xfs for local volumes under nova is fine too? | 20:04 |
| SpamapS | yes | 20:04 |
| SpamapS | probably better | 20:04 |
| lifeless | we could always use zfs | 20:05 |
| SpamapS | it handles concurrent writing particularly better than ext4 | 20:05 |
| SpamapS | z _IS_ after x | 20:05 |
| SpamapS | ergo: better | 20:05 |
| hewbrocca | (now you have two problems) | 20:05 |
| * lifeless successfully trolled | 20:05 | |
| lifeless | I wonder | 20:06 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-image-elements: Add tempest element https://review.openstack.org/50477 | 20:06 |
| hewbrocca | heh heh | 20:06 |
| lifeless | would image builds be faster and better if we did them with xfs - serious question | 20:06 |
| lifeless | we're making a new filesystem anyway | 20:06 |
| lifeless | and we keep hitting online resize bugs in ext4 | 20:06 |
| lifeless | dprince: ^ | 20:06 |
| SpamapS | oh | 20:07 |
| SpamapS | XFS has no online shrink | 20:07 |
| lifeless | SpamapS: we don't do an online shrink though; we visit in person | 20:07 |
| dprince | lifeless: I'm still hitting those too intermittently. | 20:07 |
| SpamapS | I don't know if the cloud-utils resize thing will resize XFS.. looking | 20:07 |
| dprince | lifeless: going to be a real time sync to troubleshoot that one I'm afraid... | 20:08 |
| lifeless | dprince: I know | 20:08 |
| lifeless | dprince: I'm pondering an end-run | 20:08 |
| dprince | lifeless: As a side note I just got a null-modem cable so I can run get better output (via minicom or whatever) from ttys0 when this happens. | 20:09 |
| dprince | lifeless: this only happens to me on one of my baremetal machines so it seems to be somewhat hardware specific | 20:09 |
| SpamapS | \o/ | 20:10 |
| SpamapS | cloud-init resizes xfs | 20:10 |
| SpamapS | dprince: it may be geometry specific in that case | 20:10 |
| lifeless | real 14m31.848s | 20:11 |
| SpamapS | cloud-init will also resize btrfs, if we want to find a better pain source than toothpicks under our toenails | 20:11 |
| lifeless | real 0m7.737s | 20:11 |
| lifeless | ext4 ^ | 20:11 |
| SpamapS | lifeless: yeah, extents ftw | 20:11 |
| lifeless | thought it would be way faster | 20:11 |
| lifeless | SpamapS: its not the extents | 20:11 |
| SpamapS | _more_ sparse superblocks? | 20:11 |
| dprince | SpamapS: This is a 1TB drive, so it could be. CHanging the root partision size seems to help... but since it is intermittent it is hard to say | 20:12 |
| lifeless | SpamapS: nah, lazy init inodes | 20:12 |
| SpamapS | lifeless: OH | 20:12 |
| lifeless | SpamapS: that 5GB of IO was all writing \0\0\0\0\0\0\0 to the inodes | 20:12 |
| SpamapS | lifeless: xfs has the distinct advantage that its inode tables can be modified post-mkfs | 20:13 |
| SpamapS | lifeless: particularly good idea for root FS images where we are just guessing as to what it might be grown to | 20:14 |
| lifeless | so, in the name of conservatism, i'm going to just toggled ext3 -> ext4 | 20:14 |
| lifeless | in ephemeral | 20:14 |
| SpamapS | lifeless: good news regarding ext4. Is that a config option we can make default in nova? | 20:14 |
| lifeless | I will be raising that discussion in just a second :) | 20:14 |
| SpamapS | cool | 20:14 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-incubator: Use pypi-openstack in devtest https://review.openstack.org/63206 | 20:15 |
| * bnemec finally got his environment stable enough to do that^ | 20:15 | |
| SpamapS | lifeless: ok so regarding the waitcondition thing.. without modifications to Heat we are kind of stuck right now. I think the right thing is to raise the service-list timeout on updates and then we need to start looking closely at stevebaker's new software deployer thing which may solve this. | 20:15 |
| *** Haomeng has quit IRC | 20:17 | |
| lifeless | SpamapS: is steve's thing in trunk? Can it be in trunk next [work] week ? | 20:17 |
| lifeless | SpamapS: if not, I think we need to do a Q&D patch like an option to the waitcondition to reset everything on update | 20:17 |
| *** Haomeng has joined #tripleo | 20:17 | |
| lifeless | because stevebaker's thing is, AIUI, part of a much larger architectural work | 20:17 |
| *** CaptTofu has joined #tripleo | 20:18 | |
| SpamapS | lifeless: closer look includes where it stands | 20:19 |
| SpamapS | lifeless: and agreed :) | 20:19 |
| *** cd-undercloud has joined #tripleo | 20:25 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:25 |
| *** cd-undercloud has quit IRC | 20:25 | |
| *** AaronGR is now known as AaronGR_lunch | 20:27 | |
| dprince | lifeless/SpamapS: in the HP CD cloud does the seed still act as the DHCP server for undercloud? | 20:28 |
| jog0 | rpodolyaka1: ping | 20:28 |
| lifeless | dprince: we haven't rebuilt things since we moved to neutron everywhere, so the answer is not the answer for other regions | 20:28 |
| lifeless | dprince: but in principle - the seed is the DHCP server for the undercloud control plane machine yes | 20:29 |
| lifeless | dprince: via Neutron | 20:29 |
| rpodolyaka1 | jog0: pong | 20:30 |
| dprince | lifeless: Right. Eventually we'll want a kill off the seed and just have the undercloud stand on its own right? | 20:30 |
| lifeless | dprince: yes, once we have HA setups we can work on that | 20:30 |
| jog0 | I have a question on https://review.openstack.org/#/c/59468/15/nova/compute/manager.py | 20:30 |
| rpodolyaka1 | sure | 20:30 |
| lifeless | dprince: the seed is also the metadata and heat server for the undercloud too, today. | 20:31 |
| jog0 | rpodolyaka1: hmm guess I didn't leave the comment in there | 20:31 |
| jog0 | rpodolyaka1: do we always want to do block device detach and re-attach? | 20:31 |
| dprince | lifeless: yep, got all that. Just wanted to make sure you guys had not done anything fancy (undocumented) to step off the seed yet. I'll follow a similar path with our RH CD cloud then. | 20:32 |
| rpodolyaka1 | jog0: good question | 20:32 |
| jog0 | if so we can pull that out of the driver side rebuild logic | 20:32 |
| lifeless | dprince: yeah, when teh time comes it will all go straight in trunk | 20:32 |
| lifeless | dprince: otherwise we can't expect a joint admin team to well, admin jointly | 20:32 |
| rpodolyaka1 | jog0: we could, but it depends on how rebuild is implemented in a particular driver when to do detach/attach | 20:33 |
| rpodolyaka1 | jog0: I mean, only driver knows that | 20:33 |
| rpodolyaka1 | jog0: e.g. for BM we don't do destroy() at all | 20:34 |
| rpodolyaka1 | jog0: and for libvirt we do destroy() + spawn() | 20:35 |
| rpodolyaka1 | jog0: so I believe it's one the thing the driver must be in control of | 20:35 |
| rpodolyaka1 | jog0: though, I agree, passing partials doesn't look nice :( | 20:35 |
| lifeless | rpodolyaka1: also, we should probably put up another step in the series to fix the ordering of detach/destroy as a separate thing | 20:36 |
| rpodolyaka1 | lifeless: yeah, completely forgot about this | 20:36 |
| jog0 | rpodolyaka1: why don't we detach attach? | 20:37 |
| jog0 | because we don't have any volume support anyway? | 20:37 |
| rpodolyaka1 | yeo | 20:37 |
| rpodolyaka1 | *yep | 20:37 |
| jog0 | rpodolyaka1: so don't we always want detach retach .. just in our case it is a noop for now | 20:38 |
| jog0 | lifeless: detach destroy order? | 20:38 |
| lifeless | arata is working on iscsi voodoo I think | 20:38 |
| lifeless | jog0: the existing rebuild code does this: | 20:38 |
| lifeless | detach_volumes | 20:38 |
| lifeless | driver.destroy | 20:38 |
| lifeless | poweroff does | 20:38 |
| lifeless | destroy | 20:39 |
| lifeless | detach | 20:39 |
| lifeless | I suspect rebuild is happily causing dirty filesystems everytime it's used. | 20:39 |
| lifeless | rpodolyaka1: ^ do I remember correctly? | 20:39 |
| rpodolyaka1 | lifeless: yes, I think so | 20:40 |
| * lifeless WATs at nova/virt/xenapi/vm_utils.py line 2227 | 20:40 | |
| jog0 | lifeless: ahh | 20:41 |
| jog0 | lifeless rpodolyaka1: so are you both fine with moving detach_block_devices out of driver.rebuild, and just pass in the reattach info | 20:44 |
| lifeless | no | 20:45 |
| lifeless | because if the order is meant to be | 20:45 |
| lifeless | destroy | 20:45 |
| lifeless | detach | 20:45 |
| lifeless | you can't move it out | 20:45 |
| jog0 | lifeless: ack | 20:45 |
| jog0 | lifeless: why should the order be that vs detach destroy | 20:46 |
| lifeless | not sure, revisiting it now | 20:46 |
| lifeless | ok, see _shutdown_instance | 20:46 |
| lifeless | destory | 20:46 |
| lifeless | detach | 20:46 |
| lifeless | jog0: oh, the order? | 20:47 |
| *** boris-42 has quit IRC | 20:47 | |
| lifeless | jog0: what happens if you detach a block device that's in use? | 20:47 |
| jog0 | lifeless: bad things right | 20:47 |
| jog0 | but does destroy allow for the volume to be cleaned up first | 20:48 |
| jog0 | I thought destroy was the equivalent of 'pulling the power cable' | 20:48 |
| lifeless | it calls libvirt's destroy method | 20:49 |
| lifeless | I'm fairly sure thats a little graceful | 20:49 |
| lifeless | as there is a timeout facility on it | 20:49 |
| jog0 | lifeless: I only recal that because there was a patch we reverted about graceful shutdown | 20:50 |
| lifeless | destroy destroy (stop) a domain | 20:50 |
| lifeless | thats the primary interface to libvirt to turn off a vm | 20:50 |
| lifeless | it is equivalent to pushing the power button AIUI | 20:50 |
| *** boris-42 has joined #tripleo | 20:50 | |
| lifeless | point is though | 20:50 |
| lifeless | rebuild does the two ops in the opposite direction to shutdown | 20:50 |
| lifeless | this is bonkers | 20:50 |
| *** vkozhukalov has quit IRC | 20:51 | |
| lifeless | regardless of gracefulness, it's almost certain to tickle bugs due to different expectations in driver writers | 20:51 |
| *** boris-42 has quit IRC | 20:52 | |
| *** epim has joined #tripleo | 20:53 | |
| jog0 | lifeless: I would file a bug on this and come back to this after our patch set lands | 20:53 |
| jog0 | as it doesn't touch us | 20:53 |
| jog0 | for our codepath | 20:53 |
| lifeless | jog0: ok | 20:54 |
| lifeless | jog0: it does inform the question you ask though about moving the detach outside | 20:55 |
| lifeless | rather not have churn we don't need | 20:55 |
| jog0 | lifeless: agreed | 20:55 |
| jog0 | I won't touch it | 20:55 |
| *** lblanchard has quit IRC | 21:00 | |
| *** cd-undercloud has joined #tripleo | 21:01 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 21:01 |
| *** cd-undercloud has quit IRC | 21:01 | |
| *** michchap_ has joined #tripleo | 21:04 | |
| *** CaptTofu has quit IRC | 21:05 | |
| *** michchap has quit IRC | 21:06 | |
| *** AaronGR_lunch is now known as AaronGR | 21:06 | |
| lifeless | I'm going to monkey patch in the ext3->4 change | 21:07 |
| jog0 | geting ready for a patch to the stack FYI | 21:14 |
| *** vipul is now known as vipul-away | 21:16 | |
| lifeless | restarted with ext4 patch | 21:19 |
| *** lifeless changes topic to "tripleo-cd running preserve-ephemeral WIP patches + ext4 patch | Using OpenStack to deploy OpenStack; meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 21:19 | |
| *** vipul-away is now known as vipul | 21:21 | |
| rpodolyaka1 | good night all | 21:34 |
| *** cody-somerville has joined #tripleo | 21:37 | |
| *** cody-somerville has joined #tripleo | 21:37 | |
| *** rpodolyaka1 has quit IRC | 21:39 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Add mock api calls to the UI https://review.openstack.org/62554 | 21:40 |
| *** jdob has quit IRC | 21:46 | |
| SpamapS | lifeless: | f6d61e10-31a9-4cfb-91f4-3dd8b5a0e0c9 | undercloud-undercloud-rznm7hvopr3g | BUILD | deleting | NOSTATE | ctlplane=192.0.2.2 | | 21:52 |
| SpamapS | lifeless: any idea how to get rid of that? | 21:52 |
| SpamapS | nova delete won't | 21:52 |
| lifeless | SpamapS: is that local? | 21:53 |
| SpamapS | lifeless: yeah I'm trying to test some heat patches out here in my dev env | 21:53 |
| lifeless | SpamapS: stop and start nova-compute | 21:53 |
| lifeless | then try again | 21:53 |
| SpamapS | lifeless: ahh good thats what I just did. :) | 21:53 |
| SpamapS | interesting.. nova resource replacement doesn't work on the seed node | 21:55 |
| SpamapS | because delete and subsequent boot doesn't leave enough time for nova to reclaim the resource | 21:55 |
| lifeless | seed only has one node attached | 21:55 |
| lifeless | there's a bug open on that | 21:55 |
| SpamapS | yeah | 21:55 |
| lifeless | PRESERVE_EPHEMERAL should work | 21:55 |
| SpamapS | so this is testing the "retry on failed update" | 21:55 |
| lifeless | well, you'll get lots of that ;) | 21:56 |
| *** ccrouch has joined #tripleo | 21:56 | |
| SpamapS | I think it would be fine if I had n+1 resources | 21:56 |
| SpamapS | and if I manually delete the node and let it be reclaimed before doing the retry that seems to be working | 21:57 |
| *** epim has quit IRC | 22:04 | |
| *** julim has quit IRC | 22:09 | |
| *** cwolferh has joined #tripleo | 22:17 | |
| *** jcooley_ has joined #tripleo | 22:19 | |
| *** cd-undercloud has joined #tripleo | 22:22 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 22:22 |
| *** cd-undercloud has quit IRC | 22:22 | |
| lifeless | -> lunch do | 22:30 |
| SpamapS | lifeless: have fun and merry christmas :) | 22:34 |
| *** matty_dubs is now known as matty_dubs|gone | 22:39 | |
| *** vipul is now known as vipul-away | 22:56 | |
| *** cd-undercloud has joined #tripleo | 23:25 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 23:25 |
| *** cd-undercloud has quit IRC | 23:25 | |
| *** rongze has joined #tripleo | 23:43 | |
| *** rongze has quit IRC | 23:48 | |
| *** CaptTofu has joined #tripleo | 23:52 | |
| *** CaptTofu has quit IRC | 23:54 | |
| *** gothicmindfood has joined #tripleo | 23:58 | |
| *** dprince has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!