Monday, 2014-02-17

*** jcooley_ has joined #tripleo00:07
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Shift NODE_{CPU,DISK,MEM} into the JSON data file
*** jcooley_ has quit IRC00:13
*** cd-undercloud has joined #tripleo00:21
cd-undercloud************** overcloud complete status=1 ************00:21
*** cd-undercloud has quit IRC00:21
*** matsuhashi has joined #tripleo00:23
*** jcooley_ has joined #tripleo00:30
*** e0ne_ has quit IRC00:31
*** matsuhashi has quit IRC00:33
*** matsuhashi has joined #tripleo00:34
*** jcooley_ has quit IRC00:37
*** matsuhashi has quit IRC00:38
*** tserong has quit IRC00:41
*** tserong has joined #tripleo00:44
*** weshay has quit IRC01:03
*** matsuhashi has joined #tripleo01:04
*** dkehn__ has joined #tripleo01:04
*** dkehn has quit IRC01:05
*** nosnos has joined #tripleo01:05
*** dkehn__ has quit IRC01:05
*** dkehn__ has joined #tripleo01:05
*** dkehn_ has quit IRC01:06
*** dkehn has joined #tripleo01:07
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Stop exporting in devtest_testenv
lifelessyay bandersnatch up to R01:09
*** dkehn__ is now known as dkehn_01:09
*** jcooley_ has joined #tripleo01:30
StevenKlifeless: So, re: apt-sources, we have now have DIB_DISTRIBUTION_MIRROR supported by the ubuntu element itself, and DIB_DEBIAN_MIRROR used by debian, as well as DIB_APT_SOURCES01:31
StevenKIn other words ":-("01:32
*** jcooley_ has quit IRC01:37
lifelessStevenK: right01:38
lifelessStevenK: so, apt-sources is the most general01:38
lifelessdebian should clearly use DIB_DISTRIBUTION_MIRROR01:38
StevenKI suppose apt-sources allows you to inject third-party sources01:41
StevenKWhich may or may not be intended01:41
lifelessdisable or enable universe01:41
lifelessmultiverse etc01:41
lifelesslike e.g. a vendor local repo01:41
*** cd-undercloud has joined #tripleo01:41
cd-undercloud************** overcloud complete status=1 ************01:41
*** cd-undercloud has quit IRC01:41
lifelessor a chef local repo01:41
*** e0ne has joined #tripleo01:42
StevenKlifeless: See 'third-party sources'01:43
lifelessfrom the perspective of someone doing that, they're first party ;)01:43
*** e0ne has quit IRC01:47
openstackgerritLee Li proposed a change to openstack/tuskar: Replace assertEqual(None, *) with assertIsNone in tests
*** yamahata__ has quit IRC01:49
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Shift debian element to DIB_DISTRIBUTION_MIRROR
StevenKlifeless: Given the above, I'll re-review the apt-sources reviews after lunch01:53
*** yamahata_ has joined #tripleo01:54
lifelessSpamapS: ideas on on expressing apache configurations via heat?01:55
*** ramishra has joined #tripleo01:55
*** ramishra has quit IRC01:55
*** ramishra has joined #tripleo01:56
SpamapSlifeless: don't do it? ;)01:58
lifelessSpamapS: ok, and more usefully?01:58
SpamapSlifeless: sorry what exactly do you want? Full structure -> apache generator, or just common patterns?01:58
lifelessSpamapS: specifically I want to export a debian mirror and a pypi mirror on a cinder volume01:58
lifelessthe apache2 element is very limited today.01:59
SpamapSlifeless: Heat will give you the volume ID and its device mapping... I'd use that to mount it in a known place and just have a static apache config.01:59
SpamapSperhaps allow customizing the Location02:00
lifelessdoes cloud-init automate mounting of volumes?02:00
SpamapSIt can, but Heat makes it hard to put cloud-config syntax in.02:01
SpamapSlifeless: so you'll have a VolumeAttachment resource to reference, and I believe you'll have the offset (like block-device0) to dig out in the ec2 metadata block-device-mapping..02:02
SpamapSlifeless: ^^02:03
lifelessSpamapS: so the thing in my head is that we manually make the volume, so that when we heat stack-delete we don't have to lose the data :P02:03
lifelessfor now02:03
SpamapSlifeless: so /dev/disk/by-id/virtio-<VolumeId>02:03
lifelesslsblk | grep VolumeId ?02:04
SpamapSlifeless: same thing02:04
SpamapSlifeless: the volume_id is passed in as a property of VolumeAttachment02:04
lifelessso; tl;dr - static file in the element with the vhost in it02:05
lifelesselements for the mirror etc02:05
lifelessroll the http element + the mirror elements into one image and booyah02:05
lifelessits a backburner thing, not actively on it02:06
lifelessbut it would likely make test runs substantially faster02:06
SpamapSlifeless: cool02:06
lifelessStevenK: failed tess02:08
StevenKlifeless: Yes, I saw. But the toci change to add the keys to the json landed02:13
*** matsuhashi has quit IRC02:18
lifelessStevenK: oh!02:19
lifelessStevenK: so ok, this is something, but I actually had a rather more comprehensive refactoring in mind :)02:19
openstackgerritRichard Su proposed a change to openstack/tripleo-incubator: Messaging backend selection
openstackgerritRichard Su proposed a change to openstack/tripleo-incubator: Separate OVERCLOUD_DIB_EXTRA_ARGS into two new variables
lifelessStevenK: rather than redeploying the testenvironments right now, how about going straight onto the bigger thing ? [your call]02:20
StevenKlifeless: The nodes: [] thing?02:20
SpamapSlifeless: have we put the cd cloud entirely on the backburner at this point?02:21
lifelessSpamapS: nope02:21
SpamapSI feel like I'm the only one who cares about it anymore.02:21
lifelessSpamapS: I am all about it02:21
lifelessI'm logged in right now poking around02:21
openstackgerritRichard Su proposed a change to openstack/tripleo-incubator: Messaging backend selection
StevenKlifeless: But we'll need both?02:21
lifelessSpamapS: I'm providing care and feeding for the folk that can't ssh into it to debug because they haven't earnt admin access yet02:21
SpamapSlifeless: I mean tripleo-cd .. not all the other weird stuff02:22
lifelessSpamapS: root@undercloud-notcompute-jws3awlsb2kh:~#02:22
lifelessSpamapS: ci-overcloud only exists because we still have downtime on cd-overcloud.02:22
SpamapSthe ci thing.. I really want that to work, but I feel like it has taken all of the focus away from "having a working cloud"02:22
SpamapSlifeless: have we got working swift yet? Once Greg gets Galera working, we'll have to get an HA glance backend up.02:23
lifelessStevenK: can you enlarge please?02:23
lifelessSpamapS: yes, we use swift for images already by default02:24
SpamapSassuming Rabbit works too .. :)02:24
SpamapSoh thats good ok02:24
lifelessremember I was debugging that02:24
lifelessSpamapS: so, re: focus.02:24
lifelessSpamapS: to a degree, yes. But...02:24
lifelessSpamapS: the CI folk were doing CI previously anyhow, in the little RH lab toci stuff02:24
lifelessSpamapS: what we've done is brought them so they are working on the elements that make up cd-overcloud02:25
lifelessSpamapS: which is a huge step into common effort.02:25
lifelessSpamapS: there is still a layer that cd-overcloud's deployment and use-as-a-user doesn't need that CI does.02:25
lifelessSpamapS: its a bit of a balance02:25
*** jcooley_ has joined #tripleo02:26
lifelessStevenK: the reason I'm suggesting StevenK get this refactoring done isn't for CI, but for bare metal deployers02:26
SpamapSI love the balance. A CD system without automated tests isn't really a CD system I want to use.. ;)02:26
*** matsuhashi has joined #tripleo02:26
SpamapSI just think the current tripleo-cd cloud is basically forgotten.02:26
lifelessSpamapS: who need to be able to describe their hardware to the system02:26
SpamapSUnless I log in and look at errors and attack them.02:26
lifelessSpamapS: or I do..02:26
lifelessSpamapS: I seem to recall monkeying around with mysql and tokens a tonne last week02:27
lifelessSpamapS: to fix cd-overcloud deploys02:27
SpamapSI am starting to think that focusing on a running cloud, before focusing on testing for it, seems cart before horse.02:27
lifelessSpamapS: I like to think of it as cart beside horse02:27
SpamapSwe have many carts and horses.. :-P02:27
SpamapSoh I think we might have a status=0 right now02:28
lifelesscreate-complete anyhow02:28
SpamapSbut.. we've yet to pass an update in a while.02:28
SpamapSlifeless: anyway I'm just making sure I'm focusing my non-heat-get-things-landed time on the right things.02:29
lifelessSpamapS: HA updates is the critical thing I believe02:29
lifelessSpamapS: btw I asked wendar to do a favor for me02:29
lifelessSpamapS: for an HP internal firedrill-priority security review02:29
SpamapSstevebaker seems to think he'll have his hot-software-config ready for our consumption the next few days or so..02:29
SpamapSand he also thinks we'll be able to use that for specifying graceful updating rather than a new field on the resource root.02:30
stevebakerthats the plan02:30
lifelessthen we just need to rewrite to use it :)02:30
SpamapSIf we can't manage that pipeline in time..02:30
lifelessusers are being created02:30
lifelessso yes, cloud up02:30
SpamapSI have already made a POC graceful update that is just a subclass of server02:30
lifelesswhich is great, because I'm now going to disable tripleo-cd so I can really poke at live migration02:31
SpamapSand takes a property which is the structure in my etherpad.02:31
lifelessI got some infos from HP's Nova folk02:31
SpamapSanyway.. should go do some stuff at home02:31
*** jcooley_ has quit IRC02:31
*** cd-undercloud has joined #tripleo02:32
cd-undercloud************** overcloud complete status=0 ************02:32
*** cd-undercloud has quit IRC02:32
*** lifeless changes topic to "/win 48"02:32
greghaynesnot sure we can handle that much win as a topic02:32
lifelesstripleo-cd stopped for live migration tests | tripleo-cd running preserve-ephemeral WIP patches and | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt02:33
lifeless| d0738177-1b69-4cac-9e23-95242e216315 | live-migration-test | ERROR  | None       | NOSTATE     | default-net= |02:36
lifelessle cloud le fail02:36
lifeless|                                      |     _("Unexpected vif_type=%s") % vif_type)02:38
lifelessnovacompute0 neutron agent hasn't checked in.02:39
lifelessSpamapS: ^ remember what I said :P02:39
*** StevenK changes topic to "tripleo-cd stopped for live migration tests | tripleo-cd running preserve-ephemeral WIP patches and | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"02:39
lifeless2014-02-16 21:53:53.174 2633 CRITICAL neutron [-] Unexpected boolean value ''02:41
lifeless2014-02-16 21:53:53.399 2754 CRITICAL neutron [-] Unexpected boolean value ''02:41
lifelessin the ovs log02:41
lifelessdate again02:45
lifeless4.5 hours out02:46
lifelessthat may explain a bunch of the fails02:46
greghaynesre the DIB_APT_SOURCES and friends: although the apt souces is most general, it seems like it could lead to pain if two elements wanted to depend on it to add their own vendor repo02:47
lifelessgreghaynes: its for end users.02:48
lifelessgreghaynes: elements adding a vendor repo should add a dedicated list file.02:48
greghaynesmakes sense02:48
lifelessoh man02:49
lifelessI do have to replace the entuire upstart service file for libvirt02:49
lifelessbecause the 'will be deprecated soon' actually means 'does not work as of now'02:49
greghaynesso really the only use case I can think of is enabling distribution components02:49
greghaynesand disabling02:49
lifelessgreghaynes: and local repos02:50
lifelessgreghaynes: which you can admittedly write a custom element for02:50
greghaynesah, I get what youre saying re the end user thing now :)02:50
* lifeless whinges in public02:53
lifelessSpamapS: another thing we need02:55
lifelessSpamapS: a mapping of all the hypervisors to the hosts files for the hypervisors02:55
lifelessoh man03:00
lifelessI still can't get nova keypair-show working03:01
lifelessnova keypair-show default03:01
lifelessERROR: can't set attribute03:01
*** jcooley_ has joined #tripleo03:01
lifelessruh roh03:02
lifeless2014-02-17 03:02:39,353 -[WARNING]: Calling '' failed [18/120s]: bad status code [500]03:02
lifelessthats not good03:02
lifelessoh yay03:04
lifelessmetadata agent barfing all over03:04
lifelessI will bet large amounts of money its an unreleased version of a client library03:05
lifelesshuh, no03:06
lifelessnot keystoneclient anyhow03:06
lifelesswhich is what I was expecting03:06
lifelessfixed in trunk03:09
lifelessnot released03:09
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Shift debian element to DIB_DISTRIBUTION_MIRROR
lifelessSpamapS: around ?03:25
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Workaround broken Neutron requirements.
lifelessSpamapS: if you get a timeslice ^ is needed to unfuck the neutron metadata agent on cd-overcloud03:26
StevenKlifeless: ... why not ?03:26
lifelessyeah just fixing that03:26
lifelessas to why, because I resurrected an old file03:27
StevenKlifeless: Not that it means much, but +1 from me when I see a patchset with git.o.o03:27
SpamapSlifeless: it's almost like Neutron isn't actually used by any CD shops. :-/03:27
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Workaround broken Neutron requirements.
lifelessSpamapS: ^ fixed version03:28
StevenKSpamapS: Perhaps they're still using nova-network03:29
lifelessok, vm up03:29
lifelesslets see, whats else on the checklist to test this?03:29
lifelessoh right, go pick up Cynthia03:29
lifelessI'm going to leave tripleo-cd stopped, I will poke at this more in a few03:29
lifelessSpamapS: remember you can +2 that03:29
lifelessSpamapS: as you +m = 2 reviewers and its a firedrill03:30
*** ENDERSON has joined #tripleo03:37
*** ENDERSON has left #tripleo03:38
*** dkehn__ has joined #tripleo03:39
*** dkehn_ has quit IRC03:41
*** dkehn has quit IRC03:41
*** dkehn has joined #tripleo03:42
*** matsuhashi has quit IRC03:48
*** jcooley_ has quit IRC03:53
*** ccrouch has joined #tripleo04:15
*** dkehn__ is now known as dkehn_04:16
*** AaronGr_Zzz is now known as AaronGr04:22
*** AaronGr is now known as AaronGr_Zzz04:23
*** matsuhashi has joined #tripleo04:32
*** noslzzp has quit IRC04:35
*** Qiming has joined #tripleo04:39
*** edmund has quit IRC04:45
*** ramishra has quit IRC04:53
*** ramishra has joined #tripleo05:04
*** AaronGr_Zzz is now known as AaronGr05:22
*** AaronGr is now known as AaronGr_Zzz05:23
*** killer_prince has quit IRC05:35
*** killer_prince has joined #tripleo05:39
*** rcarrillocruz has joined #tripleo06:03
tchaypothat awkward moment when HR asks you to submit paperwork within 14 days to authorise something that's happening in ~6.5 days.06:04
*** ramishra has quit IRC06:06
*** rcarrillocruz has quit IRC06:06
*** rcarrillocruz has joined #tripleo06:09
*** saju_m has joined #tripleo06:15
*** rcarrillocruz1 has joined #tripleo06:21
*** rcarrillocruz has quit IRC06:23
*** rcarrillocruz1 has quit IRC06:24
*** rcarrillocruz has joined #tripleo06:26
*** akuznetsov has quit IRC06:26
*** akuznetsov has joined #tripleo06:29
*** rcarrillocruz has quit IRC06:31
openstackgerritYongli He proposed a change to openstack/tuskar: Use six.moves cStringIO instead of cStringIO
*** rcarrillocruz has joined #tripleo06:38
*** nosnos has quit IRC06:42
lifelesstchaypo: lol06:45
*** nosnos has joined #tripleo06:52
*** rlandy has joined #tripleo07:04
*** mrunge has joined #tripleo07:18
openstackgerritYongli He proposed a change to openstack/tuskar: Use six.moves cStringIO instead of cStringIO
marioslifeless: ping. hi, i see you picked up one of my neutron reviews @
marioslifeless: i was away all last week, so just getting through email now07:21
*** rcarrillocruz has quit IRC07:26
*** ifarkas has joined #tripleo07:30
lifelessmarios: I fixed it up enough to work07:35
lifelessmarios: didn't do any of the review feedback though07:35
lifelessmarios: welcome back :)07:35
marioslifeless: hehe :) thanks, it *was* working as far as i recall, didn't get to it yet though. is there something i can/need to do to it today? or just waitin on reviews now?07:36
lifelessmarios: there was a -1 on it07:37
lifelessmarios: needs fixing style stuff07:37
lifelessmarios: and no, it wasn't working when you pushed it, it just didn't crash - totally different ;)07:37
lifelessmarios: I wrote a client patch too07:37
*** boris-42 has quit IRC07:40
marioslifeless: haha, well if its isn't crashing ... :) ok thanks, will get to it in a bit07:40
lifelessmarios: I'd live it if you pick that up, just rebase it off my other patch onto trunk and add co-authored-by you ;)07:41
lifelessmarios: in you can see the missing code07:42
lifelessmarios: that I added07:42
lifelesss/live it/love it/07:43
marioslifeless: i see - so i wasn't creating the new pools correctly07:46
marioslifeless: i wonder if i should add more tests then cos to be sure I didn't really stress this code too much when i was testing07:47
marioslifeless: (i mean to exercise the code you added )07:47
marioslifeless: i will add that client review to my list07:47
*** jprovazn has joined #tripleo07:47
*** akuznetsov has quit IRC07:48
*** dko has joined #tripleo07:49
*** akuznetsov has joined #tripleo07:51
*** jtomasek has joined #tripleo07:51
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Wrong params passed to tempalte create
*** bauzas has joined #tripleo08:22
*** jcoufal has joined #tripleo08:26
*** jcoufal has quit IRC08:28
*** jcoufal has joined #tripleo08:29
*** saju_m has quit IRC08:31
lifelessmarios: yes, you should :)08:34
lifelessmarios: yeah, the pool has a separate object tracking free slots, which wasn't filled out08:35
*** dtantsur has joined #tripleo08:35
*** rpodolyaka has joined #tripleo08:37
rpodolyakamorning tripleo08:37
*** akrivoka has joined #tripleo08:38
*** hashar has joined #tripleo08:43
*** hashar has quit IRC08:44
*** hashar has joined #tripleo08:46
*** kwwii has joined #tripleo08:49
*** jtomasek has quit IRC08:52
*** jtomasek has joined #tripleo08:52
lifelessifarkas: hi - on there was a failed jenkins test08:55
lifelessifarkas: please don't approve such things!08:56
ifarkaslifeless, ah, okay, sry. I should start with emails in the morning I guess :-)08:57
ifarkastaking back my approved vote08:57
*** saju_m has joined #tripleo08:58
jprovaznlifeless: so if there is a jenkins failure on patches prior your mail (e.g. should I resubmit the patch to enfore jenkins re-run?08:59
*** jistr has joined #tripleo09:01
lifelessjprovazn: recheck no bug09:01
lifelessjprovazn: or a rebase, either will trigger a new check09:01
*** e0ne has joined #tripleo09:02
ifarkaslifeless, should I revoke my +A from tuskar patches too? eg.
ifarkasI guess they have nothing to do with the jenkins failure09:02
lifelessifarkas: we're not testing tuskar [yet] so less worried about interactions with the new CI stuff there09:02
ifarkascool. that was what I thought09:03
lifelesshowever we really should be :)09:04
lifelesslike, there should be a job that bootstraps via devtest_seed and then runs tuskar API calls to deploy an overcloud09:04
ifarkaslifeless, yeah, that would be cool. lsmola pushed a patch  last Friday which allows tuskar to deploy an overcloud. A bunch of stuff still missing but it is slowly getting there...09:06
lifelessoh great, apparmor breaks live migrate. Yay.09:07
lifelessifarkas: so add it :)09:07
ifarkassure ;-)09:07
* lifeless is trolling ever so slightly09:07
lifelessI know its a ramp-up09:07
lsmolaifarkas: lifeless yeas we are getting there :-)09:10
*** e0ne has quit IRC09:11
*** e0ne has joined #tripleo09:13
*** derekh has joined #tripleo09:14
derekhlifeless: on check-tripleo-overcloud-precise if we're to treat that as a vote, do you know if its possible to up the timeout a little ?09:17
derekhthe time it takes to run the tests is pretty much the same as the time it takes for jenkins to time it out09:18
lifelessderekh: yes09:18
*** lucasagomes has joined #tripleo09:18
lifelessderekh: patch to infra09:18
derekhlifeless: ok, cool09:18
lifelessderekh: or we work on pre-heated cloud-before's.09:18
derekhlifeless: yup, that would work too :-)09:19
lifelessor both!09:20
derekhlifeless: so as I see it we have 3 options, for the pre-heated clouds, I'll write them down somewhere and you can comment or add a 4th option etc..09:21
lifelesstripleo-parallel-tests ?09:23
lifelessI'll add a card09:23
lifelessderekh: ^ has a etherpad url in it09:24
derekhlifeless: ok, will start to populate it09:24
*** mrunge has quit IRC09:43
openstackgerritDmitry Tantsur proposed a change to openstack/tuskar-ui: Add list/create/delete operations for node profiles
*** boris-42_ has joined #tripleo09:52
*** bogdando has joined #tripleo09:53
*** Qiming has quit IRC09:53
*** matsuhashi has quit IRC09:54
*** markmc has joined #tripleo09:55
*** lazy_prince has joined #tripleo09:57
*** killer_prince has quit IRC09:58
*** lazy_prince is now known as killer_prince09:58
*** athomas has joined #tripleo10:02
*** matsuhashi has joined #tripleo10:02
*** akuznetsov has quit IRC10:08
*** pbelanyi has joined #tripleo10:09
*** akuznetsov has joined #tripleo10:09
*** juice has quit IRC10:12
*** cd-undercloud has joined #tripleo10:13
cd-undercloud************** overcloud complete status=0 ************10:13
*** cd-undercloud has quit IRC10:13
*** juice has joined #tripleo10:15
*** matsuhashi has quit IRC10:17
*** matsuhashi has joined #tripleo10:17
*** matsuhashi has quit IRC10:22
*** gcha has joined #tripleo10:22
*** e0ne_ has joined #tripleo10:29
*** panda_ has joined #tripleo10:32
*** e0ne has quit IRC10:33
*** panda has quit IRC10:34
*** mrunge has joined #tripleo10:37
*** ramishra has joined #tripleo10:38
*** martyntaylor has joined #tripleo10:45
*** vkozhukalov has joined #tripleo10:46
*** matsuhashi has joined #tripleo10:47
*** al has quit IRC10:55
*** al has joined #tripleo10:56
*** ramishra has quit IRC11:08
*** ramishra has joined #tripleo11:09
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Passing real params to TuskarAPI
*** athomas has quit IRC11:34
*** cd-undercloud has joined #tripleo11:35
cd-undercloud************** overcloud complete status=1 ************11:35
*** cd-undercloud has quit IRC11:35
*** e0ne has joined #tripleo11:37
*** matsuhashi has quit IRC11:37
*** e0ne_ has quit IRC11:41
*** athomas has joined #tripleo11:44
*** gcha has quit IRC11:47
openstackgerritA change was merged to openstack/tripleo-image-elements: Update keystone config to use external paste.ini
*** CaptTofu has joined #tripleo11:59
*** killer_prince has quit IRC12:01
*** jang1 has joined #tripleo12:16
*** jang1 has left #tripleo12:17
*** jang1 has joined #tripleo12:18
*** e0ne has quit IRC12:18
*** e0ne_ has joined #tripleo12:18
*** e0ne_ has quit IRC12:26
*** e0ne has joined #tripleo12:26
*** e0ne_ has joined #tripleo12:28
*** e0ne has quit IRC12:28
*** lucasagomes is now known as lucas-hungry12:32
*** e0ne_ has quit IRC12:41
*** e0ne has joined #tripleo12:42
*** gcha has joined #tripleo12:43
*** weshay has joined #tripleo12:45
*** cd-undercloud has joined #tripleo12:57
cd-undercloud************** overcloud complete status=1 ************12:57
*** cd-undercloud has quit IRC12:57
*** jcoufal has quit IRC12:58
*** jcoufal has joined #tripleo12:59
*** noslzzp has joined #tripleo13:00
*** nosnos has quit IRC13:08
*** CaptTofu has quit IRC13:13
*** mrunge has quit IRC13:16
lsmoladerekh: hello13:22
lsmoladerekh: trying latest master, there is again a problem with ssh to seed13:22
lsmoladerekh: but reverting to your patch, it works13:22
lsmoladerekh: I am hallucinating? :-D13:22
openstackgerritStuart McLaren proposed a change to openstack/tripleo-image-elements: Glance: make 'workers' configurable
*** jcoufal has quit IRC13:29
*** hashar has quit IRC13:32
*** jcoufal has joined #tripleo13:36
derekhlsmola: hadn't noticed any problems, I'll kick of a clean run now and see13:38
derekhlsmola: kicked it off, I'll get back to you in a bit13:39
marioslsmola: have you seen this at all?
marioslsmola: trying a post http://localhost:8585/v1/overclouds/13:41
*** jdob has joined #tripleo13:41
*** lucas-hungry is now known as lucasagomes13:42
derekhso ci-overcloud controller  appears to be busted, can't ssh to it from the undercloud node, logging onto the console I see ssh packets hitting it and it sends out response but response doesn't get back to the controller, any ideas ...13:43
*** dprince has joined #tripleo13:43
*** d0ugal has quit IRC13:45
lsmolamarios: :-o13:45
lsmolamarios: please apply the one small patch from me in tuskar13:46
derekhproblem appears to have started around 2:47AM utc13:46
lsmolamarios: I am woking on proper fix13:46
marioslsmola: get same for python tools/initial_data.py13:46
marioslsmola: ok, going to fetch it now13:46
lsmolamarios: ok, then you have some db problem13:46
*** d0ugal has joined #tripleo13:47
lsmolamarios: the initial data should work13:47
marioslsmola: gives me 404 strangely enough
*** hashar has joined #tripleo13:49
derekhNg: SpamapS lifeless any ideas if anything was happening on the rack at that time ^^?13:49
lsmolamarios: :-O13:50
lsmolamarios: can you check it your db is ok?13:50
lsmolamarios: liek you have the schema, etc.13:50
marioslsmola: yah am trying again13:51
mariosdbsync et13:51
*** funzo has joined #tripleo13:52
lsmolamarios: ok13:53
jprovaznderekh: routing?13:53
openstackgerritStuart McLaren proposed a change to openstack/tripleo-image-elements: Glance: Add 'db' to
*** kwwii has quit IRC13:56
marioslsmola: does your fix
derekhjprovazn: seems like it but I think routes on the box are ok13:56
*** rbrady has joined #tripleo13:56
lsmolamarios: yes13:56
*** funzo has quit IRC13:58
*** CaptTofu has joined #tripleo13:59
*** ramishra_ has joined #tripleo14:01
marioslsmola: gives me  400. but strange thing is when i get v1/overclouds/template_get > OUT.txt and heat-template-validate OUT.txt it gives True14:01
*** ramishra has quit IRC14:02
*** pbelanyi has quit IRC14:04
*** ramishra_ has quit IRC14:06
*** julim has joined #tripleo14:09
*** lblanchard has joined #tripleo14:13
lsmolamarios: hm, this is after applying the patch and restarting API?14:14
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Passing correct count to template creation
*** funzo has joined #tripleo14:15
*** ramishra has joined #tripleo14:15
*** saju_m has quit IRC14:15
*** cd-undercloud has joined #tripleo14:18
cd-undercloud************** overcloud complete status=1 ************14:18
*** cd-undercloud has quit IRC14:18
marioslsmola: yeah, i put in a pdb.set_trace and trying to see why14:18
*** jdob has quit IRC14:19
lsmolamarios: this seems like there is still some parameter that is going to template14:19
derekhlsmola: Waiting for the undercloud stack to be ready14:19
*** jdob has joined #tripleo14:19
derekhlsmola: so got paste to point where your hitting problems14:19
lsmoladerekh: hm, ok14:19
lsmoladerekh: thanks, I am going to rebuild everything and try again14:20
derekhlsmola: is it possible the premissions on your private key are wrong ?14:21
*** jeremydei has quit IRC14:24
*** jpeeler has joined #tripleo14:24
*** jpeeler has joined #tripleo14:24
*** jeremydei has joined #tripleo14:24
*** ramishra has quit IRC14:31
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Passing correct count to template creation
marioslsmola: i think may be my undercloud setup14:36
*** jdob_ has joined #tripleo14:37
*** jdob has quit IRC14:37
*** blamar has joined #tripleo14:38
*** jdob_ is now known as jdob14:38
*** blamar has quit IRC14:40
*** blamar has joined #tripleo14:40
*** dkehn__ has joined #tripleo14:41
*** dkehn has quit IRC14:43
*** dkehn_ has quit IRC14:44
*** dkehn has joined #tripleo14:46
*** dkehn__ is now known as dkehn_14:46
*** matty_dubs|gone is now known as matty_dubs14:47
*** CaptTofu has quit IRC14:52
*** killer_prince has joined #tripleo14:52
*** noslzzp has quit IRC14:57
* marios biab15:01
*** julim_ has joined #tripleo15:02
*** julim has quit IRC15:05
*** rwsu has joined #tripleo15:09
*** cd-undercloud has joined #tripleo15:11
cd-undercloud************** overcloud complete status=1 ************15:11
*** cd-undercloud has quit IRC15:11
*** noslzzp has joined #tripleo15:26
*** coolsvap has joined #tripleo15:26
*** CaptTofu has joined #tripleo15:31
*** derekh changes topic to "Firedrill : CI down, Approve patches as usual | tripleo-cd stopped for live migration tests | tripleo-cd running preserve-ephemeral WIP patches and | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"15:31
*** ramishra has joined #tripleo15:32
*** beekneemech is now known as bnemec15:34
*** AaronGr_Zzz is now known as AaronGr15:35
*** ramishra has quit IRC15:37
*** e0ne has quit IRC15:38
*** ramishra has joined #tripleo15:44
*** edmund has joined #tripleo15:45
openstackgerritAna Krivokapic proposed a change to openstack/tuskar: Fix typos and formatting in documentation
*** e0ne has joined #tripleo15:47
*** hashar has quit IRC15:54
*** hashar has joined #tripleo15:54
*** ramishra has quit IRC15:55
*** jprovazn has quit IRC15:56
fungiderekh: i see your /topic and recent discussion... is it safe to say that is out of commission for an indeterminate amount of time, starting around 00:00 utc today?15:58
*** e0ne_ has joined #tripleo15:59
derekhfungi: yup, I had estimated the time around 2:50 utc15:59
derekhfungi: but maybe earlier15:59
fungilooks like maybe started by 01:0816:00
fungiyou have a tripleo-incubator change waiting for a check job worker assignment for 14 hours 53 minutes at this point16:01
*** boris-42_ has quit IRC16:01
derekhfungi: ok16:01
fungicool, just making sure this is a known issue and no eta for resolution yet, so we know what to expect from the nodepool/zuul end of things. thanks again!16:02
*** e0ne has quit IRC16:02
derekhfungi: no prob, trying to figure out the problem but no ETA will update once I know more16:02
*** dko has quit IRC16:11
*** CaptTofu has quit IRC16:14
*** CaptTofu has joined #tripleo16:23
*** CaptTofu has quit IRC16:23
*** cd-undercloud has joined #tripleo16:32
cd-undercloud************** overcloud complete status=1 ************16:32
*** cd-undercloud has quit IRC16:32
*** jcoufal has quit IRC16:34
*** dtantsur has quit IRC16:35
*** jistr has quit IRC16:35
*** bauzas has quit IRC16:38
mordredhow are you guys installing pip for your pip based things in dib?16:42
mordredon a quick scan, it appears you may be using pip from distro packages?16:42
mordredand it looks like it gets installed via os-config-applier elements?16:44
*** lazy_prince has joined #tripleo16:44
mordredoh - I found the install_packages python-pip lines16:45
mordredI'm going to suggest that you do not want to do that16:45
mordredI'm going to STRONGLY suggest that16:45
mordredusing distro pip is always the wrong choice16:46
*** e0ne has joined #tripleo16:48
*** e0ne_ has quit IRC16:50
*** jp_at_hp has joined #tripleo16:51
jp_at_hpNobodyCam: or SpamapS,  I'm looking at the environment override variables for the source-repositories element, and am concerned that a lot of the names have a "-" in them...16:53
*** gcha has quit IRC16:54
*** lazy_prince has quit IRC16:55
* NobodyCam takes look16:55
jp_at_hpI can see the bash variable substitution to replace - with _ in the source_repositories element16:57
NobodyCamlooks like /98-source-repositories handles that16:57
NobodyCam:-p ya16:57
*** vkozhukalov has quit IRC17:00
*** rcarrillocruz has joined #tripleo17:05
*** hashar has quit IRC17:09
*** bauzas has joined #tripleo17:11
*** martyntaylor has left #tripleo17:11
*** lazy_prince has joined #tripleo17:12
*** e0ne has quit IRC17:16
openstackgerritA change was merged to openstack/tuskar-ui: Add list/create/delete operations for node profiles
*** cwolferh has joined #tripleo17:18
openstackgerritA change was merged to openstack/tuskar-ui: Fixing the nodes table
*** akuznetsov has quit IRC17:26
*** matty_dubs is now known as matty_dubs|lunch17:39
*** boris-42_ has joined #tripleo17:40
*** marun has joined #tripleo17:41
*** akuznetsov has joined #tripleo17:42
*** akuznetsov has quit IRC17:42
*** akrivoka has quit IRC17:44
derekhlifeless: you there yet ?17:45
*** akuznetsov has joined #tripleo17:47
*** cd-undercloud has joined #tripleo17:52
cd-undercloud************** overcloud complete status=1 ************17:52
*** cd-undercloud has quit IRC17:52
derekhlifeless: sending you a mail, will be back later.17:54
*** derekh has quit IRC17:54
*** lazy_prince has quit IRC17:55
*** akuznetsov has quit IRC17:57
*** akuznetsov has joined #tripleo18:00
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Passing correct count to template creation
*** jp_at_hp has quit IRC18:08
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Passing correct count to template creation
*** athomas has quit IRC18:12
*** matty_dubs|lunch is now known as matty_dubs18:14
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: Remove ext4 resize= option.
*** jang1 has quit IRC18:24
lifelessdprince: good morning :P18:29
dprincelifeless: Hi. I figured we'd bump into each other today :P18:30
dprincelifeless: I can make the conversion to raw go away...18:30
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Passing real params to TuskarAPI
*** lucasagomes is now known as lucas-afk18:32
lifelessdprince: conversion to raw?18:33
dprincelifeless: Sorry. I've been touching too many things. That was a reply to your Nova branch.18:33
dprincelifeless: My ext4 resize2fs fixes actually make your threading issue (by avoiding the conversion to raw) a possibility18:34
lifelessso, ah18:36
lifelessyeah, I think you and I have a fairly fundamental disagreement on this :)18:37
dprincelifeless: which issue.18:37
lifelessI consider dd / its just a bunch of bytes as a fundamental abstraction here.18:38
dprincelifeless: I question the utility of that abstraction. It gives us a really crappy FS layout, and is causing chicken/egg sorts of problems w/ DIB.18:39
dprincelifeless: Plus w/ mkfs/rsync you get it done 15-20 percent faster. Better and faster. Sounds like a win to me.18:40
lifelessdprince: it gives us precisely the layout every other instance created via nova has, virt or otherwise18:40
dprincelifeless: By my reading the ext4 maintainers are saying 'Don't do this' to us.18:40
dprincelifeless: resize2fs wasn't intended to be used on an order of magnitude.18:41
dprincelifeless: For cloudy images... sure.18:41
dprincelifeless: it is okay to use it there. Going from say 1 2 GB iamge to a 200GB image.18:41
lifelesswell, we're going from 2GB to 20GB18:42
lifelessand its failing18:42
dprincelifeless: But going from a 2G image to 1PB is uncharted territory. We are hitting bugs (which may get fixed) but Ted basically said it wasn't designed for that.18:42
dprincelifeless: it is failing because of the parameters to resize=18:42
dprincelifeless: not because we are using the mechanism itself.18:42
lifelessdprince: I haven't seen that discussion with him :(18:42
dprincelifeless: it is in the ticket I filed... one sec18:43
*** vkozhukalov has joined #tripleo18:44
dprincelifeless: I also find other links with similar info18:44
dprincelifeless: words like, crazy, insane insane, performance-destroying hack are how he describes what we are doing now...18:45
dprincelifeless: at least that is how I read it...18:45
dprincelifeless: all we gotta do is mkfs/rsync at the right place and everything works perfectly...18:45
dprincelifeless: I actually would like that place to be the ramdisk (long term). But today we use nova-baremetal-deploy-helper for this so I see this as a step in the right direction.18:46
lifelessdprince: mmm so that was 4 years ago18:47
dprincelifeless: The new thread (which uses my image from two weeks ago) is fresh!18:48
lifelessmissed that link, looking now18:49
lifelessDon't get me wrong; we should be able to handle this case correctly,18:49
lifelessand not trigger a BUG_ON, but this is why most people aren't seeing"18:49
dprincelifeless: handling it is one thing. It clearly isn't as effecient to do this.18:50
dprincelifeless: mkfs.ext4 works best when used *on* the actual file system where it intends to live18:50
dprincelifeless: So while he'll look at fixing (may take weeks or months to propigate that fix) we should still consider ways to raise the bar here anyway.18:51
dprincelifeless: we already do this for ephemeral18:51
dprincelifeless: all we are doing here is making it right for the root as well18:52
lifelessthere's a couple of huge differences18:52
lifelessthe primary one is that we're reopening the quite significant security hole that guestfs is aimed at18:53
lifelessand limiting users to whatever the cloud provider offers fs wise18:53
lifelessthats less of alimit with ephemeral (we only format there because nova does, not because I think we *should*)18:53
*** openstackgerrit has quit IRC18:53
lifelessbah that sounds arrogant18:54
*** openstackgerrit has joined #tripleo18:54
dprincelifeless: we aren't even using guestfs w/ TripleO right now?18:54
lifelessI mean to say that in my ideal world cloud-init would be responsible for formatting the ephemeral partition on boot18:54
lifelessdprince: we haven't needed to because we don't poke inside user supplied images18:54
lifelessdprince: your patch changes that :(18:54
dprincelifeless: This isn't poking around.18:54
dprincelifeless: just mounting.18:55
lifelessright, which means a bad fs image can attack kernel drivers.18:55
dprincelifeless: which we *have* done before18:55
lifelessdprince: in nova bm? no18:55
dprincelifeless: yes. We already have code that mounts for the injection. Now I understand we don't need that anymore but still.18:55
lifelessdprince: not since we got rid of file injection18:55
lifelessdprince: we got rid of file injection to get rid of this18:55
dprincelifeless: code is there, it is just off18:55
lifelessdprince: it doesn't exist in ironic at all, and won't.18:56
dprincelifeless: Look, regardless of what you think of my Nova patch... the DIB patch is the right thing to do IMO. A 1PB root is like overshooting the moon.18:56
dprincelifeless: Who has a 1PB read only root anyways?18:57
* dprince knows we aren't using read only yet... but still18:57
*** e0ne has joined #tripleo18:57
dprincelifeless: the DIB fix is the right thing to do IMO. See patchset 1 where I made it an option if you like.18:57
dprincelifeless: once I figured out the Nova side of things I became convinced this was the right thing to do and just removed it entirely18:58
*** lucas-afk is now known as lucasagomes18:58
lifelessdprince: sorry, little distracted with family breakfast19:02
lifelessdprince: haven't abandoned the conversation19:02
*** jprovazn has joined #tripleo19:04
lifelessdprince: making it an option seems like an important facility given the bug in resize2fs19:05
dprincelifeless: Well, I'm happy to resurect that as a means to move forward. But I'd like to explore the Nova side further... perhaps using libguestfs even to ward off that concern.19:06
lifelessdprince: it also means we'll have massively more complexity as we move to multicast/torrent approaches, particularly for things like selinux19:07
lifelessdprince: I'm presuming your x% faster metric was on 10Gbs baremetal to baremetal test rig?19:07
dprincelifeless: my take home for all this is we should run mkfs on filesystems if we can and avoid resize2fs for grossly out of range resizes.19:08
openstackgerritRichard Su proposed a change to openstack/tripleo-heat-templates: Add username parameter for rabbit
lifelessdprince: I agree we need to avoid the resize bug until its fixed; I'm not at all convinced of the rest19:08
*** e0ne has quit IRC19:09
*** e0ne has joined #tripleo19:09
lifelessdprince: this stuff isn't any cheaper for kvm, but we don't mkfs the root in kvm19:12
lifelessblock devices as a bucket of bits is supremely useful at preserving the integrity of operating system images19:12
lifelessI think we should think -very- long and hard before looking through that layer19:12
dprincelifeless: right. And we don't have TBs or PBs of data for VMs so this isn't a concern.19:12
dprincelifeless: for bare metal having as efficent FS as possible seems like a reasonable goal19:13
dprincelifeless: Just because the cloud does something doesn't mean it is an excuse for us to use it at the lower level here.19:14
*** cd-undercloud has joined #tripleo19:14
cd-undercloud************** overcloud complete status=1 ************19:14
*** cd-undercloud has quit IRC19:14
*** e0ne has quit IRC19:14
*** jcooley_ has joined #tripleo19:15
lifelessdprince: I'm not arguing excuses, I'm arguing useful abstractions, more flexability for users, lower design risk of security holes19:17
lifelessdprince: greater ease of support for alternative operating systems19:17
lifelessdprince: easier support for massively parallel deployments19:17
dprincelifeless: we can itereate to solve the security issues fairly quickly here I think. As for the rest I'm arguing for the same things.19:18
dprincelifeless: My main arguement is DIB shouldn't have to know about the max size of the eventual root partition.19:19
lifelessdprince: I agree that it shouldn't19:19
dprincelifeless: For bare metal we are potentially dealing with *much* larger disks that normal VMs have.19:19
lifelessdprince: the only reason it does is an ext4 bug; if we built xfs images or btrfs or zfs it wouldn't need to.19:19
lifelessdprince: How so?19:20
lifelessdprince: VMs (on cinder devices) may have many TB volumes.19:20
lifelessdprince: larger than baremetal infact - thats why we went with 1PB, *for* VMs19:20
dprincelifeless: sure. And we can got into the TB with the ext defaults.19:20
dprincelifeless: re, btrfs or zfs, that is what you *think*. We don't know what would happend if we tried the same things there...19:21
lifelessI'm not feeling swayed19:22
lifelessand it's clear I'm not managing to communicate the terror of this abstraction violation that I have19:22
lifelessso lets focus on finding a compromise so we can fix the pragmatic issue and move forward19:22
dprincelifeless: I like how it is okay for TripleO to do injection for months but this terrifies you :)19:23
dprincelifeless: I'm talking about RO mounting an image w/ guestfs and that terrifies you.19:23
dprincelifeless: *and* we already run mkfs for the ephemeral partition19:23
*** e0ne has joined #tripleo19:23
dprincelifeless: My take is like get over it, Its Nova. That is what Nova does.19:24
dprincelifeless: the FS guys basically told us this is a design issue. i.e. run mkfs on the disk where you use it (if you can) and you'll be good.19:24
dprincelifeless: I think that is good advice.19:24
dprincelifeless: Now we can do the OpenStack thing and "roll our own" here but I think that may be questionable19:25
dprincelifeless: at least it is by me19:25
lifelessdprince: file injection was never ok19:28
lifelessdprince: I hated every minute of it, but we had a plan to get away from it, and we did19:28
dprincelifeless: this isn't file injection. It is mount (safely) a RO partition and copy data to something we just ran fdisk/mkfs on. (which we do already)19:30
lifelessdprince: now we're going in circles. The mount isn't safe (today), its limited by fs's the deploy engine supports, the output is limited by fs's the deploy engine supports, the copy isn't as trivially parallelised, and the fact we run mkfs for ephemeral is itself arguably a bug (but its what nova does)19:32
dprincelifeless: I don't think we are going in circles at all. Just a simple step in the right direction. At the end of the day I would rather our deployment ramdisk to much of this heavy lifting for us.19:33
lifelessdprince: I know its not file injection, you said that my being terrified by the consequences of the proposed approach was hypocritical because we did file injection - and I pointed out that the very first thing I did when I realised we did file injection was to file bugs...19:33
lifelessso its not hypocritical at all19:34
lifelessthe partition table on a machine is a well defined standard, its a grey area whether we should touch it or not (smoser argues we shouldn't - I argue that its very well defined)19:35
dprincelifeless: And the ext4 authors say it is best to mkfs *on* the disks themselves. That will give us the best layout.19:36
*** weshay has quit IRC19:36
lifelessthats a problem shared with vms, and solving it should be shared with vms19:39
lifelessthats a fundamental design approach for tripleo, that we align with and share solutions with the whole stack19:39
dprincelifeless: the issue we are dealing w/ is really large root filesystems.19:39
dprincelifeless: I don't think that is that common for VMs19:40
dprincelifeless: most VMs are going to have a smaller root and larger ephemeral...19:40
lifelessour rootfs is 20GB deployed today, which is not really large19:40
lifelesswhy do you say we're dealing with really large root file systems?19:40
dprincelifeless: then why not remove the 1PB resize= option like I'm proposing?19:40
lifelessdprince: because VM's suffer the issue; which is why I said I'm happy for it to be made optional19:41
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: Make max-online-resize an option.
dprincelifeless: so be it ^^19:42
lifelesswhat do you think of the suggestion I made on patch set 1 - to link to the resize2fs bug ?19:43
dprincelifeless: it is linked?
lifelessdprince: not *our* bug, whatever bug ted will close when he fixes the issue19:44
dprincelifeless: That is beyond the scope of DIB IMO. Their mailing list is the place for that I think.19:45
lifelessdprince: right, my thought was to have a link to something so that folk who use the option, shoot themselves in the foot by making it too big, have a pointer for where to go19:46
dprincelifeless: I linked to the thread in the bug already :)19:46
lifelessdprince: from the commit message19:46
lifelessdprince: ?19:46
lifelessdprince: For clarity - I'm suggesting in the help, so that users see it without having to have access to the git history.19:47
*** e0ne has quit IRC19:47
lifelessdprince: because folk with installed packages won't have git history.19:47
*** e0ne has joined #tripleo19:47
dprincelifeless: this is why I wanted to rip this out, It is a bad option.19:47
dprincelifeless: but whatever. I can pollute the codebase further I guess...19:48
dprincelifeless: I personally prefer less options here. they are always better if you can pull it of (which we can)19:48
*** jcoufal has joined #tripleo19:51
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Add LICENSE
openstackgerritJay Dobies proposed a change to openstack/tuskar: Eager load role data with count information
*** lucasagomes is now known as lucas-afk20:02
*** gcha has joined #tripleo20:10
*** weshay has joined #tripleo20:13
lifelesshad to do kindy run, sorry20:24
lifelessdprince: I agree with a preference for less options, but not at the cost proposed - which is going from a single narrow interface (partition table + PXE boot) that supports most OS's to support on an OS by OS basis.20:24
lifelessdprince: that shifts things inappropriately20:25
*** jcooley_ has quit IRC20:25
dprincelifeless: I think you overstate this as an OS issue. It isn't like what we do today supports Windows or something. We support any OS that has ext varients currently and if people are interested in other filesystems we can add those easy enough I think.20:26
lifelessdprince: AIUI we support windows just fine in nova-bm20:27
lifelessdprince: you have to change a bunch of global options, but no code20:27
lifelessdprince: ditto freebsd etc20:27
dprincelifeless: I really like how we throw around rsync for other TripleO use cases (like potentially for in-place image updates) but I throw it out here for provisioning and all of a sudden it is evil.20:28
lifelessdprince: its not evil, its an abstraction layer issue20:28
*** e0ne has quit IRC20:28
lifelessdprince: in-instance its clearly fine, it doesn't affect the core20:28
lifelessdprince: it will work for both VMs and BM20:28
dprincelifeless: an ephemeral mkfs. I suppose taht works on "other OS's" too then?20:28
*** e0ne has joined #tripleo20:29
dprincelifeless: again, I think you are overstating the facts a bit20:29
*** akuznetsov has quit IRC20:29
lifelessdprince: I am not going to support ephemeral mkfs, because I don't like it, but yes - for other OSs they can format that partition themselves.20:29
lifelessdprince: which I think we should be doing anyhow20:29
lifelessdprince: Any chance you'll remove your -2 on ? I don't think it conflicts with your proposed change at all (and even with your change, I still would worry about thrashing)20:30
dprincelifeless: I'd like to test it first....20:31
dprincelifeless: but then maybe. ALthough I do think it may be worth discussing this on the list too.20:31
dprincelifeless: perhaps if nothing else this helps us get a good end goal in mind here.20:31
lifelessdprince: in case you aren't aware, please read through the discussion I had with deva on that patch about concurrency elsewhere in the system.20:31
dprincelifeless: I'm displeased with DIB at this point and how it mucks w/ the filesystem.20:32
devanandalifeless: oh hai!20:32
lifelessdprince: its really orthogonal to your patch - it applies whether or not you use rsync or dd, and whether or not one does a convert to raw or not.20:32
dprincelifeless: So we have a tradeoff here. Abstraction vs. a bad file system layout.20:32
lifelessdprince: what does 'bad filesystem layout' mean for this specific use case20:32
lifelessdprince: in really specific terms.20:33
dprincelifeless: I'm not talking about your patch man :). I already gave you the answer there... I just want to test it.20:33
lifelessdprince: I am20:33
*** e0ne has quit IRC20:33
lifelessdprince: ok; please do.20:33
devanandalifeless: haven't seen you chime in the couple times it's come up, so just checking - have you seen the comments re: ironic's graduation blocked on Nova driver CI ?20:34
lifelessdevananda: possibly not; link me up ?20:34
lifelessdprince: w.r.t. bad filesystem layout, I meant - whats 'bad' about it, what performance /robustness  impact does it have (given that its (in principle) read only?)20:35
*** cd-undercloud has joined #tripleo20:35
cd-undercloud************** overcloud complete status=1 ************20:35
*** cd-undercloud has quit IRC20:35
devanandalifeless: that's the tail of the ML thread with russel's comment20:36
devanandahe repeated it in today's meeting20:36
*** markmc has quit IRC20:36
lifelessso, AIUI you can deploy cirros images in devstack-gate, so you should be able to do full CI there.20:37
lifelessyes, it doesn't test IPMI, but CI and testing-every-single-implementation are separate things20:38
lifelesslike, we don't test every version of kvm20:38
lifelessIronic is a layer, so to me you test the nova-ironic-$pick-a-power-driver-deploy path works, which tells you that nova<->Ironic - the nova ironic driver - works20:39
lifelessand you test all the possible Ironic backends separately20:39
lifelessI don't think we can even once RH region comes online, do per-nova commit baremetal test runs20:39
lifelesstoo much bandwidth20:39
lifelesswe can do per-Ironic baremetal runs I expect20:40
lifelessbut a lot of optimisation will be needed20:40
lifelessor we can do a check-on-+2 feature for zuul20:40
lifelesswhich is whats needed I think anyhow, based on a discussion with sdague20:40
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Add RHEL 7 image element
devanandalifeless: right20:42
devanandalifeless: that's the plan20:42
devanandalifeless: in infra's VMs, do integration tests with cirros images20:42
devanandawhich will exercise nova, nova-ironic driver, ironic-neutron interaction, and the PXE driver, etc...20:43
devanandai'm going to focus on that, and hopefully a few folks will dog pile with me on getting it done ASAP20:43
*** ifarkas has quit IRC20:44
devanandaonce we have it, hopefully nova-core will start actually reviewing our driver patches and grant a FFE ...20:44
*** jcoufal has quit IRC20:48
*** jcoufal has joined #tripleo20:48
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Nova package install support (openSUSE)
*** jtomasek has quit IRC21:18
*** gcha has quit IRC21:20
*** derekh has joined #tripleo21:31
lifelessderekh: HALLO MY FRIEND21:32
lifelessahh I'm finally into the box21:32
lifelessserial console headaches21:32
derekhlifeless: to get a console on the overcloud machine I used the interface, cou;dn't get sol to work21:32
lifelessderekh: anyhow, this is a known issue21:33
derekhlifeless: *web interface21:33
lifelessnow that I'm on the box21:33
lifelessin trello, firedrill column21:33
lifelessshould be fixed now21:35
lifelessI've made eth2 manual\n pre-up ip link set eth2 up21:36
lifelessrather than dhcp21:36
lifelessit'll probably take out the compute nodes next21:36
derekhlifeless: ok, cool,  I saw the duplicate IP on br-untagged and eth2 but comparing it to ci-overcloud-novacompute9 it seemed to be the same21:37
lifelessderekh: yeah, thats a broken config that only ever works randomly21:37
openstackgerritDirk Mueller proposed a change to openstack/diskimage-builder: Pep8/Pyflakes fixing
lifelessderekh: look at /etc/network/interfaces now on the control plane21:38
*** lucas-afk has left #tripleo21:38
*** e0ne has joined #tripleo21:39
derekhlifeless: ok, got it21:39
derekhdarn it, spend a crap load of today trying to figure it out :-(21:40
lifelesssorry - there is a critical bug filed about it :)21:41
openstackgerritDirk Mueller proposed a change to openstack/diskimage-builder: Add helper to map service users
lifelessalways look for firedrill cards/critical bugs :)21:41
derekhlifeless: yup21:42
lifelessso, we now need to fixup the compute nodes21:42
lifelesswhere fixup == ifdown eth2; fix /e/n/i; ifup eth221:42
lifelessfixup /e/n/i, kill the rogue dhclient, and ip addr del the address from eth221:43
lifelesswhich can be done without console access21:43
*** rbrady is now known as rbrad-afk21:43
*** rbrad-afk is now known as rbrady-afk21:43
derekhbrb, can help out with it in a couple of minutes21:44
*** coolsvap has quit IRC21:44
*** e0ne has quit IRC21:44
*** hashar has joined #tripleo21:44
*** hashar has quit IRC21:52
*** vkozhukalov has quit IRC21:53
derekhlifeless: passing over machines doing option 221:53
*** jdob has quit IRC21:54
*** jprovazn has quit IRC21:55
lifelessderekh: awesome, sorry, multiplexing here21:56
*** cd-undercloud has joined #tripleo21:56
cd-undercloud************** overcloud complete status=1 ************21:56
*** cd-undercloud has quit IRC21:56
*** cwolferh has quit IRC22:01
*** spzala has joined #tripleo22:02
ccrouchswift question:22:08
ccrouchso what does the mount of /srv/node/d1 needed for22:08
*** jcoufal has quit IRC22:12
*** lblanchard has quit IRC22:13
derekhccrouch: needs to be a filesystem that supports extended attributes22:14
derekhccrouch: swift doesn't need it to be a seperate filesystem unless configured that way, checking22:15
derekhlifeless: done22:15
lifelessderekh: cool22:15
derekhccrouch: so
derekhccrouch: i.e. it doesn;t need to be a seperate mount22:16
ccrouchderekh: ah!22:16
ccrouchso it will just22:16
ccrouchmkdir d122:16
derekhccrouch: swift will do that iirc22:16
ccrouchright, ok that makes a tonne of sense, and explains how this works at all :-)22:17
derekhccrouch: np22:17
*** julim_ has quit IRC22:19
derekhlifeless: so we got a load of instances in error state, do they need to be deleted by us?22:21
lifelessderekh: if they are nodepool instances, shouldn't no.22:21
lifelesslets -> -infra22:21
derekhlifeless: k22:21
SpamapS[    3.356865] init: mountall main process (209) terminated with status 12722:25
SpamapS[  303.072130] EXT4-fs (vda1): error count: 922:25
SpamapS[  303.073668] EXT4-fs (vda1): initial error at 1392632867: ext4_mb_generate_buddy:75522:25
SpamapS[  303.076503] EXT4-fs (vda1): last error at 1392633406: ext4_mb_generate_buddy:75522:25
SpamapSsomething borken on booting instance22:25
SpamapSmordred: FYI, we do not use distro pip for anything other than installing virtualenv. From that point on we use the pip from the virtualenv.22:27
mordredSpamapS: you shouldn't use distro pip for that22:28
mordredit's probably broken or insecure22:28
SpamapSso, download pip tarball?22:28
mordredyeah. I can link you to scripts and whatnot22:28
SpamapSor you could submit a patch22:28
*** cwolferh has joined #tripleo22:32
derekhlifeless: ok, verified cloud, create instance, ssh to vm so we should be good to go22:35
*** dprince has quit IRC22:35
*** vipuls is now known as vipul22:38
*** EntropyWorks has quit IRC22:38
bnemecmordred: SpamapS: FWIW, I had to install pip from source in my rhel 7 image:
bnemecI see it's available in epel now, but it wasn't when I started working on that.22:41
*** EntropyWorks has joined #tripleo22:41
lifelessderekh: lol22:41
lifelessERROR: Cannot 'forceDelete' while instance is in vm_state error (HTTP 409) (Request-ID: req-1c4a88c3-4ea1-45a8-b987-629a69b4af06)22:41
lifelessforce delete not so forceful22:41
derekhmaybe there is a --really-force flag22:45
slaglei fall back to mysql22:45
slaglei know, shame on me22:45
lifelessslagle: do you file bugs ? :)22:46
* lifeless files a bug22:47
slaglei didn't consider it a bug22:48
lifelessa VM you can't delete? definitely a bug22:48
derekhlifeless:  ok, I'm outta here, cloud is up, cancel firedrill as appropriate ,22:49
*** derekh has quit IRC22:49
lifelessderekh: thanks man!22:49
mordredbnemec: well.... let me make some suggestions...22:52
*** cwolferh has quit IRC22:56
*** rcarrillocruz has quit IRC23:00
*** rcarrillocruz has joined #tripleo23:00
*** noslzzp has quit IRC23:09
*** rlandy has quit IRC23:13
*** cd-undercloud has joined #tripleo23:18
cd-undercloud************** overcloud complete status=1 ************23:18
*** cd-undercloud has quit IRC23:18
*** matty_dubs is now known as matty_dubs|gone23:22
*** rcarrillocruz has quit IRC23:24
*** noslzzp has joined #tripleo23:25
*** cwolferh has joined #tripleo23:28
rwsuis there a reason why horizon isn't installed on the overcloud control?23:33
SpamapSrwsu: nobody asked for it yet? ;)23:33
rwsuhaha, ok I think it is time23:34
*** jcooley_ has joined #tripleo23:37
mordredbnemec: lines 20-26 are what you want23:46
mordredbnemec: if you want to install pip somewhere23:47
mordredlifeless: ^^23:47
mordredthose lines encapsulate the published process from setuptools and pip upstream23:48

Generated by 2.14.0 by Marius Gedminas - find it at!