Friday, 2015-11-06

jeblairclarkb: i find my computer's ability to carry firefox's memory/cpu footprint approximately equal to the space for horizontal tabs, so it generally works out for me.  :)00:00
*** sputnik13 has joined #openstack-infra00:00
clarkbjeblair: haha00:01
*** akshai has quit IRC00:01
clarkbjeblair: I definitely find myself with 6GB firefox processes then realize my 400 tabs should probably be brought back under control00:01
anteayacan you even remember what is open when you have 400 tabs going?00:02
*** freyes has joined #openstack-infra00:02
clarkbanteaya: usually a good chunk can be removed but if properly organized in trees it usually makes sense00:02
openstackgerritMichael Johnson proposed openstack/diskimage-builder: Fix diskimage-builder image size
*** wenlock_ has quit IRC00:03
*** tnarg has quit IRC00:03
*** davideagnello has joined #openstack-infra00:03
clarkbI also end up making it very spacial, I have zuul status and email pinned, other monitoring tabs go near the top (graphite graphs etc), documentation type stuff in the middle, changes at the end00:03
zaroanteaya: i've seen his tabs.  it's insane.00:03
clarkbwith horizontal tabs you get no indication of special relativity00:03
clarkbso I hvae no idea if I am at the end or middle or beginning00:03
clarkbonce I installed a plugin to limit myself to 100 tabs00:03
clarkbI uninstalled it within a couple hours00:03
*** tnarg has joined #openstack-infra00:04
*** wenlock has quit IRC00:05
SamYapleclarkb: ho00:05
mordredclarkb: I would be unfunctional with that00:05
*** aeng has joined #openstack-infra00:05
SamYaplemordred told me to talk to you about multinode gating00:05
jesusaurusim weird and have a dozen windows each with a dozen tabs for general organization00:05
*** dingyichen has quit IRC00:05
clarkbSamYaple: hi00:06
mordredclarkb: yah - SamYaple was asking about making a multi-node test. my answer was "clarkb"00:06
clarkbso first question I like to ask is do you have a single node test that works? if not start there. Your debugging will be much much simpler00:06
*** dingyichen has joined #openstack-infra00:06
SamYaplein fact i have a multinode-on-single-node patch that works.... but i OOM00:07
*** camunoz has joined #openstack-infra00:07
clarkbwell I wouldn't even multinode node on single node00:07
clarkbjust an all in one setup00:08
*** sputnik13 has quit IRC00:08
clarkbbut anyways if you have a thing that works on one node yay00:08
anteayaclarkb: yeah I work with a spacial mindset as well00:08
SamYapleyea it was an experiment00:08
anteayazaro: I believe you00:08
SamYapleclarkb: that multinode code isn't merged. if i can get true multinode i want to go that direction00:08
*** tnarg has quit IRC00:08
*** heyongli has quit IRC00:09
clarkbSamYaple: the basic setup is two machines with one attached to jenkins as the "primary" node00:09
anteayaclarkb: well if I have tab organization questions, I know who to talk to, thanks00:09
clarkbwe use the primary node as the control machine for setup and it configures itself and the "subnode" to run openstack00:09
*** heyongli has joined #openstack-infra00:09
*** sputnik13 has joined #openstack-infra00:10
*** cloudtrainme has quit IRC00:10
clarkbthe biggest limitation here is we don't have control over our networking so we setup overlay networks documented at
* SamYaple reading00:10
clarkbmy suggestion would be to not reinvent that ovelray networking because 1) it actually was a huge pain to get owrking in the first place and 2) if our clouds change infra will update that to accomodate the changes and you would get that for free00:10
clarkbI actually need to finish ansibling the multinode setup so that we don't do the manual remote ssh stuff00:12
*** _ianychoi_ is now known as ianychoi00:13
clarkbbut most of the setup at this point is done by ansible because it is really easy to assign nodes to groups and have ansible do the right thing for an arbitrary number of hosts00:13
jlkjeblair: is it always one host that has slow runs, or is it spread?00:13
SamYapleclarkb: yea this looks like what i did in the RAX public cloud for testing stuff with the overlays00:14
SamYaplewhats the bandwidth like between nodes?00:14
clarkbSamYaple: its whatever $cloud provides to that flavor00:15
clarkbI think generally somewhere between 100mbps and 1gbps00:15
clarkbhasn't been a problem00:15
SamYapleok so these vms are not split between clouds00:16
clarkbno, they will always be in the same region00:16
SamYapletheyll all be in DFW or ORD or similiar00:16
SamYapleok cool00:16
*** yamamoto has joined #openstack-infra00:16
anteayaSamYaple: mind if I ask what project you are going to be testing on multinode?00:17
SamYapleanteaya: Kolla00:17
anteayaSamYaple: thank you00:18
SamYapleclarkb: all this seems fairly straight forward to how ive done it before00:18
SamYaplei dont for see issues00:18
*** sridhar_ram1 has quit IRC00:18
*** ddieterly has joined #openstack-infra00:19
*** heyongli has quit IRC00:19
SamYaplewhats the proceedure here for testing this? request a 2 node gate and then build the gate tests?00:19
clarkbSamYaple: add a new job that runs on the 2-node label to experimental queue of your project and iterate on that00:20
*** heyongli has joined #openstack-infra00:20
*** vilobhmm11 has quit IRC00:21
* SamYaple dives into project-config again00:21
SamYaplei dont think we have an experimental queue running anything yet00:22
SamYaplethanks will get started!00:22
clarkbruagair: mordred et al why are we not using libapache2-mod-auth-openid for phabricator auth?00:24
clarkbreading up on cauth it seems needlessly complicated and doesn't do anything that the existing apache module for openid auth can't do for us more easily00:24
*** smarcet has joined #openstack-infra00:24
clarkbjesusaurus: ^ I want to say you used this at one time? any cahcne you want to share your thoughts on using it?00:25
*** sputnik13 has quit IRC00:26
*** tonytan4ever has joined #openstack-infra00:26
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an attachment storage driver option
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add top-level attachments support
clarkbjesusaurus: one lsat question about the remainder stuff now that I read it again. should we skip the for loop if remainder is set?00:27
clarkbjesusaurus: so if remainder do remainder stuff else for loop00:27
*** cody-somerville has quit IRC00:29
*** sambetts has quit IRC00:29
jesusaurusclarkb: I dont recall using mod-auth-openid, but I may have touched it briefly once00:29
clarkbI thought you had put kibana or something in front of openid auth00:29
*** smarcet has quit IRC00:29
*** heyongli has quit IRC00:29
*** andersonvom has joined #openstack-infra00:29
jesusaurusno, that was ldap auth00:30
*** heyongli has joined #openstack-infra00:30
jesusaurusi think i used openid auth with lightdm-login briefly years ago but didnt get that working00:31
*** ybathia has quit IRC00:32
*** sabeen1 has quit IRC00:32
jesusaurusclarkb: for the remainder stuff, we still need the loop because we are reading from a generator00:33
clarkboh right00:33
clarkbbasically thats how we convert the iterator to a list00:33
jeblairjlk: it's spread00:34
*** andersonvom has quit IRC00:34
*** tlian has quit IRC00:34
*** ybathia has joined #openstack-infra00:34
*** sridhar_ram has joined #openstack-infra00:34
*** sambetts has joined #openstack-infra00:34
*** ybathia has quit IRC00:35
ruagairgood question clarkb00:39
*** bhunter71 has quit IRC00:39
ruagairI'm coming to a similar conclusion regarding cauth's complexity as I'm in th emidst of deploying it now.00:39
*** heyongli has quit IRC00:39
*** tnarg has joined #openstack-infra00:40
ruagairI've been deferring at present to the decisions made before me clarkb. I'm open to what ever auth mechanism we consider most appropriate.00:40
*** heyongli has joined #openstack-infra00:40
*** sputnik13 has joined #openstack-infra00:40
*** tlian has joined #openstack-infra00:41
jlkjeblair: well darn.00:41
ruagairlibapache2-mod-auth-openid pass through REMOTE_USER, clarkb?00:41
ruagairThen suprficially, I can't see a barrier.00:41
fungisounds like a much simpler solution, if it works00:42
ruagairWith authentication system are we hooking back to for OpenStack SSO?00:42
ruagaircauth is proving less elegant thna I'd like.00:42 eventually, though possibly in the interim00:43
ruagairSo i'm happy to turn my attention to libapache2-mod-auth-openid00:43
*** tnarg has quit IRC00:43
openstackgerritSam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla
jeblairmordred: ^ did you have a reason for favoring cauth over libapache2-mod-auth-openid ?00:43
*** sridhar_ram1 has joined #openstack-infra00:43
ruagairThanks fungi. I was at the point of pointing cauth to *something* and just realised I was unclear on what that sometihng was :-)00:43
*** tnarg has joined #openstack-infra00:43
SamYapleclarkb: how does that look?00:43
*** Liuqing has joined #openstack-infra00:43
fungiruagair: is also available for testing purposes if needed00:44
fungithough that's really better suited to testing changes to the openstackid project itself00:44
ruagairIs only dev status atm, fungi?00:44
fungino problem authenticating against production from development systems00:44
jeblairyeah, i think now that prod is up and running, probably not a compelling reason not to use it00:44
*** sridhar_ram has quit IRC00:45
clarkbSamYaple you should put multinode in the job name then the correct nodes will be used00:45
fungiruagair: it's being used for authentication to, and optionally at the moment00:45
ruagairIf is prod, should I not be using that over launchpad?00:45
SamYapleclarkb: ill adjust it from multi to multinode00:45
*** jamielennox is now known as jamielennox|away00:46
fungiruagair: i think it's a question of mapping the current users from launchpad bugs to maniphest bugs00:46
*** austin81 has left #openstack-infra00:46
funginot sure if we care, ultimately00:46
fungiit would of course be much simpler not to care ;)00:46
*** tnarg_ has joined #openstack-infra00:46
openstackgerritSam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla
SamYapleclarkb: done, only concern?00:47
ruagairThat is a mapping problem I'm having in general, fungi.00:47
*** sridhar_ram has joined #openstack-infra00:47
*** xyang1 has quit IRC00:47
*** jamielennox|away is now known as jamielennox00:47
ruagairOk, so i'll look at00:47
*** sputnik13 has quit IRC00:48
clarkbSamYaple shouldnt need the functions update anymore00:48
anteayaSamYaple: you need an entry in zuul/layout.yaml for kolla00:48
*** daemontool has quit IRC00:48
ruagairOk, so i'll look at libapache2-mod-auth-openid and try connecting either it or cauth to and or, which ever gives me the sanest mapping.00:48
ruagairclarkb fungi ^^00:48
*** daemontool has joined #openstack-infra00:48
*** sputnik13 has joined #openstack-infra00:48
*** tnarg has quit IRC00:49
asselinzaro,  is up now? it seems to be...00:49
SamYapleclarkb: last time i didnt do the function updates my job stayed in 'non-registered' stater00:49
*** mgrosz has joined #openstack-infra00:49
SamYapleanteaya: done00:49
anteayaSamYaple: great00:49
anteayaasselin: it is up for me00:49
*** sridhar_ram1 has quit IRC00:50
*** heyongli has quit IRC00:50
openstackgerritSam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla
*** heyongli has joined #openstack-infra00:50
fungiruagair: sounds not too terrible a combination of options to test00:51
*** Piet has joined #openstack-infra00:51
anteayaSamYaple: sorry I mean here:
*** mgrosz has quit IRC00:51
clarkbSamYaple: there ar ealready rules in place that match multinode00:51
anteayaSamYaple: after gate: put experimental: and the name of the job00:51
SamYapleclarkb: ok ill remove the regex match00:51
SamYapleanteaya: so append expiremental to gate, not replace gate?00:52
clarkbruagair: yup sounds good to me00:52
*** tlian has quit IRC00:52
*** dims has joined #openstack-infra00:52
SamYapleanteaya: scratch that question00:53
*** hemna is now known as hemnafk00:53
* anteaya scratches that question00:54
*** eil397 has quit IRC00:55
openstackgerritSam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla
SamYaplealright. i believe that may be correct...00:57
*** sputnik13 has quit IRC00:57
SamYaplei fear i may be starting to understand project-config. scary thought!00:57
*** tlian has joined #openstack-infra00:57
anteayaSamYaple: reviewed00:58
anteayaSamYaple: run away00:58
*** jamielennox is now known as jamielennox|away00:58
anteayaSamYaple: one oops the rest looks good00:59
SamYaplei give up. *abandonded*01:00
openstackgerritSam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla
*** heyongli has quit IRC01:00
SamYaplethanks for helping me out here. it would have been a while for me to do it alone01:00
bardiahow long does nodepool image-build dpc usually take?01:01
timrcjeblair: Sorry I've been heads down on some other stuff today... so... you have a theory that your variances in performance are coming from disk io due to spinny disks?01:01
*** heyongli has joined #openstack-infra01:01
anteayaSamYaple: the run away was in response to the scary thought01:02
anteayaSamYaple: you patch looks good, thanks01:02
*** sridhar_ram1 has joined #openstack-infra01:03
anteayabardia: good question01:03
SamYapleno thank you anteaya! youre the real hero here01:03
anteayaha ha ha, never a hero01:03
*** sridhar_ram has quit IRC01:03
anteayaglad you got what you needed01:03
SamYapleclarkb: i believe i addressed your concerns as well. thanks for the input. i may (will probably) hit you up again when i start ahmmering on this01:03
clarkbbardia: I am not sure what dpc is but usually snapshot builds take ~1 hour and dib builds take ~15 minutes with a warm cache01:03
anteayaSamYaple: I don't understand mulitnode like clarkb does01:04
SamYapleanteaya: everyone keeps saying that....01:04
bardiaclarkb: dpc is the image build name given in sample nodepool.yaml01:04
* SamYaple wonders what he is getting into01:04
*** sputnik13 has joined #openstack-infra01:04
*** Liuqing has quit IRC01:05
asselinzaro, sorry but I don't know how to 'review' anything in review-dev.o.o . I see the 'toggle CI' button, but no 'review' button'. where should I look?01:05
SamYapleasselin: are you signed in?01:05
asselinSamYaple, yes01:05
clarkbasselin: the "Reply" button01:05
bardiaclarkb: this is the disk-image-create that nodepool is running: /bin/bash /usr/local/bin/disk-image-create -x -t qcow2 --no-tmpfs --qemu-img-options compat=0.10 -o /opt/nodepool_dib/dpc-1446771643 ubuntu vm openstack-repos puppet nodepool-base node-devstack01:05
*** sputnik13 has quit IRC01:05
asselinI see...ok thanks :)01:05
clarkbbardia: ya so that takes about 15-20 minutes for us with a warm cache iirc01:06
*** jamielennox|away is now known as jamielennox01:06
clarkbbardia: a cold cache is going to take longer and depend on how fast your network IO is01:06
bardiaoh ok01:06
bardiaclarkb: thanks I was worried I was doing something wrong01:06
bardiaclarkb: how do you know nodepool successfully connected with the cloud provider? when I run nodepool list it doesn't list anything01:08
clarkbbardia: nodepool list won't show anything until it has an image built to boot01:08
bardiadoes it list a node after I upload an image to the provider? (nodepool image-upload)?01:08
bardiaah ok01:08
SamYapleclarkb: question, how does the logs publishing work for multinode?01:10
*** heyongli has quit IRC01:10
clarkbSamYaple: devstack-gate runs an ansible file copy to copy everything to the primary node so jenkins can copy it off01:11
*** heyongli has joined #openstack-infra01:11
*** nadya has joined #openstack-infra01:13
clarkbSamYaple: you should look at devstack-gate to see how it sets thing up (and again I highly recommend using the network setup there at the very least)01:15
asselinzaro, fyi I have 3rd party ci zuul listening to events on review-dev. will setup some test jobs now.
*** dims has quit IRC01:17
*** cschwede has quit IRC01:17
*** Daisy has joined #openstack-infra01:17
*** nadya has quit IRC01:18
*** Daisy has quit IRC01:19
*** Daisy has joined #openstack-infra01:20
*** heyongli has quit IRC01:20
*** otter768 has joined #openstack-infra01:20
*** heyongli has joined #openstack-infra01:21
*** sdake has quit IRC01:22
*** andersonvom has joined #openstack-infra01:23
anteayaasselin: you can add line and file comments, nice01:24
*** sdake has joined #openstack-infra01:24
asselinanteaya, yes, but that was more to test my ci getting gerrit events :) not much going on over there01:24
*** otter768 has quit IRC01:25
asselinanteaya, but yes learned how to do that, and also 'reply' to a review01:25
*** lyz has joined #openstack-infra01:27
*** dangers is now known as dangers_away01:27
anteayayeah not a lot happening in review-dev atm01:27
asselinanteaya, I have to admit, even knowing what I'm doing, and with common-ci, setting up 3rd party ci is not easy01:28
*** andersonvom has quit IRC01:28
*** Daisy has quit IRC01:28
anteayanot easy01:28
anteayaI don't foresee a future where it ever will be01:29
anteayadoes that make me a bad person?01:29
*** pvaneck has quit IRC01:29
anteayaasselin: did you get my comments?01:29
asselinanteaya, I'm looking forward to zuulv3. Taking out jenkins will simplify a lot.01:30
asselinanteaya, no...let me check01:30
anteayaI too am looking forward to v301:30
*** Liuqing has joined #openstack-infra01:30
*** heyongli has quit IRC01:31
asselinanteaya, which patch?01:31
anteayaasselin: top 3 patches on this list:
anteayasorry, that won't help you01:31
*** heyongli has joined #openstack-infra01:31
anteayatry this:
anteayareviewer:self is non-tranferable01:32
*** Daisy has joined #openstack-infra01:34
SamYapleclarkb: I will be reusing everything i can. I am a stong believe in someone-else-should-do-the-work01:34
*** rfolco has joined #openstack-infra01:35
*** asettle has quit IRC01:36
*** EricGonczer_ has joined #openstack-infra01:36
asselinanteaya, sorry..what's your question? I didn't get any e-mail about you commenting.01:37
*** yamamoto has quit IRC01:38
anteayaasselin: did stream-events not emit my comments01:38
anteayaasselin: or I guess you are only looking at patch uploads, right?01:39
*** Daisy has quit IRC01:39
asselinanteaya, oh ok I see. yes it did: 2015-11-06 01:28:20,154 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added gtest-org/test2 master 5366,1>01:39
asselin2015-11-06 01:28:35,071 DEBUG zuul.Scheduler: Adding trigger event: <TriggerEvent comment-added gtest-org/gtest master 5370,1>01:39
clarkbjesusaurus: I did +2 that os-loganalyze change. Other reviewrs that would be good to take a look at it are jhesketh and sdague01:39
jheskethclarkb: got a link?01:40
*** jyuso1 has joined #openstack-infra01:41
*** heyongli has quit IRC01:41
*** salv-orlando has joined #openstack-infra01:41
*** wenlock has joined #openstack-infra01:41
*** heyongli has joined #openstack-infra01:42
*** bharathm has quit IRC01:43
bardiawhat's the next step after building image and uploading to provider? run jenkins dsvm-tempest-full and it will launch a node and run the tests?01:43
clarkbbardia: nodepool should boot a test node and attach it ot jenkins01:44
clarkbthen jenkins can run jobs on it01:44
bardiaclarkb: does nodepool image-upload take care of this?01:45
clarkbbardia: no, nodepool the daemon does01:45
asselinbardia, I'm really hoping my documentation explains that..01:47
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Add acceptance tests for puppet-diskimage_builder.
openstackgerritMerged openstack-infra/puppet-bandersnatch: Add acceptance tests for puppet-bandersnatch
bardiaasselin: I think you should mention nodepool image-upload in your docs01:48
bardiaasselin: its a missing step01:49
bardiai think01:49
bardiaasselin: right after sudo service nodepool start line 20601:49
*** Swami has quit IRC01:49
asselinbardia, it should happen automatically when you do that01:49
bardiait didn't01:50
bardiabug in nodepool?01:50
asselinbardia, but I can add that regardless. It's helpful in case there are upload errors01:50
*** zhenguo has joined #openstack-infra01:50
asselinbardia, did image-upload work for you?01:50
bardiaasselin: it uploaded the image but its failing to start an instance01:50
bardiaasselin: trying to figure out why01:51
bardia2015-11-05 17:50:17,397 ERROR nodepool.NodeLauncher: LaunchStatusException launching node id: 56 in provider: local_01 error:01:51
bardiaTraceback (most recent call last):01:51
bardia  File "/home/coho/nodepool/nodepool/", line 406, in _run01:51
bardia    dt = self.launchNode(session)01:51
bardia  File "/home/coho/nodepool/nodepool/", line 474, in launchNode01:51
bardia    server['status']))01:51
bardiatraceback from nodepool log01:51
*** heyongli has quit IRC01:51
asselinbardia, can you paste more from the /var/log/nodepool/debug.log here:
*** heyongli has joined #openstack-infra01:52
bardiaits just been stuck in a loop where it tries to start an instance on the provider but provider errors out and it just keeps trying01:54
asselinbardia, which provider are you using?01:55
bardialocal devstack deployment01:55
asselinbardia, please check your devstack nova screen  logs for errors01:55
bardiaasselin: checking right now01:56
bardiaasselin: what screen number is it?01:56
asselin probably n-cpu01:56
jheskethclarkb, jesusaurus: reviewed01:57
*** otter768 has joined #openstack-infra01:57
*** tiswanso has joined #openstack-infra01:57
*** tiswanso has quit IRC01:57
*** tiswanso has joined #openstack-infra01:58
*** salv-orlando has quit IRC01:59
*** heyongli has quit IRC02:01
*** Daisy has joined #openstack-infra02:02
SamYapleclarkb: i have followed the code but im a bit fuzzy on how you use ansible to push the ssh auth to the subnodes. If ansible can ssh to these nodes, then aren't the keys already setup?02:02
*** heyongli has joined #openstack-infra02:02
asselinSamYaple, I belive the ssh auth should already be set on the primary and subnodes. same user and ssh keys02:05
SamYaplethen why does this function exist
SamYapleasselin: if they keys are already setup then its fine02:06
clarkbbecause libvirt02:07
clarkbso ansible is fine but libvirt requires extra help02:07
SamYapleok that explains the chicken and the egg i was finding02:08
SamYapleit all makes sense now how its setup02:08
clarkbone is test env layer the other is cloud under test layer02:09
clarkbseparate concerms02:09
*** Daisy has quit IRC02:09
*** andersonvom has joined #openstack-infra02:09
*** Daisy has joined #openstack-infra02:09
SamYapleits a pretty decent piece of setup work there clarkb. my stuff was far more basic because i could control the networks attached to the vms on a single cloud02:11
*** heyongli has quit IRC02:12
*** heyongli has joined #openstack-infra02:12
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Add documentation for third-party-ci setup
clarkbSamYaple we dont have that kind of control heck one cloud gives us a /3202:14
*** Daisy has quit IRC02:14
*** wenlock has quit IRC02:14
SamYapleright which is why i am admiring your work. mine i could expect thigns :)02:14
*** unicell has quit IRC02:17
*** sridhar_ram1 has quit IRC02:17
*** woodster_ has quit IRC02:19
*** heyongli has quit IRC02:22
bardiainteresting in devstack I can create a m1.medium instance but not m1.large. it throws02:23
bardiaNo valid host was found. There are not enough hosts available.02:23
bardiaFile "/opt/stack/nova/nova/conductor/", line 359, in build_instances request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/", line 343, in wrapped return func(*args, **kwargs) File "/opt/stack/nova/nova/scheduler/client/", line 52, in select_destinations context, request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/client/", line 37, in __02:23
bardiarun_method return getattr(self.instance, __name)(*args, **kwargs) File "/opt/stack/nova/nova/scheduler/client/", line 34, in select_destinations context, request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/", line 122, in select_destinations request_spec=request_spec, filter_properties=filter_properties) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py02:23
bardia", line 158, in call retry=self.retry) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/", line 90, in _send timeout=timeout, retry=retry) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/", line 464, in send retry=retry) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/", line 455, in _send raise result02:23
bardiaNov. 6, 2015, 2:20 a.m.02:23
bardiaanyone faced the same issue?02:23
*** heyongli has joined #openstack-infra02:23
clarkbbardia please dont paste like that, use a paste service. you probably dont have enough ram02:23
clarkbcheck the nova scheduler log02:23
bardiaclarkb: sorry will use paste.openstack next time02:23
bardiaclarkb: I have 16 GB of ram02:24
*** sridhar_ram has joined #openstack-infra02:24
clarkbcheck the nova scheduler log02:24
*** asettle has joined #openstack-infra02:27
*** salv-orlando has joined #openstack-infra02:27
*** sridhar_ram1 has joined #openstack-infra02:27
*** yamamoto has joined #openstack-infra02:28
*** sridhar_ram has quit IRC02:28
*** Daisy has joined #openstack-infra02:28
*** gyee has quit IRC02:31
*** heyongli has quit IRC02:32
*** heyongli has joined #openstack-infra02:33
bardiaclarkb: yeah didn't have enough hdd for m1.large02:33
bardiaclarkb: thanks for your hel02:33
*** bpokorny_ has joined #openstack-infra02:33
*** mtanino has quit IRC02:34
*** bpokorny has quit IRC02:37
*** bpokorny_ has quit IRC02:38
*** barnaby has quit IRC02:39
*** heyongli has quit IRC02:42
*** Daisy has quit IRC02:42
*** heyongli has joined #openstack-infra02:44
*** rfolco has quit IRC02:44
*** otter768 has quit IRC02:44
*** sridhar_ram1 has quit IRC02:45
*** inteq has joined #openstack-infra02:45
openstackgerritLiuqing Jing proposed openstack-infra/project-config: Add Evoque client project
*** mahatic has joined #openstack-infra02:45
*** sabeen1 has joined #openstack-infra02:49
*** salv-orlando has quit IRC02:49
*** rebase has quit IRC02:51
*** barnaby has joined #openstack-infra02:52
*** heyongli has quit IRC02:53
*** heyongli has joined #openstack-infra02:54
jesusaurusjhesketh: why would it be better to build a new generator class?02:54
*** btully has quit IRC02:55
*** wenlock has joined #openstack-infra03:00
*** heyongli has quit IRC03:00
*** Daisy has joined #openstack-infra03:04
jheskethjesusaurus: well not necessarily a whole new class, but a generator so that we can stream the contents as they are read rather than reading them all and then passing them to the wsgi app03:05
*** btully has joined #openstack-infra03:05
jesusaurusjhesketh: but you need to read the entire contents in order to return only the last X bytes03:05
jheskethjesusaurus: for that use case, yes, but in the case of reading the first X bytes, they could be done in parts03:06
*** Daisy has quit IRC03:08
*** rguillebert has quit IRC03:08
jesusaurusjhesketh: so you think it would be better if the type of generator created happened after the logic of determining what kind of range should be returned?03:10
*** EricGonczer_ has quit IRC03:13
jheskethjesusaurus: that might make it a bit easier... but you could just solve it for the general case and it shouldn't be any less efficient03:14
jheskethconsider if somebody wants the bytes from A -> X-1 where X is the last byte03:14
jheskeththere is no way to tell they wanted nearly all of them, so you may as well start scrolling through until you are ready to yield and continue from there03:14
jheskethoh, I see the issue, needing to determine the negatives... yes, handling that case differently would probably be best03:15
jheskethit could, however, be optimised by looking at the size of the file on disk or from swift03:15
jheskethwhich I believe is passed through03:15
clarkbyou cant03:16
clarkbbecause we add so much junk03:16
jheskethah yes, you're right03:16
clarkband its line based03:16
clarkbwhich swidt and disk dont count for you03:17
clarkbso ypu have to iterate through regardless03:18
*** salv-orlando has joined #openstack-infra03:19
clarkbI dont thinkwe get better performance changing it03:19
*** baoli has joined #openstack-infra03:19
*** rebase has joined #openstack-infra03:19
*** rebase has quit IRC03:19
*** tnarg_ has quit IRC03:20
jheskethclarkb: we do in the case where it's not a negative amount requested03:20
jheskethwe can start streaming the results rather than bufferring it all and stop once we reach either the end or the amount of bytes requested03:20
clarkbarent they equivalent?03:21
*** sabeen1 has quit IRC03:21
*** bardia has quit IRC03:21
jheskethyou have to wait for osloganalyze to compute the end before you start receiving a respons03:21
*** otter768 has joined #openstack-infra03:22
clarkbok so you want the appemds changed to yields?03:22
jheskethright, rather than building a list, yield it out03:23
*** boris-42 has joined #openstack-infra03:24
jesusaurusbut then we would need to special case bytes=-5 rather than a general solution, and only to optimize the first byte received, the last byte would still be pretty much the same03:29
clarkbjesusaurus you could put the yielding in an inner method that is returned if not negative index else convert to list and count backwards03:30
clarkbso a fairly small special case but yes would be special03:30
*** Daisy has joined #openstack-infra03:32
*** baoli has quit IRC03:36
*** sdake has quit IRC03:36
*** Daisy has quit IRC03:37
*** salv-orlando has quit IRC03:40
*** skolekonov has quit IRC03:48
*** mohankumar has joined #openstack-infra03:48
*** skolekonov has joined #openstack-infra03:50
*** aukhan has joined #openstack-infra03:54
*** Daisy has joined #openstack-infra03:54
*** mohankumar has quit IRC03:56
*** baoli has joined #openstack-infra03:57
*** Daisy has quit IRC03:57
*** Sree has joined #openstack-infra03:58
*** links has joined #openstack-infra03:59
*** tlian has quit IRC04:10
*** tiswanso has quit IRC04:14
*** denisra has quit IRC04:18
*** denisra has joined #openstack-infra04:18
*** fawadkhaliq has joined #openstack-infra04:20
*** FallenPegasus has quit IRC04:22
*** rlandy has quit IRC04:28
*** otter768 has quit IRC04:29
*** sdake has joined #openstack-infra04:35
*** wenlock_ has joined #openstack-infra04:35
*** FallenPegasus has joined #openstack-infra04:38
*** salv-orlando has joined #openstack-infra04:40
*** wuhg has joined #openstack-infra04:43
*** salv-orlando has quit IRC04:49
*** amotoki has joined #openstack-infra04:51
*** baoli has quit IRC04:51
*** sdake has quit IRC04:53
*** sdake has joined #openstack-infra04:58
*** vilobhmm11 has joined #openstack-infra04:59
*** Ramanjaneya has joined #openstack-infra04:59
*** ybathia has joined #openstack-infra05:01
*** vilobhmm11 has quit IRC05:02
*** ybathia_ has joined #openstack-infra05:02
*** Liuqing has quit IRC05:05
*** ybathia has quit IRC05:06
*** ybathia_ is now known as ybathia05:06
*** Liuqing has joined #openstack-infra05:07
*** btully has quit IRC05:09
*** mohankumar has joined #openstack-infra05:14
*** vilobhmm11 has joined #openstack-infra05:20
*** wenlock_ has quit IRC05:25
*** asettle has quit IRC05:27
*** mohankumar has quit IRC05:27
*** mrmartin has joined #openstack-infra05:29
*** mohankumar has joined #openstack-infra05:31
*** nadya has joined #openstack-infra05:39
*** heyongli has joined #openstack-infra05:40
*** heyongli has quit IRC05:44
*** heyongli has joined #openstack-infra05:44
*** nikhil_k has joined #openstack-infra05:48
*** nikhil has quit IRC05:48
*** mahatic_ has joined #openstack-infra05:50
*** btully has joined #openstack-infra05:53
*** mahatic has quit IRC05:54
*** wenlock1 has joined #openstack-infra05:54
*** heyongli has quit IRC05:54
*** heyongli has joined #openstack-infra05:55
*** bdemers has quit IRC05:57
*** sdake has quit IRC06:00
*** sdake has joined #openstack-infra06:00
openstackgerritEli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum
*** BharatK has joined #openstack-infra06:03
*** oomichi_ has quit IRC06:03
*** heyongli has quit IRC06:05
*** heyongli has joined #openstack-infra06:05
*** bdemers has joined #openstack-infra06:07
*** hdd has joined #openstack-infra06:08
*** legoktm has left #openstack-infra06:09
*** nadya has quit IRC06:12
openstackgerritEli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum
*** heyongli has quit IRC06:15
*** heyongli has joined #openstack-infra06:15
*** mrmartin has quit IRC06:16
*** vilobhmm111 has joined #openstack-infra06:18
*** vilobhmm11 has quit IRC06:19
openstackgerritMerged openstack-infra/project-config: Update vitrage-specs
*** heyongli has quit IRC06:21
*** heyongli has joined #openstack-infra06:22
*** unicell has joined #openstack-infra06:24
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** heyongli has quit IRC06:25
*** heyongli has joined #openstack-infra06:25
*** harlowja_at_home has joined #openstack-infra06:26
*** heyongli has quit IRC06:27
*** yonglihe has joined #openstack-infra06:27
*** salv-orlando has joined #openstack-infra06:29
*** otter768 has joined #openstack-infra06:30
*** openstackgerrit has quit IRC06:31
*** FallenPegasus has quit IRC06:32
*** openstackgerrit has joined #openstack-infra06:32
openstackgerritEli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum
*** otter768 has quit IRC06:34
*** yonglihe has quit IRC06:35
*** yonglihe has joined #openstack-infra06:36
*** harlowja_at_home has quit IRC06:37
*** david-lyle has joined #openstack-infra06:40
*** salv-orlando has quit IRC06:41
*** tonytan4ever has quit IRC06:44
*** peterlisak has joined #openstack-infra06:44
*** david-lyle has quit IRC06:45
*** yonglihe has quit IRC06:46
*** yonglihe has joined #openstack-infra06:46
openstackgerritEli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum
*** mahatic_ has quit IRC06:47
*** mahatic has joined #openstack-infra06:49
*** fedexo has joined #openstack-infra06:51
*** gildub has quit IRC06:53
*** vilobhmm111 has quit IRC06:54
*** yonglihe has quit IRC06:56
*** yonglihe has joined #openstack-infra06:56
openstackgerritgreghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror
*** jamielennox is now known as jamielennox|away06:59
*** baoli has joined #openstack-infra07:03
*** unicell1 has joined #openstack-infra07:03
*** unicell has quit IRC07:04
*** yonglihe has quit IRC07:06
*** yonglihe has joined #openstack-infra07:06
*** ybathia has quit IRC07:07
*** baoli has quit IRC07:08
*** gampel has joined #openstack-infra07:10
*** Liuqing has quit IRC07:10
*** e0ne has joined #openstack-infra07:10
*** Liuqing has joined #openstack-infra07:11
*** sdake has quit IRC07:14
*** sdake has joined #openstack-infra07:14
*** yonglihe has quit IRC07:16
*** yonglihe has joined #openstack-infra07:17
*** salv-orlando has joined #openstack-infra07:17
*** fawadkhaliq has quit IRC07:19
*** salv-orlando has quit IRC07:19
*** salv-orlando has joined #openstack-infra07:20
*** e0ne has quit IRC07:21
*** eil397 has joined #openstack-infra07:25
*** andersonvom has quit IRC07:26
*** mohankumar has quit IRC07:26
*** sdake has quit IRC07:26
*** yonglihe has quit IRC07:27
*** yonglihe has joined #openstack-infra07:27
*** sdake has joined #openstack-infra07:27
*** wenlock1 has quit IRC07:30
*** rossella_s has joined #openstack-infra07:33
*** fawadkhaliq has joined #openstack-infra07:34
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Add job for the ovs mechanism driver with native mode
*** Daisy has joined #openstack-infra07:36
*** someara2 has joined #openstack-infra07:36
*** yonglihe has quit IRC07:37
*** abregman has joined #openstack-infra07:37
*** yonglihe has joined #openstack-infra07:37
*** k4n0 has joined #openstack-infra07:39
*** sdake_ has joined #openstack-infra07:40
*** someara2 has quit IRC07:41
*** someara2 has joined #openstack-infra07:41
*** sdake has quit IRC07:42
*** someara2 has quit IRC07:45
*** someara2 has joined #openstack-infra07:46
*** yonglihe has quit IRC07:47
*** yonglihe has joined #openstack-infra07:47
*** shardy has joined #openstack-infra07:49
*** fawadkhaliq has quit IRC07:52
*** cody-somerville has joined #openstack-infra07:55
*** Guest65445 is now known as tdurakov07:56
tdurakovclarkb, hi, are you there?07:56
*** yonglihe has quit IRC07:57
*** yonglihe has joined #openstack-infra07:58
*** fawadkhaliq has joined #openstack-infra07:58
*** ihrachys has joined #openstack-infra07:58
*** mrmartin has joined #openstack-infra07:59
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets
*** devvesa has joined #openstack-infra08:03
*** someara2 has quit IRC08:04
*** eil397 has quit IRC08:06
*** yonglihe has quit IRC08:08
*** yonglihe has joined #openstack-infra08:08
*** matrohon has joined #openstack-infra08:11
*** fedexo has quit IRC08:13
*** yonglihe has quit IRC08:18
*** yonglihe has joined #openstack-infra08:18
*** HeOS has quit IRC08:19
*** placko has joined #openstack-infra08:20
*** mohankumar has joined #openstack-infra08:23
*** gampel has quit IRC08:24
*** kiran-r has joined #openstack-infra08:27
*** mohankumar has quit IRC08:28
*** yonglihe has quit IRC08:28
*** yonglihe has joined #openstack-infra08:28
*** otter768 has joined #openstack-infra08:31
openstackgerrityolanda.robla proposed openstack-infra/puppet-graphite: Add support for apache >= 2.4
*** sdake_ has quit IRC08:33
*** binbincong has quit IRC08:34
*** btully has quit IRC08:34
*** abregman has quit IRC08:35
*** otter768 has quit IRC08:36
*** zigo has quit IRC08:37
*** arxcruz has joined #openstack-infra08:37
*** Daisy has quit IRC08:38
*** ZZelle has quit IRC08:38
*** yonglihe has quit IRC08:38
*** ZZelle has joined #openstack-infra08:38
*** Daisy has joined #openstack-infra08:38
*** yonglihe has joined #openstack-infra08:39
*** aeng has quit IRC08:42
*** zigo has joined #openstack-infra08:42
*** openstackstatus has quit IRC08:42
*** Daisy has quit IRC08:43
*** jlanoux has joined #openstack-infra08:44
*** binbincong has joined #openstack-infra08:45
*** hashar-away is now known as hashar08:45
*** yonglihe has quit IRC08:48
openstackgerritMerged openstack-infra/project-config: Change acl for freezer* projects
*** yonglihe has joined #openstack-infra08:49
*** jordanP has joined #openstack-infra08:51
*** ifarkas has joined #openstack-infra08:51
*** Daisy has joined #openstack-infra08:52
*** Daisy has quit IRC08:54
*** kengo_sakai has joined #openstack-infra08:55
*** sergmelikyan has joined #openstack-infra08:57
*** yonglihe has quit IRC08:58
*** fhubik has joined #openstack-infra08:59
*** yonglihe has joined #openstack-infra08:59
*** Ala has joined #openstack-infra08:59
*** dgonzalez has quit IRC09:03
*** dgonzalez has joined #openstack-infra09:03
*** nadya has joined #openstack-infra09:07
*** yonglihe has quit IRC09:07
openstackgerritMerged openstack-infra/project-config: Run Ironic grenade jobs with IPA
*** links has quit IRC09:11
*** derekh has joined #openstack-infra09:11
*** Liuqing has quit IRC09:12
*** sorantis has joined #openstack-infra09:13
*** nadya has quit IRC09:13
sorantisCould you guys help me with updating gerrit group members?09:14
sorantisthe project I’ve created is
sorantiskingbird-core and kingbird-release are the groups.09:14
sorantisCould you please add the users [sorantis, joehuang] to both?09:15
*** bauwser has joined #openstack-infra09:15
*** asselin_ has joined #openstack-infra09:18
*** jistr has joined #openstack-infra09:19
yolandahi sorantis, sure09:20
sorantisah, great! I was just going to write an email :)09:21
*** achanda has quit IRC09:21
yolandasorantis, do you have the change for project creation?09:21
yolandaso i can verify09:21
sorantisyou mean this one?
yolandasorantis, as project creation is done by Dimitri Mazmanov, i'd normally add him to that group09:23
sorantisthat’s me :)09:23
yolandaah ok :)09:23
sorantissuper! thank you!09:25
sorantisI still have only  +1/-1 though09:26
sorantisgot it now09:26
*** e0ne has joined #openstack-infra09:27
*** fhubik is now known as fhubik_brb09:28
*** HeOS has joined #openstack-infra09:30
*** bdemers has quit IRC09:30
yolandaah, you need to refresh cache09:32
*** camunoz has quit IRC09:33
*** sergmelikyan has quit IRC09:35
*** Sree has quit IRC09:36
*** Sree has joined #openstack-infra09:37
*** cdent has joined #openstack-infra09:38
*** mgrosz has joined #openstack-infra09:46
*** mgrosz has quit IRC09:47
*** fhubik_brb is now known as fhubik09:49
SamYapleo/ yolanda how are you09:58
*** zhenguo has quit IRC09:58
*** rguillebert has joined #openstack-infra10:00
*** nijaba has quit IRC10:01
*** nadya has joined #openstack-infra10:03
*** someara2 has joined #openstack-infra10:03
asselin_anyone know how to get past this error? OpenStackCloudException: Error fetching image list: HTTPMultipleChoices (HTTP 300) Requested version of OpenStack Images API is not available. (Inner Exception: )10:08
*** kiran-r has quit IRC10:08
*** fawadkhaliq has quit IRC10:10
asselin_mordred, ^^ ?10:11
*** mfalatic has joined #openstack-infra10:13
*** jyuso1 has quit IRC10:15
*** mfalatic_ has quit IRC10:15
openstackgerritThomas Morin proposed openstack-infra/project-config: Rename project networking-bagpipe(-l2)
*** jyuso1 has joined #openstack-infra10:16
openstackgerritJulien Danjou proposed openstack-infra/project-config: Move #openstack-ceilometer to #openstack-telemetry
*** julim_ has joined #openstack-infra10:19
*** tdurakov has quit IRC10:20
*** Guest13749 has joined #openstack-infra10:20
*** tdurakov has joined #openstack-infra10:21
*** julim has quit IRC10:22
openstackgerritJohn Garbutt proposed openstack-infra/project-config: Add tempest-dsvm-lxc as experimental
*** otherwiseguy has quit IRC10:28
*** tdurakov has quit IRC10:29
*** aukhan has quit IRC10:30
*** aukhan has joined #openstack-infra10:30
openstackgerritSam Betts proposed openstack-infra/project-config: Disable py34 for networking-cisco kilo and juno
*** otherwiseguy has joined #openstack-infra10:30
openstackgerritJulien Danjou proposed openstack-infra/project-config: Move #openstack-ceilometer to #openstack-telemetry
*** sergmelikyan has joined #openstack-infra10:31
*** otter768 has joined #openstack-infra10:32
*** someara2 has quit IRC10:34
*** ldnunes has joined #openstack-infra10:36
*** otter768 has quit IRC10:36
asselin_mordred, nevermind figured it out: needed profile:hp in clouds.yaml.10:36
*** Daisy has joined #openstack-infra10:37
*** bdemers has joined #openstack-infra10:37
*** ihrachys has quit IRC10:37
*** ihrachys has joined #openstack-infra10:38
*** sergmelikyan has quit IRC10:40
mordredasselin_: ++10:41
mordredasselin_: dude. where are you? I feel like you should not be awake10:41
*** Daisy has quit IRC10:41
asselin_mordred, jet lag10:41
asselin_I should be awake10:42
asselin_should not10:42
mordredjeblair, ruagair: my only preference for cauth was that it was what our enovance friends were using in software-factory already10:42
asselin_mordred, any tips for this? OpenStackCloudException: Image creation failed: 403 Forbidden: Access was denied to this resource. (HTTP 403) (Inner Exception: )10:43
*** dzamboni has joined #openstack-infra10:43
mordredasselin_: you have to request image uploads be turned on on your hp cloud account10:44
mordredasselin_: it's not enabled by default10:44
mordredjeblair, clarkb, ruagair: "I'm looking for additional contributers to take this project over! Email me (I'm bamuller on gmail) if you can help out. Thanks!"10:44
*** someara2 has joined #openstack-infra10:44
asselin_mordred, cool thanks10:44
mordredjeblair, clarkb, ruagair: so, if libapache-mod-auth-openid works, we might want to adopt it10:44
*** someara2 has quit IRC10:45
*** shardy has quit IRC10:46
mordredhowever, libapache-mod-auth-openid looks great10:47
*** yamamoto has quit IRC10:48
*** someara2 has joined #openstack-infra10:48
*** nadya has quit IRC10:49
*** jamielennox|away is now known as jamielennox10:49
*** Sree has quit IRC10:51
*** someara2 has quit IRC10:51
daemontoolHi all, how can I add myself or other the this group?,members10:52
*** someara2 has joined #openstack-infra10:54
*** electrofelix has joined #openstack-infra10:57
*** kozhukalov_ has joined #openstack-infra10:58
*** gildub has joined #openstack-infra10:59
*** pc_m has joined #openstack-infra11:01
*** sergmelikyan has joined #openstack-infra11:02
*** aysyd has joined #openstack-infra11:04
*** aukhan has quit IRC11:05
*** sorantis has quit IRC11:05
*** aysyd has quit IRC11:08
*** aysyd has joined #openstack-infra11:09
*** jaosorior has joined #openstack-infra11:10
*** jamielennox is now known as jamielennox|away11:11
*** wenlock has quit IRC11:14
*** sergmelikyan has quit IRC11:14
*** someara2 has quit IRC11:14
*** achanda has joined #openstack-infra11:22
*** achanda has quit IRC11:26
*** mwagner has quit IRC11:27
*** dims has joined #openstack-infra11:27
openstackgerritDerek Higgins proposed openstack-infra/project-config: Set ZUUL_BRANCH and ZUUL_REF
*** yamamoto has joined #openstack-infra11:32
*** yamamoto has quit IRC11:32
*** yamamoto has joined #openstack-infra11:32
*** yamamoto_ has joined #openstack-infra11:33
openstackgerritSean Dague proposed openstack-infra/project-config: Create jobs for a wheel mirror
*** yamamoto has quit IRC11:36
openstackgerritMonty Taylor proposed openstack-infra/shade: Let os-client-config handle session creation
ihrachyshi folks. I need some advice on ZUUL_BRANCH variable. I see that it's passed in gate lbaas jobs for Liberty but not in periodic stable jobs. That results in the latter jobs to fetch master neutron and run against it (and fail). I wonder whether there is some reason the var is not passed into periodic jobs, and if not, how can I set it?11:43
*** pbourke has joined #openstack-infra11:45
daemontoolanyone around can help me understand how can I add members to this group?,members11:50
*** Liuqing has joined #openstack-infra11:54
openstackgerritMonty Taylor proposed openstack/requirements: Bump upper-constraint for os-client-config
*** baoli has joined #openstack-infra12:05
*** dprince has joined #openstack-infra12:06
*** fhubik is now known as fhubik_brb12:08
*** baoli has quit IRC12:09
*** rfolco has joined #openstack-infra12:09
*** gampel has joined #openstack-infra12:11
*** Daisy has joined #openstack-infra12:13
*** Daisy has quit IRC12:14
*** sorantis has joined #openstack-infra12:17
*** gampel has quit IRC12:18
*** pc_m has quit IRC12:20
*** pc_m has joined #openstack-infra12:20
*** Sree has joined #openstack-infra12:22
*** aysyd has quit IRC12:23
*** aysyd has joined #openstack-infra12:24
*** achanda has joined #openstack-infra12:24
*** smarcet has joined #openstack-infra12:24
*** rossella_s has quit IRC12:25
*** k4n0 has quit IRC12:27
*** dingyichen has quit IRC12:29
*** achanda has quit IRC12:30
*** wuhg has quit IRC12:31
*** someara2 has joined #openstack-infra12:31
*** otter768 has joined #openstack-infra12:32
*** zigo has quit IRC12:36
*** zigo has joined #openstack-infra12:37
*** abregman has joined #openstack-infra12:37
*** otter768 has quit IRC12:37
*** weihan has joined #openstack-infra12:38
*** fhubik_brb is now known as fhubik12:39
*** rossella_s has joined #openstack-infra12:41
*** Sree has quit IRC12:43
anteayadaemontool: first a gerrit admin has to add the owner of the patch that created that group12:43
*** Sree has joined #openstack-infra12:43
anteayadaemontool: then the first person in the group can add others12:44
openstackgerritSean Dague proposed openstack-infra/project-config: Create jobs for a wheel mirror
sdaguegreghaynes / fungi ^^^ - still not passing tests, but it's at a different failure now12:45
*** abregman has quit IRC12:45
*** thorst has joined #openstack-infra12:45
anteayasdague: morning12:45
anteayaso next week I was thinking of convening the first patch of folks following the learning how to debug the gate thread12:46
anteayasdague: was thinking of booking the sprint channel for two hours next thursday12:46
anteayasdague: might you have a window next thursday at all?12:46
anteayas/first patch/first batch12:47
*** tdurakov has joined #openstack-infra12:47
sdaguehonestly, I don't know. I'm still unpacking what I signed up for at summit.12:47
anteayafair enough12:47
anteayawell ideally I would pick a time that works for you, mriedem, and mtreinish12:48
anteayanot that you have to commit to the whole thing, but at least not conflicting with other things12:49
tdurakovsdague, hi12:49
anteayasdague: if you can let me know what works for you, that would be great12:49
anteayawasn't going to post to the list until next week12:49
tdurakovsdague, working on live-migration job for nova, need help with local env12:49
*** trown|outttypeww is now known as trown12:50
*** weihan has quit IRC12:51
*** kengo_sakai has quit IRC12:53
anteayafungi clarkb this patch for today's rename was respun: it looks fine to me, I have pinged mestery in -neutron to ask him to review12:56
anteayasorry I don't have time and attention to update the etherpad to include this patch12:57
openstackgerritAlexey Deryugin proposed openstack-infra/project-config: Add missing project openstack/yaql to zuul layout
*** boris-42 has quit IRC12:58
*** sorantis has quit IRC12:59
*** Ramanjaneya has quit IRC12:59
*** gordc has joined #openstack-infra13:00
anteayasdague: I worded that poorly, you don't have to commit to anything, but it would be great for me if you could attend13:01
*** gildub has quit IRC13:02
*** EricGonczer_ has joined #openstack-infra13:03
*** kgiusti has joined #openstack-infra13:06
*** mwagner has joined #openstack-infra13:06
*** thorst has quit IRC13:07
*** baoli has joined #openstack-infra13:07
*** jyuso1 has quit IRC13:08
*** sorantis has joined #openstack-infra13:10
*** spzala has quit IRC13:13
*** alivigni has joined #openstack-infra13:18
*** kozhukalov_ has quit IRC13:18
*** tlian has joined #openstack-infra13:21
*** wenlock has joined #openstack-infra13:25
openstackgerritAlexey Deryugin proposed openstack-infra/project-config: Add missing project openstack/yaql to zuul layout
*** dkranz has joined #openstack-infra13:28
*** Liuqing_ has joined #openstack-infra13:28
*** wenlock has quit IRC13:29
*** Liuqing has quit IRC13:31
*** nadya has joined #openstack-infra13:31
*** edmondsw has joined #openstack-infra13:32
*** thorst has joined #openstack-infra13:32
*** kozhukalov_ has joined #openstack-infra13:33
*** xyang1 has joined #openstack-infra13:37
freerunnerHey folks! Looks like when was committed, AJaeger forget about openstack/yaql repository. Here the patch, which add yaql back to zuul layout. .  <--- fungi clarkb sdague anteaya13:37
*** peterlisak has quit IRC13:40
*** peterlisak has joined #openstack-infra13:41
*** baoli_ has joined #openstack-infra13:41
*** ericksonsantos has quit IRC13:41
anteayafreerunner: thank you, +213:42
*** baoli has quit IRC13:44
*** cloudtrainme has joined #openstack-infra13:45
*** EricGonc_ has joined #openstack-infra13:46
*** ericksonsantos has joined #openstack-infra13:47
*** peterlisak has quit IRC13:47
*** peterlisak13 has joined #openstack-infra13:47
*** regXboi has joined #openstack-infra13:47
*** spzala has joined #openstack-infra13:48
*** inteq has quit IRC13:49
*** EricGonczer_ has quit IRC13:50
*** pradk has joined #openstack-infra13:50
*** Shrews has quit IRC13:50
*** salv-orlando has quit IRC13:51
*** salv-orlando has joined #openstack-infra13:51
openstackgerritSean Dague proposed openstack-infra/devstack-gate: multinode and grenade sanity checking.
anteayamestery: thank you13:54
*** e0ne has quit IRC13:55
*** Shrews has joined #openstack-infra13:55
tristanCmordred: in all fairness, cauth is still being used and developed, we are looking for supporting more auth backends13:56
mordredtristanC: yah. that it has active developers is one of the nice things13:56
*** e0ne has joined #openstack-infra13:56
mordredruagair: perhaps also connecting with tristanC about cauth directly is useful/worthwhile?13:56
tristanCmordred: and other improvement like supporting same user's mail from different backend like github and launchpad mail conflicts13:57
tristanCruagair: mordred: well sure it's opensource and user story are more than welcome fwiw13:57
openstackgerritMerged openstack-infra/project-config: Add missing project openstack/yaql to zuul layout
*** rossella_s has quit IRC13:59
*** rossella_s has joined #openstack-infra14:00
*** tdurakov has quit IRC14:00
SamYaplecan I get someone to ack ? adding the devstack-logs publisher for kolla14:00
*** tdurakov has joined #openstack-infra14:02
*** tiswanso has joined #openstack-infra14:03
*** kushal has joined #openstack-infra14:04
*** ddieterly has quit IRC14:05
pabelangerilyashakhat: ping. Was looking to get some info about the system requirements for  Specifically, how much RAM are you running14:07
*** akshai has joined #openstack-infra14:07
mhutristanC, moravec ruagair : cauth might be a bit overkill for what you want to do, indeed. If an application doesn't support your auth protocol of choice, chances are you can simply delegate this to apache with the right mod and use REMOTE_USER14:08
mhutristanC, mordred ruagair : our use case with software factory was that we didn't want to configure each service we ship for the auth, depending on whether or not they support our auth protocol of choice14:09
mhutristanC, mordred ruagair so instead, each service will use REMOTE_USER + cauth, cauth holding the auth config for every service in one place14:10
* anteaya is traveling the rest of the day14:11
mhutristanC, mordred ruagair which is convenient when people deploying Software Factory want to choose the auth protocol(s) they want to offer14:12
*** amitgandhinz has joined #openstack-infra14:13
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API
*** Piet has quit IRC14:14
mordredmhu: indeed. whereas pluggable backend auth strategies are not very important for infra itself, as there is one and only one auth14:15
*** sdake has joined #openstack-infra14:16
*** hdd has quit IRC14:17
daemontoolanteaya,  ty14:17
*** rlandy has joined #openstack-infra14:17
*** mriedem has joined #openstack-infra14:18
daemontoolyolanda, or anyone can add me as admin to thie group,members as per:
*** salv-orlando has quit IRC14:22
*** salv-orlando has joined #openstack-infra14:26
*** placko has quit IRC14:30
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Add a job around using tripleo with docker
*** jsavak has joined #openstack-infra14:32
*** Sree has quit IRC14:33
*** otter768 has joined #openstack-infra14:33
*** nadya has quit IRC14:33
*** ddieterly has joined #openstack-infra14:36
*** btully has joined #openstack-infra14:36
*** bswartz has joined #openstack-infra14:36
*** tdurakov has quit IRC14:36
*** yfried has quit IRC14:36
*** jsavak has quit IRC14:37
*** otter768 has quit IRC14:38
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Add a job around using tripleo with docker
yolandahi daemontool, done14:39
matrohoncan anyone help me to understand why some patches don't change the bug status in launchpad?14:41
*** zz_gondoi is now known as gondoi14:41
matrohonthis is the patch :
*** jsavak has joined #openstack-infra14:42
openstackgerritAntoine Musso proposed openstack-infra/gear: inline helps for interactive commands
*** achanda has joined #openstack-infra14:46
*** achanda has quit IRC14:51
*** alaski is now known as lascii14:52
ddieterlycan we get someone to review
*** wenlock has joined #openstack-infra14:53
*** Sree has joined #openstack-infra14:53
daemontoolyolanda,  thank you14:54
ddieterlyalternatively, explain the process that we need to go thru to get patches to openstack-infra/project-config reviewed14:54
derekhHi, the tripleo periodic jobs that are now running fail because ZUUL_BRANCH isn't set (I think), if anybody has a chance can you take a look at this patch that I think will fix it
*** burgerk has joined #openstack-infra14:57
*** wenlock has quit IRC14:57
*** Sree has quit IRC14:58
fungimatrohon: lack of sufficient permissions. you need to add the "OpenStack Infra (hudson-openstack)" user to
*** larainema has joined #openstack-infra14:59
fungionly bug supervisors can reassign a bug14:59
matrohonfungi : done, thanks15:00
*** Piet has joined #openstack-infra15:01
*** wenlock has joined #openstack-infra15:03
*** tdurakov has joined #openstack-infra15:03
matrohonfungi : do I need to wait so that the hudson-openstack user in the bug-supervisor team is able to reassign bug?15:08
matrohonfungi : I just updated to test the change15:08
*** skolekonov has quit IRC15:10
*** amitgandhinz has quit IRC15:15
*** Liuqing_ has quit IRC15:18
*** amotoki has quit IRC15:19
*** tongli has joined #openstack-infra15:20
*** mtanino has joined #openstack-infra15:21
*** nadya has joined #openstack-infra15:22
*** wenlock_ has joined #openstack-infra15:22
*** wenlock has quit IRC15:24
fungimatrohon: it only leaves a comment in the bug for patchset #1 (so as to avoid spamming the bug on changes with lots of patchsets)15:25
matrohonfungi : ok make sense15:25
*** wenlock has joined #openstack-infra15:26
*** tjones has joined #openstack-infra15:27
ilyashakhatpabelanger: ping15:28
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: DO NOT MERGE -- CI performance debugging
*** sfinucan has joined #openstack-infra15:31
*** sileht has quit IRC15:32
*** dansmith is now known as superdan15:33
*** tonytan4ever has joined #openstack-infra15:37
*** ajmiller has joined #openstack-infra15:39
openstackgerritKyle Mestery proposed openstack-infra/project-config: Correct ACLs for Neutron sub-projects
*** e0ne has quit IRC15:46
*** FallenPegasus has joined #openstack-infra15:48
*** ddieterl_ has joined #openstack-infra15:49
*** cloudtrainme has quit IRC15:51
*** sridhar_ram has joined #openstack-infra15:52
*** Liuqing has joined #openstack-infra15:52
*** ayoung has joined #openstack-infra15:53
*** ddieterly has quit IRC15:53
openstackgerritMerged openstack-infra/infra-specs: Update priorities for the Mitaka cycle
*** Liuqing has quit IRC15:55
*** Sree has joined #openstack-infra15:55
*** vilobhmm11 has joined #openstack-infra15:57
fungisdague: the problem with DISTRO_NAME in {} on 164927 was that the builder was being used in a template so needed {{DISTRO_NAME}} to keep it from looking like a replacement parameter15:58
*** samuelBartel has joined #openstack-infra15:59
*** Sree has quit IRC16:01
*** amitgandhinz has joined #openstack-infra16:02
*** jsavak has quit IRC16:02
*** sdake_ has joined #openstack-infra16:03
matrohonfungi : actually what I don't understand is why some changes update launchpad correctly (242143), and others don't (242437)16:03
*** freyes has quit IRC16:04
*** jsavak has joined #openstack-infra16:04
matrohonfungi, I assume that it's because in 242143, the Closes-bug statement was correct since the first patch16:04
fungimatrohon: because the script makes a set of api calls to leave a comment, adjust the bug status and reassign it. of those, reassignment at least requires being a bug supervisor, so if the bug isn't already assigned to the correct user then the other api calls are also aborted16:04
*** freyes has joined #openstack-infra16:05
*** woodster_ has joined #openstack-infra16:05
greghaynesfungi: not sure if you saw my new rev / comment on
fungimatrohon: it's not a particularly robust script, and basically dies in flames the moment it encounters a launchpad api error16:06
*** sdake has quit IRC16:06
matrohonfungi : ok, hope it will works fine now, thanks16:07
fungigreghaynes: while my puppet-fu too is weak, i believe that's simply adding the authorized key so that the jenkins master can connect and run jobs on it16:07
fungibut not setting up keys to allow that slave to authenticate to the mirror servers16:08
greghaynesso we need another key16:08
greghaynesIs there somewhere else we do that?16:08
*** jaypipes is now known as leakypipes16:09
*** fedexo has joined #openstack-infra16:09
fungigreghaynes: on proposal.slave.o.o we install an ssh keypair to be able to authenticate to gerrit so it can push patches16:10
greghaynesfungi: awesome, ty16:10
fungithat's the most similar configuration we have that comes to mind anyway16:11
*** yamamoto_ has quit IRC16:12
*** hemnafk is now known as hemna16:12
*** ifarkas has quit IRC16:14
*** achanda has joined #openstack-infra16:20
*** hashar has quit IRC16:20
*** tdurakov has quit IRC16:22
greghaynesfungi: Looks like that works by just referring to a keypair in hiera, is that something infra rooters generate by hand and instll?16:23
greghaynesinto the hiera db16:23
*** sorantis has quit IRC16:25
*** derekh has quit IRC16:26
*** freyes has quit IRC16:26
*** mriedem is now known as mriedem_away16:27
*** sdake_ is now known as sdake16:28
*** zxiiro has quit IRC16:29
*** ihrachys has quit IRC16:29
*** luqas has quit IRC16:29
*** zxiiro has joined #openstack-infra16:30
fungigreghaynes: yep. we just use ssh-keygen to create the rsa keypair and then stash the two values in hiera16:30
greghaynesfungi: makes sense, easy enough to do16:30
fungialso it would be good if we could also puppet the known_hosts entries for the mirror servers onto the mirror slaves16:30
*** hdd has joined #openstack-infra16:30
*** sabeen has joined #openstack-infra16:31
fungiso that we don't have to go manually accept all of them by hand16:31
greghaynesah, yea. I'll loo at that16:31
greghayneslook, even16:31
*** cloudtrainme has joined #openstack-infra16:31
fungicodesearch.o.o for known_hosts will probably turn up some examples of where we already do that16:32
mordredcodesearch is the best16:32
*** bpokorny has joined #openstack-infra16:33
*** otter768 has joined #openstack-infra16:34
fungithanks again, taron!16:35
*** vilobhmm11 has quit IRC16:35
*** jaosorior has quit IRC16:35
*** vilobhmm11 has joined #openstack-infra16:36
fungidid we ever announce that service officially?16:36
*** cloudtrainme has quit IRC16:36
fungii think it wrapped up during the summit so maybe we haven't made a splash with it yet16:37
*** tnarg has joined #openstack-infra16:37
jeblairdon't believe so16:37
*** HenryG has quit IRC16:37
*** e0ne has joined #openstack-infra16:38
fungii'll let taron have the honor of posting the availability announcement to the -infra and -dev mailing lists if interested16:38
*** sridhar_ram has quit IRC16:38
fungihappy to proof-read a draft and make suggestions too16:39
*** HenryG has joined #openstack-infra16:39
*** otter768 has quit IRC16:39
*** jsavak has quit IRC16:39
*** sridhar_ram has joined #openstack-infra16:39
*** jsavak has joined #openstack-infra16:40
*** matrohon has quit IRC16:41
*** sileht has joined #openstack-infra16:41
greghaynesfungi: hrm, for the known_hosts thing, we would need to know the host key beforehand16:42
fungigreghaynes: we already do16:42
greghaynesfungi: oh?16:42
fungithose servers exist16:42
greghayneswell then :)16:43
*** achanda has quit IRC16:45
pabelangerpleia2: just need to run my daughter to school, but it looks like the 15GB flavor will work for stackalytics.o.o.  Have some free time today to launch it?16:46
pabelangerilyashakhat: pong16:46
pabelangerilyashakhat: seen your email16:46
fungigreghaynes: ssh-keyscan (et cetera)16:47
greghaynesfungi: Yea, the existing solution pulls it out of hiera (I am guessing that is better in case we switch the host to a new place?)16:47
*** freyes has joined #openstack-infra16:48
fungigreghaynes: i'm torn on things like host key hashes and public keys going into our secret hiera. up to you for now. when we have a public/private hiera split of course it makes a lot of sense to keep data like that in yaml in git16:48
*** dangers_away is now known as dangers16:48
greghaynesYep, usually there is a some kind of public environment for these kinds of tools...16:49
fungiin an ideal world, the secret hiera would just be private keys, passwords, et cetera16:49
*** vilobhmm11 has quit IRC16:49
*** vilobhmm11 has joined #openstack-infra16:49
*** nadya has quit IRC16:50
sdaguefungi: ok, gotcha. Well hopefully you and greghaynes have it all sorted now16:51
fungisdague: yeah, it's very close now--thanks for hacking on it!16:51
greghaynesYep, should be close16:52
*** samuelBartel has quit IRC16:52
*** gyee has joined #openstack-infra16:52
greghaynesThis thought had never occured to me until you pointed it out yesterday, but now that we have the transitive deps defined in requirements there is actually 0 reason for pypi-mirror to exist16:53
*** vilobhmm11 has quit IRC16:54
*** sridhar_ram has quit IRC16:54
*** tjones has quit IRC16:54
*** achanda has joined #openstack-infra16:56
greghaynesSo it might be time to let that project die... again.16:56
*** sridhar_ram has joined #openstack-infra16:56
*** tjones has joined #openstack-infra16:56
*** Sree has joined #openstack-infra16:57
*** ilyashakhat has quit IRC16:57
sdaguegah, rax images without libxml2 in them ....16:57
*** fhubik has quit IRC16:57
fungigreghaynes: well, it's not just that we have resolved lists of transitive deps, but also that pip's wheel integration/caching has gotten better16:57
fungiif this was still circa pip 1.5/1.6 i don't think things would be a ssimple16:57
fungisdague: in all regions or just one that you can find? i'll delete the current images to roll back to working ones16:58
greghaynesfungi: hah, yea16:58
sdaguefungi: well, it's in one, and it's triggered on a subnode16:58
sdaguebut it made me realize we needed to fix devstack anyway16:59
*** sputnik13 has joined #openstack-infra16:59
*** sputnik13 has quit IRC16:59
fungisdague: oh, yeah subnode means multi-node means devstack means we don't normally preinstall devstack dependencies on those images16:59
sdaguebecause we're installing python-lxml, then uninstalling it all the time17:00
fungiso if libxml2 was installed on them previously, i think that was an oversight17:00
sdagueright, well, we need it17:00
sdaguen-cpu needs it because of the libvirt driver17:00
sdaguebut, anyway, we can fix it in devstack. upper constraints means that python-lxml is no longer useful to install as a package17:01
fungioh, sure, i don't question devstack needing it, just that devstack should be installing it17:01
sdaguein liberty and beyond17:01
*** Sree has quit IRC17:01
sdagueyep, I'm fine with that17:01
sdagueit's a little unfortunate, as it killed -
sdagueor more importantly, died early enough that we never ran it correctly17:01
mordredsdague: I keep meaning to go back to the pach of "never install python-* from distro packages ever" in devstack- are we still installing things from packages?17:02
sdaguemordred: a few things17:02
sdague cleans up most of them17:02
fungiso if devstack was relying on libxml2 existing preinstalled on our images, just saying that's a devstack bug and i don't know why we had libxml2 preinstalled nor what caused it to stop being preinstalled, though i can try to look into it17:02
sdaguefungi: yep, I'm completely ok with that17:02
sdaguethe patch is pushed that will fix it17:02
*** jsavak has quit IRC17:02
fungii guess it also needs backporting to devstack stable branches17:03
sdagueI don't need any investigation on this one, it was just a "gah"17:03
sdagueI backported to liberty17:03
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API
EmilienMI have a job which is hanging for a while now:
mordredsdague: cool17:03
EmilienMcan someone kill it ?17:03
sdaguepre liberty the packages should kick it up17:03
sdaguebecause we don't have upper constraints17:03
fungiahh, right-o17:04
mordredsdague: wow: files/debs/nova:python-m2crypto17:04
mordredwe haven't depended on m2crypto in like 3 years17:04
*** sputnik13 has joined #openstack-infra17:04
*** wenlock__ has joined #openstack-infra17:04
*** unicell1 has quit IRC17:04
sdaguewell, it's removed now :)17:04
*** jlanoux has quit IRC17:04
*** Hal1 has quit IRC17:05
EmilienMactually don't kill the job, but Zuul-Cloner looks very slow to clone our modules17:05
fungiyeah, i thought m2crypto was abandoned upstream and an acknowledged security risk. amusing it hung around in the list that long17:05
*** unicell has joined #openstack-infra17:05
*** eil397 has joined #openstack-infra17:05
mordredsdague: neat!17:05
mordredsdague: oh - I didn't follow your link before :)17:05
EmilienMfungi: have you noticed any issue with zuul-cloner recently ? See how it's so slow17:05
fungiEmilienM: it's possible we have an overloaded git server or two17:05
sdaguemordred: ... squirel!17:06
fungior an overloaded zuul merger perhaps17:06
mordredsdague: is there an equv patch for rpms?17:06
EmilienMfungi: ack - just wanted to make sure you're aware about that17:06
sdaguemordred: there is not yet, because I didn't want to go through all the analysis on what -dev things had to get added17:06
fungiEmilienM: yep, thanks... i'll start going through cacti graphs to see if i spot an obvious problem17:06
sdagueI will totally +2 one if you make it17:06
EmilienMfungi: I'm looking now17:07
*** wenlock__ has quit IRC17:08
openstackgerritMonty Taylor proposed openstack-infra/shade: Let os-client-config handle session creation
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove designate support
mordredfungi, Shrews, jeblair: ^^ 242095 is up for discussion and I could go either way on it - designate support literally does not work at all currently17:10
fungisdague: mordred: it's possible the list is mostly similar to what's in the bindep fallback list in project-config17:10
mordredthe unit tests work17:10
mordredbut it's a great example of the fact that the new rule needs to be "new API functions must come with functional tests"17:10
sdaguefungi: it is17:10
*** mfalatic has quit IRC17:11
sdagueI mostly wanted to get enough in to make these tests work, and if anyone else wants to optimize the rpm space, happy hacking17:11
mordredbut I could also see sticking to the letter of the law in terms of api contracts and still considering removing those bad, even though it is impossible for anyone to have ever used them17:11
clarkbmordred does designate even have >1 cloud deployment?17:12
clarkbcouldnt users use the designate client?17:12
mordredclarkb: no. the designate client constructor was the thing that was broken17:13
EmilienMfungi: i don't see issue on git servers17:13
EmilienMfungi: established tcp connections were high, but not a big deal17:13
clarkbmore specificallt I see shade as useful when multiple clouds need dealing with17:13
mordredclarkb: and yes, vexxhost has it deployed. however, they have v1 deployed, and we _also_ don't have a design for how to deal with the v1/v2 conceptual breakage17:13
fungiEmilienM: yep, that's common17:13
mordredclarkb: yah17:13
mordredso - there are 2 problems  a) it doesn't work b) we only support v2 in the current code-base, and there are zero clouds with v2 deployed17:14
mordredwith "support" being "you can list zones"17:14
greghaynesfungi: mordred I had a wierd thought - when we start doing ansible to run puppet everywhere... any reason to not have an ansible play to copy the ansible node's known hosts' file to everyone since it will by definition know who all the hosts are17:15
fungiEmilienM: your example run used zm01 and it looks not at all loaded either17:15
*** maiteb has joined #openstack-infra17:15
mordredgreghaynes: nope. no reason at all17:15
mordredgreghaynes: and we already do ansible to run puppet everywhere17:15
mordredgreghaynes: so we could do it now if we desired17:16
EmilienMfungi: I don't find any issue in our graphs, I continue to dig17:16
greghaynesI think that would remove all this known_hosts wierdness17:16
mordredgreghaynes: yah - the ongoing work is to move from running puppet agent to running puppet apply17:16
fungigreghaynes: yeah, a global /etc/ssh/ssh_known_hosts file is useful and simplifies lots of stuff17:16
fungii used to do that in other large operations where i worked17:16
*** arxcruz has quit IRC17:16
fungikeeping it in git is even more useful, because you can then point your sysadmins(/users too in some cases) to a list they can check against or even just reuse17:17
Shrewsmordred: yeah, i'd like to see it removed if it doesn't work. and ++ to requiring functional tests for new things17:18
fungiof course, publishing sshfp records could be even _more_ useful, but only in some future utopia where you have dnssec working everywhere17:18
greghaynesfungi: Yea, So maybe we copy the ansible node's in to git, then use that17:18
*** sergmelikyan has joined #openstack-infra17:18
fungigreghaynes: i would fully support that but for one problem--the list you're talking about is probably hashed. we want the normal ssh-keyscan output instead17:19
Shrewsmordred: i think we should give the author a heads-up notice about it, though17:19
mordredrcarrillocruz: ^^17:19
fungigreghaynes: however, a quick loop over our servers with ssh-keyscan (passing through a grep -v ^# to filter out comment lines) would be pretty quick to build what we need though17:19
greghaynesfungi: aye. Welp, sounds like a script to generate a good source of truth, put that in git, then an ansible or puppet module to convert that to the format we want17:19
mordredI have a questoin about server creation flow with that17:20
mordredcurrently we use launch_node.py17:20
mordredwhich is manual17:20
mordredwhich means that adding the host to the file after launching and submitting a change to git is a step the rooter who is launching can perform17:20
mordredhowever, once we have ansible-launch-node in place17:20
mordredwe have an issue that it'll be an ansible robot creating servers17:21
mordredso how do we get keys back into the git source?17:21
*** e0ne has quit IRC17:21
greghayneswe can have a bot propose the change on launch_node17:21
mordredwe could - but in this case when I say "bot" I mean "ansible cron job on puppetmaster.o.o"17:21
mordredwhich would mean we'd have puppetmaster submitting patches to gerrit17:22
mordredwhich I suppose is fine17:22 doesn't add the host key to ~root/.ssh/known_hosts (it's a manual step right now anyway), and only prevents subsequent puppet runs until such time as the global known_hosts in git is updated17:22
mordredbut ansible-launch-node will17:22
fungiahh, yeah that does then get trickier17:22
greghaynesYea, I think its not actually a technical issue, its more of a question of how we trust the data and how much auto approval we want17:22
nibalizerwhy is having a hashed known_hosts a problem?17:22
funginibalizer: makes it harder to match visually17:22
mordredright. well, we can't _actually_ trust the data at all until my patch to nova is written, approved, landed and deployed17:22
mordredbut, it's as good as we've got currently17:23
funginibalizer: the hashed version obscures the hostname17:23
greghaynesSo my answer would be that it is going to be manual until the trusted path problem goes away, which is unrelated to the automation tooling17:23
fungiand for reasons that are irrelevant to our public-facing infrastructure (we don't care if people get their hands on a list of our servers' dns names)17:23
nibalizerwe could turn off hashing in ~/.ssh/config17:24
greghaynesnibalizer: I would think that is something we want if were going to be making those files global17:24
nibalizerwe'd have to start from zero with a new known_hosts though17:24
fungiyep, which is ~= to the work required to loop ssh-keyscan over that list17:24
nibalizerfungi: something about running ssh-keyscan in a loop doesn't sit well with me, seems like we're basically giving up on host verification at that point, and depending only on dns17:26
funginibalizer: huh? we already tofu these servers17:26
nibalizerassuming its in a cronjob, i mean17:26
*** krtaylor has quit IRC17:26
greghaynesThat would be a one time17:26
fungioh, no i meant once to populate the initial list17:26
greghaynesand we can also verify it matches the existing known_hosts17:26
nibalizerah okay im on board then17:27
fungiperhaps with some effort (there is an ssh-keygen option to hash a known hosts file we could use to validate the initial list)17:27
*** Ala has quit IRC17:28
*** vponomaryov has left #openstack-infra17:28
pleia2pabelanger: yep, top of my list after I clean up the fantastic mess my coffee make made \o/17:29
*** peterlisak13 has quit IRC17:29
fungialso if we ever get to the point of having an openstack dns api for (or run our own master nameserver) then whatever automation we have to propose entries to the global known_hosts file on new server creation can also propose sshfp additions to our zone file17:31
*** tdurakov has joined #openstack-infra17:31
*** trown is now known as trown|lunch17:31
*** sputnik13 has quit IRC17:32
*** sputnik13 has joined #openstack-infra17:32
fungiclarkb: which one public cloud provider were you thinking of that has designate access for tenants?17:32
clarkbfungi: HPCloud17:32
mordredfungi: vexxhost also runs it17:32
clarkbits not going to exist for long but it does exist now17:32
fungiclarkb: got it. thought maybe you knew of one which wasn't about to go away17:33
mordredfungi: I can check with mohammed and see if he's going to upgrade to v2 any time soon17:33
fungiso with vexxhost, at least there will still be one known designate instance out there17:33
mordredfungi: if he is, I'd LOVE to suggest that we move our dns there17:33
mordredbut I do not think we should jump on v1 - v2 supports importing zone files directly17:34
openstackgerritSean Dague proposed openstack-infra/devstack-gate: multinode and grenade sanity checking.
*** tdurakov has quit IRC17:37
*** jordanP has quit IRC17:37
clarkbisn't v2 brand new and stillin development though?17:37
fungiyeah, i find that the battle-tested bind zone file format is easy to parse and update with scripts and also by hand17:38
mordredKiall: ^^ ?17:38
mordredfungi: you know - we could just run dns servers and use ansible launch-node to update them as part of the process17:38
fungimordred: yep, i even alluded to that earlier. but having our zonefiles in git might still be nice17:39
mordredfungi: indeed17:39
*** Swami has joined #openstack-infra17:39
*** krtaylor has joined #openstack-infra17:40
pleia2mordred: did you create a statalytics.o.o once? from 2014-06-08T17:08:20Z17:40
mordredfungi: although again that would give us a chicken-and-egg in terms of fully automated launch-node17:40
mordredpleia2: yes17:40
mordredpleia2: please feel free to delete/re-create17:40
pleia2mordred: thanks, on it :)17:40
fungimordred: not entirely chicken-and-egg there... puppet would run once during launch, we just wouldn't get continual puppeting until the host key update was approved17:41
pleia2we need a 15G node for this, so I'm ust going to delete and make a new one17:41
*** unicell has quit IRC17:42
fungimordred: also we could decree it acceptable to just single-core approve those updates, so whoever approves the server creation change can watch for and approve the known hosts update17:42
fungiand/or dns record addition17:42
mordredright - it's the dns record addition I was more worried about17:42
mordredit's "land this change, watch for a folow up patch, land that patch"17:43
*** dzamboni has quit IRC17:43
mordredwhich seems clunky when most of the info in the files is generated info ... maybe we could have supplimental/overlay files in git17:43
clarkbanteaya: fungi I am reviewing those project rename changes now17:44
clarkbthis is another place I feel the lack of tree tabs17:44
fungithe alternative, i guess is to simply propose those updates into gerrit as record keeping, and have the source of truth be the zone file on the master nameserver and the /etc/ssh/ssh_known_hosts on the launch master17:44
clarkbso the openstack health chnage changes the names of the groups17:44
*** mfalatic has joined #openstack-infra17:44
fungiclarkb: yeah, i ctrl-click all the urls and then go switching through all the resultant tabs in my sea of other tabs17:45
greghaynesclarkb: why dont you have tree tabs?17:45
*** nadya has joined #openstack-infra17:45
clarkbgreghaynes: ff 42 broke the plgin17:45
clarkbanteaya: fungi we should add a step to rename the group via gerrits API so we don't end up with two sets of groups17:45
fungiat least pentadactyl lets me switch between relative or arbitrary tabs with simple keystrokes17:45
greghaynesclarkb: oh. I should hold off on upgrading17:46
clarkbfungi: the problem is they have no organization17:46
clarkbwith tree tabs all these changes would hang below the etherpad17:46
fungiright, that is my main challenge as well17:46
clarkbso I can group logical sets of tabs together easily17:46
fungioh, it's smart trees? that's useful17:46
clarkbthen when work is done I close etherpad and all leaf tabs go away too17:46
fungiso you don't have to manually group the tabs into trees yourself17:46
* fungi didn't know such magic existed17:47
clarkbno by default opening a new tab from within a tab creates a child relationship17:47
clarkbexcept its all broken on ff42 so :(17:47
clarkbI will update etherpad with group rename step17:47
pabelangerpleia2: cool, and back17:48
clarkboh wait its already there I was just blind17:48
fungiclarkb: so you're saying the group rename commands in step 12 are incomplete?17:48
fungiahh, yep. cool17:48
fungianteaya's done great work on this17:48
pleia2pabelanger: great, just cleaning up our old stackalytics instance, should be getting to new one momentarily17:48
*** daniloramalho has joined #openstack-infra17:49
*** daniloramalho is now known as dramalho17:50
pleia2<insert monthly complaint about web based dns interface for deleting entries here>17:50
greghaynesMaybe we could add that to the irc bot.17:51
fungipleia2: that discussion was so 10 minutes ago17:51
pleia2fungi :)17:51
mordredfungi: that discussion is always17:51
jrollclarkb: this last line implies a job timeout, right?
fungiwe've met our quota for the month17:51
*** pradk has quit IRC17:52
jrollclarkb: that seems to be the only relevant difference between success and fail for bug 139309917:52
openstackbug 1393099 in OpenStack-Gate "test_baremetal_server_ops fails with timeout waiting for provision_state to change" [Undecided,Confirmed]
clarkbjroll: not necessarily, the clenaup could've happened before the timeout17:52
clarkbjroll: thats a generic end of job cleanup runs regardless of timeout or not17:52
jrollclarkb: oh god I can't read, sorry17:53
fungiclarkb: the only addition i can see to the current etherpad is that the networking-bagpipe-l2 -> networking-bagpipe change is now in order and needs to get entries in the various cut-n-paste sections17:53
*** sputnik13 has quit IRC17:53
jrollclarkb: though it seems to happen during a tftp operation, which means we weren't done here yet17:53
*** vilobhmm11 has joined #openstack-infra17:54
fungiclarkb: i'll get those added here shortly and start working on rebasing the 5 changes into a series in preparation for the maintenance in a couple hours17:54
clarkbjroll: if you look at the console log tempest appears to fail cleanly17:54
*** jistr has quit IRC17:54
fungionce you're done reviewing that is17:54
clarkbfungi: I am through all of them but bigpipe17:55
clarkbfungi: they look fine so far17:55
clarkbthe one quetion I did have was attci17:55
clarkbare we still atticing?17:55
*** fedexo has quit IRC17:55
clarkbbecause two akanda repos go to the attic17:55
*** vilobhmm11 has left #openstack-infra17:55
*** peterlisak13 has joined #openstack-infra17:56
fungiclarkb: i would rather not attic but it's already written up at this point and we haven't revisited that topic on the ml post stackforge retirement17:56
jrollclarkb: mmm yeah. my other theory is these dropped dhcp packets, but it eventually succeeds and that also happens on successful jobs
*** sputnik13 has joined #openstack-infra17:56
jrollso I think that's a red herring17:56
*** cdent has quit IRC17:56
clarkbfungi: and agreed looks like we just need to update the steps to include bagpipe17:57
clarkbwhich is not confusing at all ocmpared to bigpipe17:57
fungiheh, i miss old f5s17:57
*** Sree has joined #openstack-infra17:57
fungimy fingers still remember the bigpipe command17:58
*** tonytan4ever has quit IRC17:58
fungiwow, actually looks like bigpipe is still a thing in their cli17:58
clarkbin other news nodepool seems to be struggling a bit (lots of building nodes and not relaly keeping up with demand), I will look at logs there to see what is going on17:59
fungihaving that clear up before we need to push changes through and restore the queue at the end of our maintenance would be stellar18:00
*** tonytan4ever has joined #openstack-infra18:00
clarkbUnknown Error HTTP 500 :)18:01
clarkbthe best error18:01
mordredit's my favorite18:01
clarkbthats all in hpcloud18:01
fungiclarkb: floating ip leak18:01
fungisays smart money18:01
*** electrofelix has quit IRC18:01
clarkbya thats one possibility18:01
mordreds/Unknown Error/Our load balancer has a sad/18:01
clarkbI am going to see if I can nova show one of them before it goes away18:01
*** EricGonc_ has quit IRC18:01
*** Sree has quit IRC18:02
fungii'm getting a list of stale unassociated fips18:02
fungishould be able to clean them up in a few minutes if there are any18:02
clarkbnova show shows the VM as active18:02
clarkbso no errors there18:02
clarkbI will dig up my delete fips script18:03
openstackgerritSpencer Krum proposed openstack-infra/system-config: Treat openstack_health normally
mordredyah. well, that was the symptom of the db timeout thing from before18:03
*** peterlisak13 has quit IRC18:03
mordredcreate took longer than the load balancer timeout18:03
clarkbvenv/bin/neutron floatingip-list | grep -v '10\.0\.' | sed -e '1,3d' -e '$d' | cut -d'|' -f 2 | xargs -n 1 -P 1 venv/bin/neutron floatingip-delete18:03
mordredso the lb threw a 500 at us18:03
fungiclarkb: i already have mine going (well, the discovery pass anyway)18:03
mordredwhich made us mark it as failed18:03
mordredand then we deleted it18:03
funginova floating-ip-list | grep ' - ' | cut -d' ' -f4 | sort > fip-cruft-1 ; sleep 300  ; nova floating-ip-list | grep ' - ' | cut -d' ' -f4 | sort > fip-cruft-2 ; diff -U1000 fip-cruft-{1,2} | grep '^ ' > fip-cruft ; wc -l fip-cruft{,-1,-2}18:03
mordredwhich put more stress on the already stressed db18:03
nibalizer will fix once openstack-health is renamed to openstack_health18:04
clarkbheh no module named oslo.utils18:04
mordredcreate server times are slow - but not as bad as they were18:04
mordredlist servers time is at a vvery large spike18:05
clarkb--force-reinstall will hopefully fix my virutalenv18:05
clarkbyup that worked18:06
mordredfungi: you should learn about the neutron floatingip commands18:06
fungii should18:06
clarkb338 potentially leaked floating IPs18:06
mordredfungi: they have a better output format selection18:06
pabelangerpleia2: great18:06
clarkbmordred: eh18:06
clarkbmordred: more like completely undocumented and terrible18:06
fungiclarkb: is that comparing passes spaced some time apart?18:06
clarkbmordred: and to get feature parity with novaclient you have to RTFS18:06
clarkbfungi: no that was a single point in time18:06
mordredclarkb: what???18:07
fungii've found that they can show up not associated during provisioning18:07
clarkbmordred: neutronclient allows you to select columns18:07
fungiand also sometimes briefly during cleanup18:07
clarkbmordred: its completely undocumented18:07
*** dramalho has quit IRC18:07
mordredit's not18:07
mordredit's totally documented18:07
clarkbmordred: the columns are not18:07
fungimy diff has 333 entries18:07
clarkbmordred: so to figure out what you can actually display you have to rtfs18:07
fungispaced 5 minutes apart18:07
clarkbmordred: and by default you get less info than what novaclient shows18:07
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add unit tests for
clarkbI spent a good hour figuring this out one day and probably drank too much after18:08
greghaynesmordred: clarkb somewhat relavent - on my thinking was that in a case like this if we did FIP creation first it would be a lot less work on a failue18:08
clarkb"why can't I just .... OH! that would've been helpful to actually tell someone"18:08
greghaynesThe shade deal does help, but still it would be good to do the cheap work first and fail fast18:08
mordredclarkb: ok. I stand corrected. I have no idea why I knew about it18:09
greghaynesoh, actually, we cant because the shade thing attaches18:09
mordredit's documented in the openstackclient help18:09
*** mriedem_away is now known as mriedem18:09
mordredand they both use cliff18:09
*** nadya has quit IRC18:09
*** bharathm has joined #openstack-infra18:10
*** abregman has joined #openstack-infra18:11
fungiwow, pre-purchasing internet access on a cruise ship is only roughly as usurious as for an international flight18:11
mordredclarkb, fungi:
clarkbyes its cliffy but highly coupled to the api version too18:11
*** achanda has quit IRC18:12
fungi(on a per diem basis anyway)18:12
clarkbso its not really a portable thing regardless18:12
mordredthere are the docs, fwiw18:12
clarkbmordred: where?18:12
mordredclarkb: that was the output of openstack ip floating list -h | pastebinit18:12
*** unicell has joined #openstack-infra18:12
mordredI'm not sure why that help doesn't get outputted elsewhere18:12
greghaynesI just got reminded - did we ever follow up and check if the paste.o.o robots.txt had any effect?18:12
clarkbI don't care about openstack18:12
mordredclarkb: it's the same help18:12
clarkboh I see in your paste18:13
mordredneutron just doesn't display it18:13
clarkbya those docs are uesless18:13
clarkbthe -c is outputed in neutron help floatingip-list18:13
clarkbbut no information on what a valid column is is documented18:13
clarkbthats the problem18:13
clarkbits great I can select columns, I have no idea what columns I can select18:13
purpHowdy, folks. Pardon my ignorance, but if I want to get Stackalytics to properly assign email addresses to HP(E), who should I talk to?18:14
clarkband it is actually dependent on what the api returns to you raw in the api18:14
clarkbwhich means its not portable18:14
fungipurp: there's a data file in the openstack/stackalytics repo, you just propose an update to it through gerrit18:14
fungii'll see if i can find it18:14
purpfungi: I've got that.18:15
purpfungi: etc/default_data.json, yes?18:15
*** abregman has quit IRC18:16
clarkbfungi: looking at a diff over time of fips it looks like we aren't really seeing any old ones go active18:16
fungilooks like that's the one18:16
clarkbmaybe I am not waiting long enough18:16
clarkbmy diff just shows a bunch of new fips on the more recent listing18:16
clarkbnone of the old fips go away in the more recent listing18:17
*** tonytan_brb has joined #openstack-infra18:17
*** abregman has joined #openstack-infra18:17
*** tonytan_brb has quit IRC18:17
nibalizerpurp: there is a 'companies' blob in there where I think you can add HPE domains to the hp company18:17
fungiclarkb: on my diff i see 5 drop out in the second pass 5 minute slater18:18
purpfungi: thanks!18:18
clarkbI see none drop out but that may be due to different timing18:18
clarkbI will rerun a new listing in a bit to see if that changes18:18
purpnibalizer: I'll go sniffing. Guess we need to decide how we want to handle the split here.18:18
*** sputnik13 has quit IRC18:18
clarkbmy diff arrows all point > that way18:18
purpfungi nibalizer: is there someone I should chat with on Stackalytics team who would care how we handle attribution around this?18:19
*** niedbalski has quit IRC18:19
fungiclarkb: looks like,139,147,172,183 went active between my passes18:19
mordredgreghaynes: so, we could do that, although there is a problem we'd have to solve which is right now we don't have a GREAT answer for "will I need a floating ip in the future if I decide to create a server"18:19
mordredgreghaynes: we could grow one of course, since it's just programming18:19
nibalizerpurp: I don't know anyone on the stackalytics team, sorry18:19
*** tonytan4ever has quit IRC18:19
mordredgreghaynes: but at the moment the quetion is "hey, I've got this server, do I need to gt a floating ip for it?"18:20
mordredpurp: ilya doesn't seem to be in channel at the moment18:20
*** nadya has joined #openstack-infra18:20
purpI'll poke in #openstack-stackalytics and see what I get. Thanks fungi nibalizer mordred for guidance!18:21
*** e0ne has joined #openstack-infra18:21
*** cdent has joined #openstack-infra18:21
pleia2if someone has a couple minutes, I could use a few eyes on the error I'm getting when I try to bring up this server: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /opt/system-config/production/manifests/site.pp:152 on node stackalytics.openstack.org18:21
*** tjones has quit IRC18:21
pleia2that line looks fine to me, so it could be red herring because puppet18:21
pleia2my eyes can't see the issue though18:21
*** niedbalski has joined #openstack-infra18:22
*** sputnik13 has joined #openstack-infra18:22
fungiclarkb: oh, it's possible the ones i'm seeing drop off my list were in the midst of being deleted during the first pass18:22
fungirather than switching from disassociated to associated18:23
pabelangerpleia2: looks like issue getting remote data from puppetmaster.o.o18:23
*** dramalho has joined #openstack-infra18:23
fungisince i'm just filtering out all associated fips i don't see any difference between associated and deleted18:23
*** cdent has quit IRC18:24
*** tjones has joined #openstack-infra18:25
*** bardia has joined #openstack-infra18:25
*** salv-orlando has quit IRC18:25
*** salv-orlando has joined #openstack-infra18:26
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets
pleia2pabelanger: it does manage to get some data earlier in the run (and this is all initiated from the puppetmaster), so I'm not sure what would cause it to fall over here, and know exactly where it's failing18:26
*** devvesa has quit IRC18:27
pleia2could be one of the changes we recently made, but I wasn't involved with those enough to speculate18:28
pleia2and I launched codesearch.o.o ok a week ago18:28
*** abregman has quit IRC18:29
nibalizerpleia2: ill look at that18:29
clarkbpuppet is trying to run tge empty? method against nilclass and failing. without proper stacltrace hard to know wht18:29
*** abregman has joined #openstack-infra18:29
nibalizerya that sounds like a catalog compilation18:29
* nibalizer runs puppet agent --test on stackalytics18:29
jeblairformating error in ?18:29
jeblairno indedentation for the private key18:30
jeblairthough it's a yaml text block18:30
jeblairstackalytics_ssh_private_key_contents: |18:30
jeblair-----BEGIN RSA PRIVATE KEY-----18:30
jeblairshould have a couple spaces at the start of the second and subsequent lines18:30
nibalizerpleia2: did make it into dns yet?18:31
jeblairpleia2: did you use
pleia2jeblair: I did not18:31
jeblairnibalizer: that happens last18:31
*** ybathia has joined #openstack-infra18:31
jeblairpleia2: ah, i'd recommend that :)18:31
pleia2nibalizer: no, the run deletes the server upon failure, I can email you the full output18:31
openstackgerritMatt Riedemann proposed openstack-infra/devstack-gate: DO NOT MERGE: test config for gate-tempest-dsvm-cells-neutron
pleia2jeblair: thanks, I didn't know about it :)18:31
nibalizersounds like jeblair has a likely hypothesis18:31
jeblairpleia2: i manually updated the file; try again?18:32
*** dramalho has quit IRC18:32
pleia2jeblair: trying, thank you18:33
*** achanda has joined #openstack-infra18:34
*** trown|lunch is now known as trown18:35
*** otter768 has joined #openstack-infra18:35
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add unit tests for
*** achanda has quit IRC18:35
pleia2jeblair: that seemed to be it, thanks for the assist, I'll use next time18:36
jeblairpleia2: w00t18:36
*** ilyashakhat has joined #openstack-infra18:37
*** yamamoto has joined #openstack-infra18:38
*** cloudtrainme has joined #openstack-infra18:38
*** HeOS has quit IRC18:39
*** otter768 has quit IRC18:40
SpamapS# TODO: make this more ansibley18:41
* SpamapS smiles18:42
pabelangerSpamapS: # NOTE(pabelanger): Not complete until Chef hooks added18:43
*** yamamoto has quit IRC18:43
SpamapSdoes that mean when the next ansible thing comes out we'll all be hating on that lovely ansibley comment too?18:43
*** maiteb has quit IRC18:45
*** wenlock__ has joined #openstack-infra18:45
SpamapSso... I want to pull in some code I have pushed up to github and test running it as part of a pre/post test hook. Anybody have thoughts on how I might do that?18:45
pleia2pabelanger: server is up, but we have some debugging to do18:45
pabelangerpleia2: sure18:46
*** hdd has quit IRC18:47
*** abregman has quit IRC18:47
eantyshevjeblair, pleia2: Hello! I have an improvement for, hope it will make it more usable for human-viewed yaml:
*** achanda has joined #openstack-infra18:48
*** wenlock__ has quit IRC18:49
clarkbfungi: were you running a delete fip script then?18:49
clarkbfungi: wasn't clear if you were but I can in a few minute sif not18:50
fungiclarkb: i can fire it now. didn't want to step on your toes18:50
fungiit's running18:50
*** sabeen has quit IRC18:51
*** ildikov has joined #openstack-infra18:51
fungilooks like 237936 is renaming a group too so i'll add it to the group renames list18:52
fungiappears to be dropping the old networking-bagpipe-l2-release group entirely in favor of neutron-release18:53
fungiso just the core group needs renaming18:53
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API
*** pradk has joined #openstack-infra18:54
clarkbfungi: maybe rename the release group for completeness if though it will be unused?18:54
*** cloudtrainme has quit IRC18:55
fungiclarkb: better to just empty it, switch it to owned by administrator and hide it18:55
*** sputnik13 has quit IRC18:55
fungii'd love to get around to writing a script which can check our list of visible groups against all acls for direct (or indirect through arbitrary levels of inclusion) reference and then clean up any which have been orphaned18:56
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add 'testing' node to site.pp
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove global puppetmaster setting
openstackgerritSpencer Krum proposed openstack-infra/system-config: Switch ansible to applyprep and puppetapply
openstackgerritSpencer Krum proposed openstack-infra/system-config: Role to prep system for puppet apply
*** mrmartin has quit IRC18:58
*** Sree has joined #openstack-infra18:58
*** mrmartin has joined #openstack-infra18:59
*** sputnik13 has joined #openstack-infra18:59
mordredfungi: ++18:59
clarkbjust don't forget All-Projects19:00
fungiindeed, it counts as an acl too19:00
*** cloudtrainme has joined #openstack-infra19:01
*** bpokorny_ has joined #openstack-infra19:02
*** Sree has quit IRC19:02
clarkbanything else need doing in the next hour for rename prep?19:04
funginot that i can think of. i've updated the etherpad if you want to check that and am working on the rebases now19:04
*** bpokorny has quit IRC19:05
clarkbetherpad updates look good19:06
clarkbdo we want an explicit step for "disabling" that release group?19:06
fungimeh, i'll just make a note in step 1219:07
*** sputnik13 has quit IRC19:07
clarkbjeblair: you said ovh was looking better with even less oversubscription, does that mean we can start adding some load to it now? or do we still have api side stuff to wait on?19:08
*** e0ne has quit IRC19:08
openstackgerritVáclav Hejral proposed openstack-infra/elastic-recheck: Add query for nova bug 1512955
openstackbug 1512955 in OpenStack Compute (nova) "Race condition in nova/neutron when booting instance with xen driver" [Undecided,In progress] - Assigned to huan (huan-xie)19:08
jeblairclarkb: i believe we're now waiting for them to see if they have room for us :/19:08
greghaynesfungi: for gra1 we dont have a AAAA - do they lack ipv6 or was that an oversight?19:09
greghaynesfor pypi.gra1.o.o19:09
jeblairgreghaynes: they lack ipv619:09
*** sputnik13 has joined #openstack-infra19:10
fungiyeah, i would make some joke about the 1990s calling and... well anyway no ipv6 there (yet anyway)19:12
*** e0ne has joined #openstack-infra19:12
clarkbin fairness the routers I had from the ealry 2000s couldn't ipv619:16
clarkband they had uptime of like 6 years19:17
fungilucky you--must not have been cisco ;)19:17
clarkbthey were :)19:17
*** tonytan4ever has joined #openstack-infra19:17
clarkbrunning CatOS + IOS hybrid mode19:17
*** cloudtrainme has quit IRC19:18
fungilead-lined housings then? ours used to crash regularly and tac would blame solar flares/cosmic rays19:18
clarkbfungi: I have no idea but they did eventually crash at around the 6 year mark19:18
fungi6500 series?19:18
clarkbhad a pair of them and they died a week apart19:19
clarkbpretty sure if we went back to look at change control we would've seen they were installed a week apart19:19
clarkbactually they were in an old bunker type building, maybe that protected them from cosmic rays19:19
fungiwe had dozens of them, fully-populated, and one or two would pop every month (different ones each time)19:19
clarkbthe other hting you have to worry about is fan try dying and having half your cables strung in front of the tray so you couldn't hot swap it19:20
fungiheh, yes. always have to fight the spaghetti19:20
clarkbthere was a bug with hot swapping modules iirc19:20
fungiharlowja: i'm curious, in your zk+openjdk debunking, you never addressed the previously quoted alarming comments from the zk source code about only oracle's jdk being trusted to work correctly... stale code comments which never got cleaned up?19:21
clarkbwhere the chassis would restart power if you didn't seat the new module in the correct amount of time19:21
harlowjafungi that was not zk19:21
clarkbso ya they rebooted a lot but those 2 were special I guess19:21
harlowjathat was cassandra...19:21
harlowjanot zk19:21
fungiharlowja: you are correct--i'm confusing the two threads19:21
harlowjau aren't the only one afaik19:21
harlowjai should have mentioned that in the thread19:21
fungiall java apps look the same to me ;)19:21
openstackgerritStephen Lowrie proposed openstack-infra/project-config: Add tempest-dsvm-lxc as experimental
harlowjabut feel free to respond, saying 'and btw the comment blahblah not working was cassanddra'19:22
harlowjait wasn't zk.....19:22
harlowjafungi since u had that misconception (u aren't alone) please do respond if u want :)19:22
openstackgerritGlauco Oliveira proposed openstack-infra/subunit2sql: Add db migration to align column size to 255
harlowjaif u want/can19:23
greghaynesI think SpamapS was AFK from that session when that point came up, but he mentioned that juju did a lot of openjdk +zk deploys19:23
fungiharlowja: yep, added to my backlog for today19:23
greghaynesas another data point19:23
harlowjagreghaynes yup, feel free to respond to ;)19:23
harlowjai mentioned that greghaynes in the DLM spec19:23
harlowja*the juju usage19:23
harlowjabut more 'testimonals' are welcome19:23
greghaynesah, awesome19:23
harlowjafungi thx19:23
*** hdd has joined #openstack-infra19:24
*** achanda has quit IRC19:24
*** mahatic has quit IRC19:25
*** cloudtrainme has joined #openstack-infra19:25
*** sabeen has joined #openstack-infra19:26
*** ilyashakhat has quit IRC19:28
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Moving compass-install from stackforge to openstack namespace
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename puppet project 'puppet-openstack_health'
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Move networking-bigswitch from stackforge to openstack
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename project networking-bagpipe(-l2)
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename projects akanda-* -> astara-*
*** ilyashakhat has joined #openstack-infra19:28
fungiclarkb: rebased ^ and made sure the only one with a merge conflict is last in the stack for easier re-reviewing19:28
*** wenlock__ has joined #openstack-infra19:28
*** abregman has joined #openstack-infra19:29
*** achanda has joined #openstack-infra19:29
fungilooks like gerrit properly preserved votes for the rest of them19:29
*** sabeen2 has joined #openstack-infra19:32
jrollharlowja: more anecdotal data for you, rackspace's cloud monitoring team uses openjdk7 + zk for locking at scale without much issue19:32
harlowjajroll press reply button and send that ;)19:32
*** wenlock__ has quit IRC19:33
fungiclarkb: you okay with #status notice Gerrit will be offline at 20:00-20:15 UTC today (starting 25 minutes from now) for scheduled project rename maintenance19:33
SpamapSZK+OpenJDK should be fine19:33
*** Piet has quit IRC19:33
fungialso i'll go resurrect statusbot again19:33
SpamapSWe really do need ti dissambiguate Cassandra+OpenJDK from Zookeeper+OpenJDK19:34
*** smarcet has quit IRC19:34
*** openstackstatus has joined #openstack-infra19:35
*** ChanServ sets mode: +v openstackstatus19:35
*** sabeen has quit IRC19:35
harlowjaSpamapS fungi says he'll send email about that ;)19:36
harlowjaseems people are still having that misconception somehow...19:36
* fungi is plenty misconceptiony19:36
*** akshai has quit IRC19:37
*** cloudtrainme has quit IRC19:37
mordredyah. also, the difference between them is that zk is abstractable so can be replaced for our needs by consul if someone wants19:38
clarkbfungi: ya lgtm19:38
mordredcassandra on the other hand - you can't really abstract that19:38
fungi#status notice Gerrit will be offline at 20:00-20:15 UTC today (starting 20 minutes from now) for scheduled project rename maintenance19:39
openstackstatusfungi: sending notice19:39
*** nadya has quit IRC19:40
-openstackstatus- NOTICE: Gerrit will be offline at 20:00-20:15 UTC today (starting 20 minutes from now) for scheduled project rename maintenance19:40
openstackstatusfungi: finished sending notice19:41
pleia2I'm around, so lmk if you want me to pitch in with anything19:42
SpamapSharlowja: thanks for your email about openjdk+zk.. now.. time to rewrite it in rust.19:43
harlowjaSpamapS np, haha19:43
harlowjai did what i could, there will be crazies that will still say stuff, but meh, those people will always be there19:44
harlowjacrazies will be crazy, ha19:44
SpamapScraycrayz be craycray19:44
harlowjanew meme ftw19:44
mordredsomebody go register craycraybecraycrayyolo.io19:44
mordredit'll get a pile of VC I'm sure19:45
SpamapSI'm going to go flog myself now for that19:45
SpamapS100 lashes for going millenial19:45
clarkbmordred: cray might want to invest19:45
fungiindeed. cray be craycray19:46
fungiat least these days19:46
*** akshai has joined #openstack-infra19:46
harlowjalogo => craycray person next to old cray cray computer19:47
harlowja*with hipster in there somewhere to19:47
*** daemontool has quit IRC19:47
mordredto be truly hipster, do a DEC logo but add a wink or a beard or someting19:48
fungiwe can start a trend of dressing as hipsters and posing next to old supercomputers19:48
*** salv-orl_ has joined #openstack-infra19:48
harlowjafungi +219:48
fungii call dibs on the wang photo19:48
mordredfungi: so many jokes.19:48
docaedofungi: best line of the day19:48
harlowjai'm ok with giving someone else the wang photo, i don't need to run for congress/president or something19:49
*** mestery has quit IRC19:50
harlowjaannnd, i will say no more, ha19:50
clarkbfungi: midcycle at the living computer museum?19:50
fungiclarkb: that would be awesome19:50
fungiwe'd take that place over19:50
fungii think they do rent it out for organized functions19:50
harlowjathat'd be sweet19:50
harlowjagood idea19:50
clarkb lets do it19:51
*** salv-orlando has quit IRC19:51
clarkbdibs on the lisa running xenix19:51
* fungi puts the xenix down and slowly backs away19:51
fungithat os was scart19:52
*** peterlisak13 has joined #openstack-infra19:52
harlowjathe history museum in mountain view could work also, they probably rent rooms to19:52
fungi(scart was something else entirely)19:52
clarkbharlowja: but do they let you hack on the pdp11?19:52
clarkbon a teletype!19:52
*** mrmartin has quit IRC19:52
*** e0ne has quit IRC19:54
pleia2harlowja: they do, but only after hours (I've been to a few events there)19:54
*** pradk has quit IRC19:54
*** dangers is now known as dangers_away19:54
pleia2not hack on the pdp11, I meant renting rooms ;)19:54
*** ilyashakhat has quit IRC19:55
harlowjawell we can at least take craycray pictures next to a cray cray19:55
harlowjaand publish a calendar19:55
pleia2they do live demos of the difference engine though!19:55
harlowjaya, that thing is noisy19:55
*** cloudtrainme has joined #openstack-infra19:55
harlowja*from what i remember19:55
pleia2computers are still noisy19:56
samueldmqis our current system able to un-do workflow+1 ?19:56
nibalizerfungi: clarkb ++19:56
fungithey'll also demo the card sorter19:56
clarkbsamueldmq: yes19:56
clarkbjust vote 019:56
clarkbsamueldmq: there is no workflow on that change19:56
samueldmqclarkb: looks like it got merged ? jenkins just posted a 'check' mark on that, it's weird19:56
clarkbits not merged19:56
*** peterlisak13 has quit IRC19:57
clarkbstatus is review in progresss19:57
samueldmqclarkb: there was, and it was removed, but jenkins just posted a ceck mark there19:57
clarkbjenkins will continue to run tests19:57
samueldmqclarkb: great! just wanted to check19:57
clarkbso jenkins reported but nothing merged19:57
*** tonytan4ever has quit IRC19:57
clarkbfungi: pleia2 how do we want to split up the work?19:58
samueldmqclarkb: thanks for clarifying19:58
*** dramalho has joined #openstack-infra19:58
clarkbfungi: pleia2 I am happy to do whatever bits including github mouse clicking19:58
fungiclarkb: pleia2: i'm happy to pick up the dregs of whatever you two don't want19:58
nibalizerI can do whatever19:58
fungiwe're all so indecisive19:58
* pleia2 hops on the etherpad19:58
clarkbnibalizer: have you done the gerrity and zuul things yet?19:59
fungii'll send the status alert19:59
clarkbnibalizer: those are good to do if not19:59
nibalizerI have not done anything19:59
nibalizerI just opened laptop19:59
*** Sree has joined #openstack-infra19:59
clarkbnibalizer: I mean with previous renames19:59
*** ilyashakhat has joined #openstack-infra19:59
nibalizeri've done the zuul capture queue things19:59
nibalizerive not done the gerritdb thigns19:59
*** abregman is now known as abregman|nearby19:59
pleia2oh, I was just signing up for the gerrit stuff19:59
* clarkb nominates nibalizer for gerritdb and indexing and related items20:00
fungi#status alert Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance20:00
openstackstatusfungi: sending alert20:00
* pleia2 hands off to nibalizer 20:00
fungidid we land zaro's documentation update to parallelize the lucene reindex?20:00
fungiit should go a lot faster if we do that20:00
clarkbpleia2: you cna have it :)20:01
pleia2ok, I've got zuul things and git renames20:01
fungistopping puppet now20:01
clarkbI will do github since I mentiond that I don't mind doing the mouse clicking20:01
pleia2nibalizer: I added you to all the review.o.o stuff20:01
clarkbnibalizer: you have etherpad link?20:02
clarkblooks like you are in the user list20:02
-openstackstatus- NOTICE: Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance20:02
*** ChanServ changes topic to "Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance"20:02
nibalizerclarkb: i do20:02
pleia2fungi: good to do zuul?20:02
fungiyep, it's stopped now20:02
*** bharathm has quit IRC20:03
pleia2zuul is stopped20:03
pleia2nibalizer: you're up20:03
*** Sree has quit IRC20:04
pleia2nibalizer: let me know when you start step 7 and I'll begin the git renames :)20:04
openstackstatusfungi: finished sending alert20:04
nibalizergerrit stopped20:05
*** bpokorny_ has quit IRC20:05
fungihrm, looks like we approved the documentation update which switches the rename process to the ansible playbook20:05
clarkbI didn't realize we had a working playbook yet20:05
clarkbor that it had been tested20:06
fungii guess we can give it a shot next time20:06
clarkblast I head it was in the "known to not work" state20:06
*** sputnik13 has quit IRC20:06
* mordred hadn't spent a ton of mental energy on the playbook because I was hoping we were almost done with renames for our lives20:06
nibalizerRows matched: 0  Changed: 0  Warnings: 020:07
nibalizerjust means no one is actually watching that project ya?20:07
clarkbnibalizer: yes20:07
nibalizerwhich puppet-openstack-health I can believei20:07
fungijhesketh and yolanda must think it's working because they +2'd and approved it20:07
nibalizerthe real work is in the 'changes' table ya?20:07
*** jroll is now known as tjroll20:07
nibalizerRows matched: 10  Changed: 10  Warnings: 020:07
nibalizeris more 'work done'20:07
funginibalizer: yep, looks good20:07
funginibalizer: mainly keep an eye out for warnings/errors20:07
nibalizerkk db done20:08
nibalizermoving git repos not20:08
*** mmmpork is now known as mmmborkborkbork20:08
* pleia2 adds DONE notes to etherpad20:08
clarkbI am going to go ahead and start github things now20:09
fungianyway, see for the new and faster reindex command line20:09
*** _ody has quit IRC20:09
fungii'll update the etherpad20:09
*** mwagner has quit IRC20:09
*** bhunter71 has joined #openstack-infra20:10
fungiit's basically the old command just append --threads 420:10
fungizaro's tested it on a dump of our production data and it works well, basically a 4x speedup20:10
pleia2when threading works like it should \o/20:10
fungipuppet's already updated to do this on the corresponding exec too20:10
nibalizermoves completed20:10
nibalizerstarting reindexes20:11
fungihopefully this should now complete in just a few minutes rather than 10-2020:11
*** ilyashakhat has quit IRC20:11
*** sputnik13 has joined #openstack-infra20:11
nibalizerjava jar ir jaring20:12
nibalizeris jaring20:12
*** bpokorny has joined #openstack-infra20:12
pleia2ok, doing the git{01-08} moves20:12
*** cloudtrainme has quit IRC20:12
*** _ody has joined #openstack-infra20:12
clarkbgithub slow20:13
fungislow github is slow20:13
*** salv-orl_ has quit IRC20:13
clarkbI had to change gerrit's github perms after ownership transfer20:14
*** dramalho has quit IRC20:14
*** ldnunes has quit IRC20:14
pleia2git moves all done20:15
*** achanda has quit IRC20:15
*** dramalho has joined #openstack-infra20:15
pleia2ooh, that is speedy :)20:15
clarkbgithub is done20:17
EmilienMwhere can I patch to have openstackstatus user connected on #puppet-openstack ?20:17
clarkbI should write a patch to the docs to add the gerrit write perms to github step20:17
*** asilenkov_ has joined #openstack-infra20:17
samueldmqis gerrit down ?20:18
pleia2EmilienM: system-config manifests/site.pp20:18
EmilienMsamueldmq: yes, maintainance20:18
pabelangersamueldmq: topic20:18
EmilienMpleia2: ack thx20:18
fungisome day someone will move the statusbot config to a file in project-config20:19
samueldmqEmilienM: pabelanger nice, sorry didn't notice, got confused with tzs :)20:19
pleia2samueldmq: we're always UTC :)20:19
fungiand also the meetbot config for that matter20:19
pleia2eavesdrop in system-config does make me a sad panda20:20
fungii'm more pandasad over the giant one-line list of channels in the global site manifest for statusbot20:21
*** Piet has joined #openstack-infra20:21
pleia2it is pretty awful20:22
docaedowhich bot logs channels?20:22
clarkbdocaedo: meetbot20:22
fungie.g. the openstack_project::eavesdrop class in system-config20:22
samueldmqpleia2: nice, now I learned :)20:22
pleia2samueldmq: I put a UTC clock in the panel on my desktop to help my brain20:23
docaedoclarkb: thanks, I need to see if I missed something because I don't think #openstack-app-catalog is logged20:23
clarkbdocaedo: is the 'openstack' nick in channel?20:23
EmilienMpleia2: that's smart20:23
clarkbif yes then it should be logged20:23
docaedoclarkb: no, it's not20:23
*** rfolco has quit IRC20:23
clarkbdocaedo: then ya I think you need to update the system-config class that fungi pointed out20:24
fungii just set all my clocks to utc except the ones my wife will complain about20:24
pleia2docaedo: it's logged ^^20:24
clarkboh it is logged20:24
samueldmqpleia2: hm, good idea, looks to be very useful, thanks for sharing20:24
docaedopleia2: yes I see that, thanks! I was wrong :)20:24
docaedopleia2: yay!20:24
clarkbnibalizer: so I guess its not much faster :/20:24
clarkboh well20:24
funginibalizer: this was with --threads 4 ?20:25
*** sputnik13 has quit IRC20:25
fungizaro: ^ i guess the parallel reindex isn't appreciably faster in production after all20:26
clarkbI wonder if there is a fixed overhead we don't really get past20:26
clarkbor maybe writing to the index is serial?20:26
clarkband so 1 or 4 threads they are all synchronizing through writes20:26
purpHi, smart infra folks. I'm here to be clueless and beg help again.20:27
*** glauco_ has quit IRC20:27
*** sputnik13 has joined #openstack-infra20:27
purpTrying to git review and get an ssh error I don't know how to resolve:
fungipurp: we're wrapping up a maintenance window for our code review/ci system. will look in a moment20:27
purpNo rush.20:27
purpI suspect mine might be related. =]20:28
clarkbwell git revie wwon't work right now20:28
clarkbgerrit is off20:28
fungiyeah, the outage is running a few minutes over what i projected unfortunately20:28
clarkbits all in the index20:28
nibalizerReindexed 238640 changes in 1040.3s (229.4/s)20:29
fungilooks like we're clear to start gerrit and ok the alert20:29
nibalizeram i good to proceed win #10 ?20:29
pleia2and once gerrit has its head on straight I'll start up zuul20:30
fungipleia2: make sure you wait a moment after starting zuul before reenqueuing20:30
pleia2fungi: ok20:31
fungihelps to check the gearman status to make sure all the common node types have registered20:31
pleia2fungi: ok, how do I do that?20:31
*** tjones has quit IRC20:31
*** ildikov has quit IRC20:31
fungiotherwise you end up with a bunch of the reenqueued changes bailing out with jobs NOT_REGISTERED20:31
nibalizergerrit started20:32
fungiecho status|nc -q 3 localhost 4730|grep <stuff>20:32
clarkbI will rename groups now20:32
nibalizerpleia2: you're good to start zuul i think20:32
pleia2starting zuul20:32
*** kushal has quit IRC20:32
*** tongli has quit IRC20:33
pleia2fungi: stuff would be job names?20:33
*** pc_m has quit IRC20:33
fungipleia2: node labels. i'd use -c to just get counts and make sure they're nonzero20:33
fungipleia2: like `echo status|nc -q 3 localhost 4730|grep -c bare-centos6`20:33
pleia2ok cool20:34
asilenkov_you guys should replace that ugly 503 with sexy announce ))20:34
pabelangerasilenkov_: ha ha, working on something now20:34
clarkbgroup renaming and hiding is all done20:34
pabelangerasilenkov_: thought the same20:34
pleia2looking good20:35
fungipleia2: awesome, i'll clear the alert20:35
fungi#status ok20:35
openstackstatusfungi: sending ok20:35
openstackgerritEmilien Macchi proposed openstack-infra/system-config: site.pp: add openstackstatus to #puppet-openstack
fungiand i'll go ahead and start approving our project-config change series now20:35
*** aysyd has quit IRC20:35
clarkbfungi: I had preivously -2d some of them but all of them lgtm20:36
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"20:36
*** hrubi has quit IRC20:36
*** otter768 has joined #openstack-infra20:36
pleia2fungi: hang on, is finishing20:36
fungipleia2: it shouldn't interfere20:36
fungithese are going straight into the gate pipeline anyway20:37
*** dims is now known as dimsum__20:37
*** tnarg_ has joined #openstack-infra20:37
openstackstatusfungi: finished sending ok20:37
nibalizerim gonna go offline for a bit20:37
*** hrubi has joined #openstack-infra20:38
pleia2ok, all done with #1120:38
fungiall 5 project-config changes are in the gate pipeline now20:38
funginibalizer: i'm a little chicken-and-egg confused on 24258620:39
fungithere's a review comment saying it shouldn't merge until the rename is complete, but a comment in the etherpad saying that it needs to be merged before puppet is reenabled (yet puppet does necessary bits to complete the rename)20:39
clarkbfungi: I think update modules will just fail until we get that in20:39
clarkbso merge that change, then enable puppet20:40
fungiexcept that change won't pass tests until the rename is complete20:40
*** tnarg has quit IRC20:40
*** nadya has joined #openstack-infra20:40
clarkboh heh20:40
*** otter768 has quit IRC20:40
clarkbis it not complete enough to work20:41
clarkbit just needs to be able to clone/fetch from that location20:41
fungioh, true it's already hopefully able to clone from the new name on our git server farm. i'll recheck it now20:41
clarkbya I think it may work20:42
*** bauwser is now known as bauzas20:42
*** wenlock has quit IRC20:42
fungigate-infra-puppet-apply-dsvm-f21: NOR_REGISTERED20:43
clarkboh and thats voting20:43
fungier, NOT_...20:43
pleia2I didn't check f2120:43
fungiso we need to get devstack-f21 nodes built first and i'll recheck it again20:43
*** dramalho has quit IRC20:43
*** Guest88047 is now known as jdg20:43
*** jdg is now known as jgriffith20:44
*** dramalho has joined #openstack-infra20:44
*** cloudtrainme has joined #openstack-infra20:44
clarkbI don't see any building right now20:44
notmorganwhoopse. wrong channel20:44
*** tjones has joined #openstack-infra20:45
fungiclarkb: i see one in a used state for the past hour20:45
*** nadya has quit IRC20:45
clarkbfungi: we can probably delte that one since we restarted nodepool20:45
*** wenlock_ has quit IRC20:45
clarkber zuul20:45
fungiwe may need to wait for a second demand pulse before it notices it needs one20:45
fungiyeah, good point, that job's reporting nowhere now20:45
clarkband there won't be extra demand until we register the jobs :)20:46
clarkbmin-ready is important20:46
fungiokay, it's deleted20:46
fungido we not have a min-ready of at least 1 for those?20:46
clarkbwe probably do but thats used to determine ratio for readyness vs the other labels20:46
fungiyeah, we do have min-ready: 120:46
clarkbso it may be a lower priority20:46
fungiyep, great point. we jumped up to max quota when the changes all got reenqued20:47
fungiclarkb: also, all the same apply jobs failed on that change again anyway, so it seems to need more than just the manual rename steps completed20:48
clarkbok let me go look at logs to see if thats fixable20:48
clarkber I don't see them failing yet20:49
clarkbjust the NOT_REGISTERED20:49
*** kushal has joined #openstack-infra20:49
clarkbare we looking at the same change in zuul  status/20:50
clarkbalso I had brie and toast and ginger green tea for this project rename20:51
fungioh, i bet i was looking at the wrong change20:51
clarkbevery project rename should come with food20:51
openstackgerritMerged openstack-infra/project-config: Rename projects akanda-* -> astara-*
pleia2I have winter squash soup20:52
fungiclarkb: yeah, sorry. the not registered job has it failing, but it hasn't gotten any other nodes assigned yet either20:52
*** daemontool has joined #openstack-infra20:52
*** mwagner has joined #openstack-infra20:52
*** salv-orlando has joined #openstack-infra20:53
*** gordc has quit IRC20:53
clarkb,info I updated the description there too20:53
fungithat previously used devstack-f21 node is sitting in delete state for 7 minutes and counting. nodepool still doesn't think it has sufficient incentive to provision another one yet due to the backlog demand20:53
clarkbbut no members, not visible, and admin owned should do it20:53
fungilooks like all 5 project-config changes succeeded, so now we just need to wait for zuul to finish flushing pending events/results20:54
openstackgerritMerged openstack-infra/project-config: Rename puppet project 'puppet-openstack_health'
openstackgerritMerged openstack-infra/project-config: Rename project networking-bagpipe(-l2)
openstackgerritMerged openstack-infra/project-config: Move networking-bigswitch from stackforge to openstack
openstackgerritMerged openstack-infra/project-config: Moving compass-install from stackforge to openstack namespace
*** dramalho has quit IRC20:55
fungithat's all of them20:55
*** cloudtrainme has quit IRC20:55
fungiamd yjere20:56
*** dramalho has joined #openstack-infra20:56
*** harlowja_ has joined #openstack-infra20:56
fungithere's a devstack-f21 node building now20:56
*** harlowja has quit IRC20:56
*** dramalho has quit IRC20:56
fungionce/if it registers those jobs, i'll just enqueue 242586 straight into the gate and takes our chances20:56
*** sputnik13 has quit IRC20:57
fungiotherwise it's going to be an hour or more to get sufficient nodes to rerun check20:57
clarkbwould be nice to see the other jobs run and pass, it will all come down to how fast we can boot an f21 node20:57
clarkboh right the gate wins20:57
pleia2so would it help to do a more thorough gearman check of *all* the nodes?20:57
*** BobBall is now known as BobBall_AWOL20:57
*** sputnik13 has joined #openstack-infra20:57
fungiprobably. just nobody's written/documented one afaik20:57
clarkbor stop testing our stuff on fedora20:58
pleia2clarkb: hee20:58
clarkbis that only for the slave manifest?20:58
clarkbI doubt we would ever run services on fedora20:58
clarkbmust be for the slave manifest20:58
fungipart of the challenge is that zuul doesn't know what all the possible node labels are, only nodepool does, so we need to marry nodepool configuration with a command run on zuul20:58
*** burgerk has quit IRC20:58
* pleia2 nods20:59
fungiokay, that f21 node went ready. i'll go aheand and enqueue20:59
*** ZZelle_ has joined #openstack-infra21:00
*** Sree has joined #openstack-infra21:00
*** alivigni has quit IRC21:00
*** openstackgerrit has quit IRC21:01
fungiand it's in the gate now21:01
fungilookit that f21 job go!21:01
*** openstackgerrit has joined #openstack-infra21:02
*** kdas_ has joined #openstack-infra21:02
*** alivigni has joined #openstack-infra21:02
fungionce it merges i'll unleash the puppets21:02
clarkbnow I want to play little big planet21:03
*** mriedem has left #openstack-infra21:03
*** mriedem has joined #openstack-infra21:04
*** kushal has quit IRC21:04
*** bhunter71 has quit IRC21:04
clarkbf21 job passed21:04
clarkbso I think having the git repos renamed was sufficient21:04
fungiyep, trusty now too21:05
*** Sree has quit IRC21:05
fungishould be in the clear as soon as we get a bare-centos6 and devstack-centos7 in the gate21:05
*** kdas_ has quit IRC21:05
*** kushal has joined #openstack-infra21:05
*** alivigni has quit IRC21:06
openstackgerritEmilien Macchi proposed openstack-infra/system-config: site.pp: add openstackstatus to #puppet-openstack
*** rlandy has quit IRC21:09
*** cloudtrainme has joined #openstack-infra21:10
*** thorst has quit IRC21:11
pleia2pabelanger: wow, loading data into stackalytics isn't exactly quick :) maybe your instructions should be "never let this server turn off"21:12
*** thorst has joined #openstack-infra21:12
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add CRUD methods for keystone groups
fungipleia2: we're using a trove instance for this, right?21:12
*** rlandy has joined #openstack-infra21:13
clarkbjust one more job queued21:13
pleia2fungi: no, it doesn't use mysql, it stores stuff in RAM (yay! no wait)21:13
fungioh! no wonder it's slow21:13
* pleia2 nods21:13
fungi(and needs an insanely large server)21:13
pleia2it may be worthy of some refactoring by someone clever if we ever have time now that it will come live with us21:14
clarkband last job started21:15
openstackgerritMerged openstack-infra/system-config: Treat openstack_health normally
*** thorst has quit IRC21:16
clarkbjeblair: what do you think about removing that zuul job retry logic if it looks like a jenkins fail? Now that we have 8 masters and are restarting them weekly we seem to have much healthier jenkinses21:16
clarkbjeblair: just thinking about it due to the devstack-gate change that got stuck recently21:16
clarkbfungi: ^21:16
clarkbI think we are ready to run puppet now21:16
fungiyup, starting21:16
fungiuncommented in crontab and manually started21:20
fungifatal: repository '' not found21:20
clarkbfungi: oh!21:21
clarkbit doesn't reexec itself21:21
clarkbso I think if you rerun now it will be fine21:21
clarkbso it always runs a version behind21:21
clarkber thats not quite a correct statement21:22
pleia2confirmed it does exist on the filesystems21:22
*** thorst has joined #openstack-infra21:22
clarkbit updates the version but it does not reexec so will continue on the old version if it was updated21:22
*** abregman|nearby has quit IRC21:22
clarkbpleia2: did you see japanese force awakens trailer?21:22
*** rfolco has joined #openstack-infra21:22
pleia2clarkb: nope21:22
*** sergmelikyan has quit IRC21:22
*** signed8bit has joined #openstack-infra21:22
pleia2I did manage to get tickets for the 17th though \o/21:22
clarkbpleia2: let me get a link21:23
fungialso we didn't accept the ssh key for codesearch.o.o yet so i did that just now21:23
pleia2fungi: oops, thanks21:23
clarkb36 seconds is a pretty neat shot21:23
pleia2need to for stackalytics.o.o too21:23
clarkbfungi: so you restarted it and it was happy?21:23
*** alivigni has joined #openstack-infra21:24
fungipleia2: i went ahead and deleted/accepted the stackalytics.o.o key now too21:24
fungiclarkb: er, no, looks like it's still trying to hit the old clone url21:25
pleia2fungi: thanks21:25
fungialso it complains "destination path '/etc/puppet/modules/apps_site' already exists and is not an empty directory."21:25
clarkbfungi: oh21:25
pleia2clarkb: bb-8 makes me so happy :)21:25
clarkbfungi: I bet the repo we have on disk needs to have its remotes updated21:25
clarkbfungi: or we can delete it and script should reclone21:25
fungiahh21:25 is new as well, no accepted ssh host key yet21:27
clarkbpleia2: I am afraid to do a release night showing, I have to line up babysitters (grandparents) too21:27
pleia2clarkb: I was on a plane when tickets when on sale, but we got lucky to be on a list for $company-rented-theater showing21:28
pleia2otherwise it's impossible in the geek land that is san francisco21:28
fungiclarkb: yeah, looks like we had a stale remote there, so removed now and will see if it reclones correctly21:28
clarkbI wonder if I can watch it in a theater that sells beer21:28
clarkbfungi: ok21:28
fungialso blew away apps_site since it was erroring too21:29
fungilooks like it recloned them both sanely now21:29
pabelangerpleia2: IKR! We'll be doing daily data dumps from memcached, so its really the first import that takes a long, long time (ago?)21:30
clarkbfungi: yay21:30
pabelangerpleia2: but it is working as expected21:30
pleia2pabelanger: right, that helps :)21:30
pabelangerokay, time for a bus ride with the kids. Will be back later tonight21:31
fungipuppet is disabled on groups.o.o... will try to remember to ask mrmartin if that can be undone next time he's around21:32
fungidisabled on logstash.o.o too clarkb?21:32
fungiis that still needed?21:32
*** abregman|nearby has joined #openstack-infra21:32
clarkbfungi: if those changes to fix kibana havne't merged yet we need it21:32
clarkbI will check21:32
fungiError: /Package[faraday]: Provider gem is not functional on this host21:33
openstackgerritTushar Gohad proposed openstack/requirements: Bump PyECLib version to 1.1.1
fungilooks like we're failing to upgrade elasticsearch21:34
clarkblooks like they merged, I will do a noop run and enable if looks good21:34
fungiyeah, looks like puppet-elasticsearch is failing to apply somewhere because gem is broken21:36
clarkbfungi: which host?21:37
fungii can't tell from the log. will check puppetboard21:38
regXboiclarkb, fungi: are multinode jobs having issues?21:38
fungihrm, well the failure didn't report into puppetdb21:38
fungiregXboi: let me guess, jobs reporting a status of NOT_REGISTERED21:38
regXboifungi: not sure yet - still looking through logs21:38
fungiregXboi: oh, if there are logs then that's not it21:39
*** tonytan4ever has joined #openstack-infra21:39
fungiso probably nothing related to our ci maintenance in that case21:39
jeblairclarkb: i don't think we should remove the retry -- it actually gets triggered constantly.  instead, i think we should put a counter on it; max 3x or even 1x.21:39
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Move irc bot from #akanda->#openstack-astara
clarkbjeblair: that seems reasonable too21:39
openstackgerritAdam Gandelman proposed openstack-infra/system-config: Move eavesdrop bot from #akanda->#openstack-astara
clarkbregXboi: the only thing I know of as being broken is shelve21:40
clarkbregXboi: otherwise last I saw the test env itself was working fine21:40
clarkbfungi: I am running puppet on logstsah.o.o looks fine so will enable when done21:40
regXboifungi, clarkb: ok - I happened to poke my nose at the neutron multinode jobs and they are headed off to 100% failure rate - looking at a sample now to understand what broke things21:41
clarkbregXboi: do you have a link?21:41
*** kushal has quit IRC21:42
regXboiclarkb: to the graph or to a sample job?21:42
clarkbto the job that is failing21:42
regXboiI'm seeing a return value of 2 in the console21:43
clarkbits the lxml compile fail issue21:43
clarkbI think sdague was fixing that in devstack21:43
regXboiclarkb: ok, let me do a logstash query and see if that covers all of what I'm seeing21:44
sdagueclarkb: yes21:44
* regXboi goes and reads21:45
sdagueclarkb: you want to ack that? I'll self approve as our ds cores are sparse this week21:45
clarkblet me read it21:45
*** ZZelle_ has quit IRC21:46
*** rlandy has quit IRC21:46
clarkbsdague: is rhel based stuff ok?21:46
fungiokay, ansible puppet run completed21:46
sdaguerhel based stuff is a further fix21:46
nibalizerfungi: ah sorry, didn't mean to leave you hanging on that one21:46
nibalizerclarks analysis was correct21:46
sdaguebut that's not impacting any jobs atm21:46
clarkbsdague: ok21:46
funginibalizer: no worries. 'twas sorted21:47
clarkbsdague: last question, you remove it from keystone but don't put the libs there, is that because keystone has actually cleaned up their dep on it now?21:47
*** sridhar_ram has quit IRC21:47
sdagueit's in base21:47
fungiworking on step 17 now21:47
sdaguesorry, in general21:47
clarkbsdague: right but you added it to nova21:47
clarkbshould it be removed from nova completely too?21:47
sdagueprobably wouldn't hurt21:48
sdagueit's all addative today21:48
sdagueso also being in nova doesn't hurt anything21:48
regXboidoes logstash.o.o stash devstack logs?21:48
clarkbregXboi: yes21:48
clarkbsdague: ok21:49
clarkbregXboi: though it may not be getting the subnode logs proprely21:49
sdaguethere are follow ups, but I'm done in about 5 minutes for the week. So if you are good with what's there I'll approve it21:49
clarkbregXboi: in any case all of those fails are likely due to this bug21:49
sdagueotherwise we can fix it next week21:49
clarkbsdague: ya I +1'd21:49
sdagueok approved21:50
regXboiclarkb: It looks like it may not be getting the sub logs in an obvious fashion21:50
*** hdd has quit IRC21:50
regXboibut that's for another day21:50
sdagueyeh, I would not be surprised if subnodes aren't being indexed21:50
clarkbregXboi: its only indexing 3 of the subnode logs all nova21:51
*** tjones has left #openstack-infra21:51
regXboiclarkb: that would explain it then21:51
fungistep 17 completed. working on step 18 now21:51
*** rlandy has joined #openstack-infra21:51
*** rfolco has quit IRC21:52
*** alivigni has quit IRC21:52
*** peterlisak13 has joined #openstack-infra21:53
clarkbjenkins restarts will being in about 3 hours21:53
clarkbI am going to check and see if any have leaked since last restarts21:54
clarkb(useful to know if we are properly restarting leaky masters or if we just avoid it altogether)21:54
regXboiclarkb: if I'm still around when 242559 merges, I'll see about running a recheck on one of these patches to see if that patch did the trick21:54
clarkbregXboi: you cna also depends-on that change21:54
regXboiclarkb: yeah, but it's not my patch I'm going to recheck21:55
clarkbooh 02 has properly leaked21:55
clarkball your threads are belong to jenkins21:56
clarkb04 as well21:56
*** dprince has quit IRC21:57
*** peterlisak13 has quit IRC21:58
clarkbbtw the rolling restarts means we can do plugin upgrades unattended now too assuming they don't need config updates21:58
*** bswartz has quit IRC21:58
*** rossella_s has quit IRC22:00
*** rossella_s has joined #openstack-infra22:00
*** Sree has joined #openstack-infra22:00
clarkbjesusaurus: ok I think I am all done with rename related things, were you planning to get to os-loganalyze today?22:01
clarkblet me know if I can help with anything22:01
fungi(and assuming the plugin upgrades don't break something causing jenkins to fail to restart)22:02
*** barnaby has quit IRC22:04
*** Sree has quit IRC22:05
*** shashank_hegde has joined #openstack-infra22:05
*** achanda has joined #openstack-infra22:08
*** rlandy has quit IRC22:11
*** lascii is now known as alaski22:13
jesusaurusclarkb: im probably not going to get to that today22:16
*** boris-42 has joined #openstack-infra22:21
*** mmmborkborkbork is now known as mmmpork22:22
openstackgerritMerged openstack-infra/project-config-example: Fix doc error for PUBLISH_HOST
*** tiswanso has quit IRC22:23
*** shashank_hegde has quit IRC22:25
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openstack_health: Update .gitreview for project rename
*** August1914 has joined #openstack-infra22:26
*** achanda has quit IRC22:27
*** shashank_hegde has joined #openstack-infra22:27
*** sfinucan has quit IRC22:29
*** smarcet has joined #openstack-infra22:31
*** dtardivel has quit IRC22:33
*** signed8bit has quit IRC22:33
*** smarcet has quit IRC22:35
*** otter768 has joined #openstack-infra22:37
*** sputnik13 has quit IRC22:37
*** mriedem has quit IRC22:37
*** sputnik13 has joined #openstack-infra22:38
openstackgerritMichael Johnson proposed openstack/diskimage-builder: Fix diskimage-builder image size
EmilienMinfra puppet apply jobs are broken:
EmilienMI guess you already know, just in case of22:38
openstackgerritMerged openstack-infra/puppet-openstack_health: Update .gitreview for project rename
*** kgiusti has quit IRC22:39
*** w_ has joined #openstack-infra22:40
clarkbEmilienM: I think they are fixed now22:40
openstackgerritAlberto Gireud proposed openstack-infra/shade: Fix incorrect variable name
*** doug-fish has quit IRC22:40
*** otter768 has quit IRC22:42
fungii've got step 18 wrapped up, so i'm going to switch to friday evening chores. i'm around if anybody needs me22:42
*** olaph has quit IRC22:43
pleia2thanks fungi22:43
EmilienMclarkb: ack22:47
openstackgerritEmilien Macchi proposed openstack-infra/system-config: site.pp: add openstackstatus to #puppet-openstack
*** TheJulia has quit IRC22:48
*** amitgandhinz has quit IRC22:51
*** shashank_hegde has quit IRC22:51
*** thorst has quit IRC22:54
*** thorst has joined #openstack-infra22:54
bookwarwhat is the reason for Zuul to combine changes with same change id but to two different branches (usually cherry-picks) in one integrated queue? Is it bug of feature?22:55
clarkbbookwar: you can't do upgrade testing otherwise22:56
*** yamamoto has joined #openstack-infra22:56
*** abregman_ has joined #openstack-infra22:57
*** thorst has quit IRC22:58
*** abregman|nearby has quit IRC23:00
bookwaryes, sounds like a reason, but i am not sure is it really useful in my current case. Is there an option to disable it if i don't use upgrades test? (i mean in Zuul as a local service, not the one in Openstack Infra)23:00
clarkbbookwar: there isn't an option to disable it. Is there something specific yo uare trying to do?23:01
clarkbis it a problem that they are in the same queue?23:01
*** Sree has joined #openstack-infra23:01
bookwarclarkb: yes, i have long tests in stable branches, and i have much smaller coverage in master23:02
bookwarso my master fixes are get delayed by those huge tests running for stable23:02
bookwarideally master should be tested and merged before it is cherry-picked..23:03
bookwarmay be i should just promote this policy more, but option to disable grouping by change-id would be nice23:04
clarkbbookwar: its not actually grouping by change id23:04
clarkbbookwar: its grouping because they are the same project23:04
clarkbor it is grouping because they share a job23:05
openstackgerritSpencer Krum proposed openstack-infra/system-config: Only install modules from source
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove 'removed' modules, they have been removed
bookwarthere is no intersections in jobs23:05
bookwarbut project yes is the same as one patch is the cherrypick for another23:06
*** Sree has quit IRC23:06
clarkbyup thats why thne23:07
*** regXboi has quit IRC23:07
clarkbthe change id isn't related23:07
clarkbI am not sure if anyone has looked into adding that to zuul but I think jeblair is wanting to slow down zuul dev so that zuulv3 can be built23:07
*** achanda has joined #openstack-infra23:08
*** achanda has quit IRC23:08
bookwarwe've just started to dive into Zuul v.2, i'll take a look at v3 version plans, but for now we'll need a workaround23:10
bookwarthanks anyway23:10
*** thorst has joined #openstack-infra23:13
*** kozhukalov_ has quit IRC23:15
*** trown is now known as trown|outttypeww23:15
*** sabeen2 has quit IRC23:16
*** thorst has quit IRC23:17
*** _denisra has quit IRC23:18
*** tnarg_ has quit IRC23:18
*** yamamoto has quit IRC23:19
*** tnarg has joined #openstack-infra23:19
*** tnarg_ has joined #openstack-infra23:20
*** gondoi is now known as zz_gondoi23:22
*** tnarg has quit IRC23:24
*** xyang1 has quit IRC23:26
*** achanda has joined #openstack-infra23:29
*** cloudtrainme has quit IRC23:31
*** salv-orlando has quit IRC23:33
*** tonytan4ever has quit IRC23:33
*** ddieterly has joined #openstack-infra23:34
*** ddieterly has quit IRC23:34
*** ddieterly has joined #openstack-infra23:35
*** yamamoto has joined #openstack-infra23:35
*** thorst has joined #openstack-infra23:35
*** ddieterl_ has quit IRC23:36
*** achanda has quit IRC23:37
*** thorst has quit IRC23:40
*** ddieterly has quit IRC23:41
*** otter768 has joined #openstack-infra23:49
*** jtomasek has quit IRC23:51
*** dimsum__ has quit IRC23:52
*** yamamoto has quit IRC23:54
*** Swami_ has joined #openstack-infra23:56
*** Swami has quit IRC23:56
*** rfolco has joined #openstack-infra23:58

Generated by 2.14.0 by Marius Gedminas - find it at!