| jeblair | clarkb: i find my computer's ability to carry firefox's memory/cpu footprint approximately equal to the space for horizontal tabs, so it generally works out for me. :) | 00:00 |
|---|---|---|
| *** sputnik13 has joined #openstack-infra | 00:00 | |
| clarkb | jeblair: haha | 00:01 |
| *** akshai has quit IRC | 00:01 | |
| clarkb | jeblair: I definitely find myself with 6GB firefox processes then realize my 400 tabs should probably be brought back under control | 00:01 |
| anteaya | https://addons.mozilla.org/en-US/firefox/addon/tree-style-tab/ | 00:01 |
| anteaya | can you even remember what is open when you have 400 tabs going? | 00:02 |
| *** freyes has joined #openstack-infra | 00:02 | |
| clarkb | anteaya: usually a good chunk can be removed but if properly organized in trees it usually makes sense | 00:02 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder: Fix diskimage-builder image size https://review.openstack.org/242273 | 00:02 |
| *** wenlock_ has quit IRC | 00:03 | |
| *** tnarg has quit IRC | 00:03 | |
| *** davideagnello has joined #openstack-infra | 00:03 | |
| clarkb | I also end up making it very spacial, I have zuul status and email pinned, other monitoring tabs go near the top (graphite graphs etc), documentation type stuff in the middle, changes at the end | 00:03 |
| zaro | anteaya: i've seen his tabs. it's insane. | 00:03 |
| clarkb | with horizontal tabs you get no indication of special relativity | 00:03 |
| clarkb | so I hvae no idea if I am at the end or middle or beginning | 00:03 |
| clarkb | once I installed a plugin to limit myself to 100 tabs | 00:03 |
| clarkb | I uninstalled it within a couple hours | 00:03 |
| *** tnarg has joined #openstack-infra | 00:04 | |
| *** wenlock has quit IRC | 00:05 | |
| SamYaple | clarkb: ho | 00:05 |
| mordred | clarkb: I would be unfunctional with that | 00:05 |
| *** aeng has joined #openstack-infra | 00:05 | |
| SamYaple | mordred told me to talk to you about multinode gating | 00:05 |
| jesusaurus | im weird and have a dozen windows each with a dozen tabs for general organization | 00:05 |
| *** dingyichen has quit IRC | 00:05 | |
| clarkb | SamYaple: hi | 00:06 |
| mordred | clarkb: yah - SamYaple was asking about making a multi-node test. my answer was "clarkb" | 00:06 |
| clarkb | so first question I like to ask is do you have a single node test that works? if not start there. Your debugging will be much much simpler | 00:06 |
| SamYaple | yes | 00:06 |
| *** dingyichen has joined #openstack-infra | 00:06 | |
| SamYaple | in fact i have a multinode-on-single-node patch that works.... but i OOM | 00:07 |
| SamYaple | clarkb: https://review.openstack.org/#/c/241500/ | 00:07 |
| *** camunoz has joined #openstack-infra | 00:07 | |
| clarkb | well I wouldn't even multinode node on single node | 00:07 |
| clarkb | just an all in one setup | 00:08 |
| *** sputnik13 has quit IRC | 00:08 | |
| clarkb | but anyways if you have a thing that works on one node yay | 00:08 |
| anteaya | clarkb: yeah I work with a spacial mindset as well | 00:08 |
| SamYaple | yea it was an experiment | 00:08 |
| anteaya | zaro: I believe you | 00:08 |
| SamYaple | clarkb: that multinode code isn't merged. if i can get true multinode i want to go that direction | 00:08 |
| *** tnarg has quit IRC | 00:08 | |
| *** heyongli has quit IRC | 00:09 | |
| clarkb | SamYaple: the basic setup is two machines with one attached to jenkins as the "primary" node | 00:09 |
| anteaya | clarkb: well if I have tab organization questions, I know who to talk to, thanks | 00:09 |
| clarkb | we use the primary node as the control machine for setup and it configures itself and the "subnode" to run openstack | 00:09 |
| *** heyongli has joined #openstack-infra | 00:09 | |
| *** sputnik13 has joined #openstack-infra | 00:10 | |
| *** cloudtrainme has quit IRC | 00:10 | |
| clarkb | the biggest limitation here is we don't have control over our networking so we setup overlay networks documented at https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt | 00:10 |
| * SamYaple reading | 00:10 | |
| clarkb | my suggestion would be to not reinvent that ovelray networking because 1) it actually was a huge pain to get owrking in the first place and 2) if our clouds change infra will update that to accomodate the changes and you would get that for free | 00:10 |
| clarkb | I actually need to finish ansibling the multinode setup so that we don't do the manual remote ssh stuff | 00:12 |
| *** _ianychoi_ is now known as ianychoi | 00:13 | |
| clarkb | but most of the setup at this point is done by ansible because it is really easy to assign nodes to groups and have ansible do the right thing for an arbitrary number of hosts | 00:13 |
| jlk | jeblair: is it always one host that has slow runs, or is it spread? | 00:13 |
| SamYaple | clarkb: yea this looks like what i did in the RAX public cloud for testing stuff with the overlays | 00:14 |
| SamYaple | whats the bandwidth like between nodes? | 00:14 |
| clarkb | SamYaple: its whatever $cloud provides to that flavor | 00:15 |
| clarkb | I think generally somewhere between 100mbps and 1gbps | 00:15 |
| clarkb | hasn't been a problem | 00:15 |
| SamYaple | ok so these vms are not split between clouds | 00:16 |
| clarkb | no, they will always be in the same region | 00:16 |
| SamYaple | theyll all be in DFW or ORD or similiar | 00:16 |
| SamYaple | ok cool | 00:16 |
| austin81 | f | 00:16 |
| *** yamamoto has joined #openstack-infra | 00:16 | |
| anteaya | SamYaple: mind if I ask what project you are going to be testing on multinode? | 00:17 |
| SamYaple | anteaya: Kolla | 00:17 |
| anteaya | SamYaple: thank you | 00:18 |
| SamYaple | clarkb: all this seems fairly straight forward to how ive done it before | 00:18 |
| SamYaple | i dont for see issues | 00:18 |
| *** sridhar_ram1 has quit IRC | 00:18 | |
| *** ddieterly has joined #openstack-infra | 00:19 | |
| *** heyongli has quit IRC | 00:19 | |
| SamYaple | whats the proceedure here for testing this? request a 2 node gate and then build the gate tests? | 00:19 |
| clarkb | SamYaple: add a new job that runs on the 2-node label to experimental queue of your project and iterate on that | 00:20 |
| *** heyongli has joined #openstack-infra | 00:20 | |
| *** vilobhmm11 has quit IRC | 00:21 | |
| * SamYaple dives into project-config again | 00:21 | |
| SamYaple | i dont think we have an experimental queue running anything yet | 00:22 |
| SamYaple | thanks will get started! | 00:22 |
| clarkb | ruagair: mordred et al why are we not using libapache2-mod-auth-openid for phabricator auth? | 00:24 |
| clarkb | reading up on cauth it seems needlessly complicated and doesn't do anything that the existing apache module for openid auth can't do for us more easily | 00:24 |
| *** smarcet has joined #openstack-infra | 00:24 | |
| clarkb | jesusaurus: ^ I want to say you used this at one time? any cahcne you want to share your thoughts on using it? | 00:25 |
| *** sputnik13 has quit IRC | 00:26 | |
| *** tonytan4ever has joined #openstack-infra | 00:26 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an attachment storage driver option https://review.openstack.org/241746 | 00:26 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add top-level attachments support https://review.openstack.org/242280 | 00:26 |
| clarkb | jesusaurus: one lsat question about the remainder stuff now that I read it again. should we skip the for loop if remainder is set? | 00:27 |
| clarkb | jesusaurus: so if remainder do remainder stuff else for loop | 00:27 |
| *** cody-somerville has quit IRC | 00:29 | |
| *** sambetts has quit IRC | 00:29 | |
| jesusaurus | clarkb: I dont recall using mod-auth-openid, but I may have touched it briefly once | 00:29 |
| clarkb | I thought you had put kibana or something in front of openid auth | 00:29 |
| *** smarcet has quit IRC | 00:29 | |
| *** heyongli has quit IRC | 00:29 | |
| *** andersonvom has joined #openstack-infra | 00:29 | |
| jesusaurus | no, that was ldap auth | 00:30 |
| clarkb | ah | 00:30 |
| *** heyongli has joined #openstack-infra | 00:30 | |
| jesusaurus | i think i used openid auth with lightdm-login briefly years ago but didnt get that working | 00:31 |
| *** ybathia has quit IRC | 00:32 | |
| *** sabeen1 has quit IRC | 00:32 | |
| jesusaurus | clarkb: for the remainder stuff, we still need the loop because we are reading from a generator | 00:33 |
| clarkb | oh right | 00:33 |
| clarkb | basically thats how we convert the iterator to a list | 00:33 |
| jeblair | jlk: it's spread | 00:34 |
| *** andersonvom has quit IRC | 00:34 | |
| *** tlian has quit IRC | 00:34 | |
| *** ybathia has joined #openstack-infra | 00:34 | |
| *** sridhar_ram has joined #openstack-infra | 00:34 | |
| *** sambetts has joined #openstack-infra | 00:34 | |
| *** ybathia has quit IRC | 00:35 | |
| ruagair | good question clarkb | 00:39 |
| *** bhunter71 has quit IRC | 00:39 | |
| ruagair | I'm coming to a similar conclusion regarding cauth's complexity as I'm in th emidst of deploying it now. | 00:39 |
| *** heyongli has quit IRC | 00:39 | |
| *** tnarg has joined #openstack-infra | 00:40 | |
| ruagair | I've been deferring at present to the decisions made before me clarkb. I'm open to what ever auth mechanism we consider most appropriate. | 00:40 |
| *** heyongli has joined #openstack-infra | 00:40 | |
| *** sputnik13 has joined #openstack-infra | 00:40 | |
| *** tlian has joined #openstack-infra | 00:41 | |
| ruagair | Does | 00:41 |
| jlk | jeblair: well darn. | 00:41 |
| ruagair | libapache2-mod-auth-openid pass through REMOTE_USER, clarkb? | 00:41 |
| clarkb | yes | 00:41 |
| ruagair | Then suprficially, I can't see a barrier. | 00:41 |
| clarkb | cool | 00:41 |
| fungi | sounds like a much simpler solution, if it works | 00:42 |
| ruagair | With authentication system are we hooking back to for OpenStack SSO? | 00:42 |
| ruagair | cauth is proving less elegant thna I'd like. | 00:42 |
| fungi | openstackid.org eventually, though possibly launchpad.net in the interim | 00:43 |
| ruagair | So i'm happy to turn my attention to libapache2-mod-auth-openid | 00:43 |
| *** tnarg has quit IRC | 00:43 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla https://review.openstack.org/242283 | 00:43 |
| jeblair | mordred: ^ did you have a reason for favoring cauth over libapache2-mod-auth-openid ? | 00:43 |
| *** sridhar_ram1 has joined #openstack-infra | 00:43 | |
| ruagair | Thanks fungi. I was at the point of pointing cauth to *something* and just realised I was unclear on what that sometihng was :-) | 00:43 |
| *** tnarg has joined #openstack-infra | 00:43 | |
| SamYaple | clarkb: https://review.openstack.org/242283 how does that look? | 00:43 |
| *** Liuqing has joined #openstack-infra | 00:43 | |
| fungi | ruagair: openstackid-dev.openstack.org is also available for testing purposes if needed | 00:44 |
| fungi | though that's really better suited to testing changes to the openstackid project itself | 00:44 |
| ruagair | Is openstackid.org only dev status atm, fungi? | 00:44 |
| fungi | no problem authenticating against production openstackid.org from development systems | 00:44 |
| jeblair | yeah, i think now that prod is up and running, probably not a compelling reason not to use it | 00:44 |
| *** sridhar_ram has quit IRC | 00:45 | |
| clarkb | SamYaple you should put multinode in the job name then the correct nodes will be used | 00:45 |
| fungi | ruagair: it's being used for authentication to www.openstack.org, groups.openstack.org and optionally ask.openstack.org at the moment | 00:45 |
| ruagair | If openstackid.org is prod, should I not be using that over launchpad? | 00:45 |
| SamYaple | clarkb: ill adjust it from multi to multinode | 00:45 |
| *** jamielennox is now known as jamielennox|away | 00:46 | |
| fungi | ruagair: i think it's a question of mapping the current users from launchpad bugs to maniphest bugs | 00:46 |
| *** austin81 has left #openstack-infra | 00:46 | |
| fungi | not sure if we care, ultimately | 00:46 |
| fungi | it would of course be much simpler not to care ;) | 00:46 |
| *** tnarg_ has joined #openstack-infra | 00:46 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla https://review.openstack.org/242283 | 00:46 |
| SamYaple | clarkb: done, only concern? | 00:47 |
| ruagair | That is a mapping problem I'm having in general, fungi. | 00:47 |
| *** sridhar_ram has joined #openstack-infra | 00:47 | |
| *** xyang1 has quit IRC | 00:47 | |
| *** jamielennox|away is now known as jamielennox | 00:47 | |
| ruagair | Ok, so i'll look at | 00:47 |
| *** sputnik13 has quit IRC | 00:48 | |
| clarkb | SamYaple shouldnt need the functions update anymore | 00:48 |
| anteaya | SamYaple: you need an entry in zuul/layout.yaml for kolla | 00:48 |
| *** daemontool has quit IRC | 00:48 | |
| ruagair | Ok, so i'll look at libapache2-mod-auth-openid and try connecting either it or cauth to openstackid.org. and or launchpad.net, which ever gives me the sanest mapping. | 00:48 |
| ruagair | clarkb fungi ^^ | 00:48 |
| *** daemontool has joined #openstack-infra | 00:48 | |
| *** sputnik13 has joined #openstack-infra | 00:48 | |
| *** tnarg has quit IRC | 00:49 | |
| asselin | zaro, is review-dev.openstack.org up now? it seems to be... | 00:49 |
| SamYaple | clarkb: last time i didnt do the function updates my job stayed in 'non-registered' stater | 00:49 |
| *** mgrosz has joined #openstack-infra | 00:49 | |
| SamYaple | anteaya: done | 00:49 |
| anteaya | SamYaple: great | 00:49 |
| anteaya | asselin: it is up for me | 00:49 |
| *** sridhar_ram1 has quit IRC | 00:50 | |
| *** heyongli has quit IRC | 00:50 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla https://review.openstack.org/242283 | 00:50 |
| *** heyongli has joined #openstack-infra | 00:50 | |
| fungi | ruagair: sounds not too terrible a combination of options to test | 00:51 |
| *** Piet has joined #openstack-infra | 00:51 | |
| anteaya | SamYaple: sorry I mean here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n5640 | 00:51 |
| *** mgrosz has quit IRC | 00:51 | |
| clarkb | SamYaple: there ar ealready rules in place that match multinode | 00:51 |
| ruagair | Plan. | 00:51 |
| anteaya | SamYaple: after gate: put experimental: and the name of the job | 00:51 |
| SamYaple | clarkb: ok ill remove the regex match | 00:51 |
| SamYaple | anteaya: so append expiremental to gate, not replace gate? | 00:52 |
| clarkb | ruagair: yup sounds good to me | 00:52 |
| *** tlian has quit IRC | 00:52 | |
| *** dims has joined #openstack-infra | 00:52 | |
| SamYaple | anteaya: scratch that question | 00:53 |
| *** hemna is now known as hemnafk | 00:53 | |
| * anteaya scratches that question | 00:54 | |
| *** eil397 has quit IRC | 00:55 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla https://review.openstack.org/242283 | 00:56 |
| SamYaple | alright. i believe that may be correct... | 00:57 |
| *** sputnik13 has quit IRC | 00:57 | |
| SamYaple | i fear i may be starting to understand project-config. scary thought! | 00:57 |
| *** tlian has joined #openstack-infra | 00:57 | |
| anteaya | SamYaple: reviewed | 00:58 |
| anteaya | SamYaple: run away | 00:58 |
| *** jamielennox is now known as jamielennox|away | 00:58 | |
| anteaya | SamYaple: one oops the rest looks good | 00:59 |
| SamYaple | i give up. *abandonded* | 01:00 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Enable 2-node gating for Kolla https://review.openstack.org/242283 | 01:00 |
| *** heyongli has quit IRC | 01:00 | |
| SamYaple | thanks for helping me out here. it would have been a while for me to do it alone | 01:00 |
| bardia | how long does nodepool image-build dpc usually take? | 01:01 |
| timrc | jeblair: Sorry I've been heads down on some other stuff today... so... you have a theory that your variances in performance are coming from disk io due to spinny disks? | 01:01 |
| *** heyongli has joined #openstack-infra | 01:01 | |
| anteaya | SamYaple: the run away was in response to the scary thought | 01:02 |
| anteaya | SamYaple: you patch looks good, thanks | 01:02 |
| *** sridhar_ram1 has joined #openstack-infra | 01:03 | |
| anteaya | bardia: good question | 01:03 |
| SamYaple | no thank you anteaya! youre the real hero here | 01:03 |
| anteaya | ha ha ha, never a hero | 01:03 |
| *** sridhar_ram has quit IRC | 01:03 | |
| anteaya | glad you got what you needed | 01:03 |
| SamYaple | clarkb: i believe i addressed your concerns as well. thanks for the input. i may (will probably) hit you up again when i start ahmmering on this | 01:03 |
| clarkb | bardia: I am not sure what dpc is but usually snapshot builds take ~1 hour and dib builds take ~15 minutes with a warm cache | 01:03 |
| anteaya | SamYaple: I don't understand mulitnode like clarkb does | 01:04 |
| SamYaple | anteaya: everyone keeps saying that.... | 01:04 |
| bardia | clarkb: dpc is the image build name given in sample nodepool.yaml | 01:04 |
| * SamYaple wonders what he is getting into | 01:04 | |
| *** sputnik13 has joined #openstack-infra | 01:04 | |
| *** Liuqing has quit IRC | 01:05 | |
| asselin | zaro, sorry but I don't know how to 'review' anything in review-dev.o.o . I see the 'toggle CI' button, but no 'review' button'. where should I look? | 01:05 |
| SamYaple | asselin: are you signed in? | 01:05 |
| asselin | SamYaple, yes | 01:05 |
| clarkb | asselin: the "Reply" button | 01:05 |
| bardia | clarkb: this is the disk-image-create that nodepool is running: /bin/bash /usr/local/bin/disk-image-create -x -t qcow2 --no-tmpfs --qemu-img-options compat=0.10 -o /opt/nodepool_dib/dpc-1446771643 ubuntu vm openstack-repos puppet nodepool-base node-devstack | 01:05 |
| *** sputnik13 has quit IRC | 01:05 | |
| asselin | I see...ok thanks :) | 01:05 |
| clarkb | bardia: ya so that takes about 15-20 minutes for us with a warm cache iirc | 01:06 |
| *** jamielennox|away is now known as jamielennox | 01:06 | |
| clarkb | bardia: a cold cache is going to take longer and depend on how fast your network IO is | 01:06 |
| bardia | oh ok | 01:06 |
| bardia | clarkb: thanks I was worried I was doing something wrong | 01:06 |
| bardia | clarkb: how do you know nodepool successfully connected with the cloud provider? when I run nodepool list it doesn't list anything | 01:08 |
| clarkb | bardia: nodepool list won't show anything until it has an image built to boot | 01:08 |
| bardia | does it list a node after I upload an image to the provider? (nodepool image-upload)? | 01:08 |
| bardia | ah ok | 01:08 |
| bardia | thanks | 01:08 |
| SamYaple | clarkb: question, how does the logs publishing work for multinode? | 01:10 |
| *** heyongli has quit IRC | 01:10 | |
| clarkb | SamYaple: devstack-gate runs an ansible file copy to copy everything to the primary node so jenkins can copy it off | 01:11 |
| *** heyongli has joined #openstack-infra | 01:11 | |
| *** nadya has joined #openstack-infra | 01:13 | |
| clarkb | SamYaple: you should look at devstack-gate to see how it sets thing up (and again I highly recommend using the network setup there at the very least) | 01:15 |
| asselin | zaro, fyi I have 3rd party ci zuul listening to events on review-dev. will setup some test jobs now. http://paste.openstack.org/show/478139/ | 01:15 |
| *** dims has quit IRC | 01:17 | |
| *** cschwede has quit IRC | 01:17 | |
| *** Daisy has joined #openstack-infra | 01:17 | |
| *** nadya has quit IRC | 01:18 | |
| *** Daisy has quit IRC | 01:19 | |
| *** Daisy has joined #openstack-infra | 01:20 | |
| *** heyongli has quit IRC | 01:20 | |
| *** otter768 has joined #openstack-infra | 01:20 | |
| *** heyongli has joined #openstack-infra | 01:21 | |
| *** sdake has quit IRC | 01:22 | |
| *** andersonvom has joined #openstack-infra | 01:23 | |
| anteaya | asselin: you can add line and file comments, nice | 01:24 |
| *** sdake has joined #openstack-infra | 01:24 | |
| asselin | anteaya, yes, but that was more to test my ci getting gerrit events :) not much going on over there | 01:24 |
| *** otter768 has quit IRC | 01:25 | |
| asselin | anteaya, but yes learned how to do that, and also 'reply' to a review | 01:25 |
| *** lyz has joined #openstack-infra | 01:27 | |
| anteaya | nice | 01:27 |
| *** dangers is now known as dangers_away | 01:27 | |
| anteaya | yeah not a lot happening in review-dev atm | 01:27 |
| asselin | anteaya, I have to admit, even knowing what I'm doing, and with common-ci, setting up 3rd party ci is not easy | 01:28 |
| *** andersonvom has quit IRC | 01:28 | |
| *** Daisy has quit IRC | 01:28 | |
| anteaya | yup | 01:28 |
| anteaya | not easy | 01:28 |
| anteaya | I don't foresee a future where it ever will be | 01:29 |
| anteaya | does that make me a bad person? | 01:29 |
| *** pvaneck has quit IRC | 01:29 | |
| anteaya | asselin: did you get my comments? | 01:29 |
| asselin | anteaya, I'm looking forward to zuulv3. Taking out jenkins will simplify a lot. | 01:30 |
| asselin | anteaya, no...let me check | 01:30 |
| anteaya | I too am looking forward to v3 | 01:30 |
| *** Liuqing has joined #openstack-infra | 01:30 | |
| *** heyongli has quit IRC | 01:31 | |
| asselin | anteaya, which patch? | 01:31 |
| anteaya | asselin: top 3 patches on this list: https://review-dev.openstack.org/#/q/reviewer:self | 01:31 |
| anteaya | sorry, that won't help you | 01:31 |
| *** heyongli has joined #openstack-infra | 01:31 | |
| anteaya | try this: https://review-dev.openstack.org/#/q/reviewer:%22akuno+%253Canteaya%2540anteaya.info%253E%22 | 01:32 |
| anteaya | reviewer:self is non-tranferable | 01:32 |
| *** Daisy has joined #openstack-infra | 01:34 | |
| SamYaple | clarkb: I will be reusing everything i can. I am a stong believe in someone-else-should-do-the-work | 01:34 |
| *** rfolco has joined #openstack-infra | 01:35 | |
| *** asettle has quit IRC | 01:36 | |
| *** EricGonczer_ has joined #openstack-infra | 01:36 | |
| asselin | anteaya, sorry..what's your question? I didn't get any e-mail about you commenting. | 01:37 |
| *** yamamoto has quit IRC | 01:38 | |
| anteaya | asselin: did stream-events not emit my comments | 01:38 |
| anteaya | asselin: or I guess you are only looking at patch uploads, right? | 01:39 |
| *** Daisy has quit IRC | 01:39 | |
| asselin | anteaya, oh ok I see. yes it did: 2015-11-06 01:28:20,154 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added gtest-org/test2 master 5366,1> | 01:39 |
| anteaya | yay | 01:39 |
| asselin | 2015-11-06 01:28:35,071 DEBUG zuul.Scheduler: Adding trigger event: <TriggerEvent comment-added gtest-org/gtest master 5370,1> | 01:39 |
| clarkb | jesusaurus: I did +2 that os-loganalyze change. Other reviewrs that would be good to take a look at it are jhesketh and sdague | 01:39 |
| jhesketh | clarkb: got a link? | 01:40 |
| clarkb | jhesketh: https://review.openstack.org/#/c/241058/ | 01:40 |
| *** jyuso1 has joined #openstack-infra | 01:41 | |
| *** heyongli has quit IRC | 01:41 | |
| *** salv-orlando has joined #openstack-infra | 01:41 | |
| *** wenlock has joined #openstack-infra | 01:41 | |
| *** heyongli has joined #openstack-infra | 01:42 | |
| *** bharathm has quit IRC | 01:43 | |
| bardia | what's the next step after building image and uploading to provider? run jenkins dsvm-tempest-full and it will launch a node and run the tests? | 01:43 |
| clarkb | bardia: nodepool should boot a test node and attach it ot jenkins | 01:44 |
| clarkb | then jenkins can run jobs on it | 01:44 |
| bardia | clarkb: does nodepool image-upload take care of this? | 01:45 |
| clarkb | bardia: no, nodepool the daemon does | 01:45 |
| bardia | thanks | 01:46 |
| asselin | bardia, I'm really hoping my documentation explains that.. | 01:47 |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Add acceptance tests for puppet-diskimage_builder. https://review.openstack.org/220262 | 01:47 |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Add acceptance tests for puppet-bandersnatch https://review.openstack.org/221941 | 01:48 |
| bardia | asselin: I think you should mention nodepool image-upload in your docs | 01:48 |
| bardia | asselin: its a missing step | 01:49 |
| bardia | i think | 01:49 |
| bardia | asselin: right after sudo service nodepool start line 206 | 01:49 |
| *** Swami has quit IRC | 01:49 | |
| asselin | bardia, it should happen automatically when you do that | 01:49 |
| bardia | it didn't | 01:50 |
| bardia | bug in nodepool? | 01:50 |
| asselin | bardia, but I can add that regardless. It's helpful in case there are upload errors | 01:50 |
| *** zhenguo has joined #openstack-infra | 01:50 | |
| asselin | bardia, did image-upload work for you? | 01:50 |
| bardia | asselin: it uploaded the image but its failing to start an instance | 01:50 |
| bardia | asselin: trying to figure out why | 01:51 |
| bardia | 2015-11-05 17:50:17,397 ERROR nodepool.NodeLauncher: LaunchStatusException launching node id: 56 in provider: local_01 error: | 01:51 |
| bardia | Traceback (most recent call last): | 01:51 |
| bardia | File "/home/coho/nodepool/nodepool/nodepool.py", line 406, in _run | 01:51 |
| bardia | dt = self.launchNode(session) | 01:51 |
| bardia | File "/home/coho/nodepool/nodepool/nodepool.py", line 474, in launchNode | 01:51 |
| bardia | server['status'])) | 01:51 |
| bardia | traceback from nodepool log | 01:51 |
| *** heyongli has quit IRC | 01:51 | |
| asselin | bardia, can you paste more from the /var/log/nodepool/debug.log here: http://paste.openstack.org/ | 01:52 |
| *** heyongli has joined #openstack-infra | 01:52 | |
| bardia | http://paste.openstack.org/show/478141/ | 01:53 |
| bardia | its just been stuck in a loop where it tries to start an instance on the provider but provider errors out and it just keeps trying | 01:54 |
| asselin | bardia, which provider are you using? | 01:55 |
| bardia | local devstack deployment | 01:55 |
| asselin | bardia, please check your devstack nova screen logs for errors | 01:55 |
| bardia | asselin: checking right now | 01:56 |
| bardia | asselin: what screen number is it? | 01:56 |
| asselin | probably n-cpu | 01:56 |
| jhesketh | clarkb, jesusaurus: reviewed | 01:57 |
| *** otter768 has joined #openstack-infra | 01:57 | |
| *** tiswanso has joined #openstack-infra | 01:57 | |
| *** tiswanso has quit IRC | 01:57 | |
| *** tiswanso has joined #openstack-infra | 01:58 | |
| *** salv-orlando has quit IRC | 01:59 | |
| *** heyongli has quit IRC | 02:01 | |
| *** Daisy has joined #openstack-infra | 02:02 | |
| SamYaple | clarkb: i have followed the devstack-vm-gate.sh code but im a bit fuzzy on how you use ansible to push the ssh auth to the subnodes. If ansible can ssh to these nodes, then aren't the keys already setup? | 02:02 |
| *** heyongli has joined #openstack-infra | 02:02 | |
| asselin | SamYaple, I belive the ssh auth should already be set on the primary and subnodes. same user and ssh keys | 02:05 |
| SamYaple | then why does this function exist https://github.com/openstack-infra/devstack-gate/blob/2e63463f95dc7c3d5e527a1d9a6dfe4d9337b21c/devstack-vm-gate.sh#L53 | 02:06 |
| SamYaple | asselin: if they keys are already setup then its fine | 02:06 |
| clarkb | because libvirt | 02:07 |
| clarkb | so ansible is fine but libvirt requires extra help | 02:07 |
| SamYaple | ok that explains the chicken and the egg i was finding | 02:08 |
| SamYaple | it all makes sense now how its setup | 02:08 |
| clarkb | one is test env layer the other is cloud under test layer | 02:09 |
| clarkb | separate concerms | 02:09 |
| *** Daisy has quit IRC | 02:09 | |
| *** andersonvom has joined #openstack-infra | 02:09 | |
| *** Daisy has joined #openstack-infra | 02:09 | |
| SamYaple | its a pretty decent piece of setup work there clarkb. my stuff was far more basic because i could control the networks attached to the vms on a single cloud | 02:11 |
| *** heyongli has quit IRC | 02:12 | |
| *** heyongli has joined #openstack-infra | 02:12 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Add documentation for third-party-ci setup https://review.openstack.org/227584 | 02:13 |
| clarkb | SamYaple we dont have that kind of control heck one cloud gives us a /32 | 02:14 |
| *** Daisy has quit IRC | 02:14 | |
| *** wenlock has quit IRC | 02:14 | |
| SamYaple | right which is why i am admiring your work. mine i could expect thigns :) | 02:14 |
| *** unicell has quit IRC | 02:17 | |
| *** sridhar_ram1 has quit IRC | 02:17 | |
| *** woodster_ has quit IRC | 02:19 | |
| *** heyongli has quit IRC | 02:22 | |
| bardia | interesting in devstack I can create a m1.medium instance but not m1.large. it throws | 02:23 |
| bardia | No valid host was found. There are not enough hosts available. | 02:23 |
| bardia | Code | 02:23 |
| bardia | 500 | 02:23 |
| bardia | Details | 02:23 |
| bardia | File "/opt/stack/nova/nova/conductor/manager.py", line 359, in build_instances request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/utils.py", line 343, in wrapped return func(*args, **kwargs) File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 52, in select_destinations context, request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 37, in __ | 02:23 |
| bardia | run_method return getattr(self.instance, __name)(*args, **kwargs) File "/opt/stack/nova/nova/scheduler/client/query.py", line 34, in select_destinations context, request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/rpcapi.py", line 122, in select_destinations request_spec=request_spec, filter_properties=filter_properties) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py | 02:23 |
| bardia | ", line 158, in call retry=self.retry) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send timeout=timeout, retry=retry) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send retry=retry) File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send raise result | 02:23 |
| bardia | Created | 02:23 |
| bardia | Nov. 6, 2015, 2:20 a.m. | 02:23 |
| bardia | anyone faced the same issue? | 02:23 |
| *** heyongli has joined #openstack-infra | 02:23 | |
| clarkb | bardia please dont paste like that, use a paste service. you probably dont have enough ram | 02:23 |
| clarkb | check the nova scheduler log | 02:23 |
| bardia | clarkb: sorry will use paste.openstack next time | 02:23 |
| bardia | clarkb: I have 16 GB of ram | 02:24 |
| *** sridhar_ram has joined #openstack-infra | 02:24 | |
| clarkb | check the nova scheduler log | 02:24 |
| *** asettle has joined #openstack-infra | 02:27 | |
| *** salv-orlando has joined #openstack-infra | 02:27 | |
| *** sridhar_ram1 has joined #openstack-infra | 02:27 | |
| *** yamamoto has joined #openstack-infra | 02:28 | |
| *** sridhar_ram has quit IRC | 02:28 | |
| *** Daisy has joined #openstack-infra | 02:28 | |
| *** gyee has quit IRC | 02:31 | |
| *** heyongli has quit IRC | 02:32 | |
| *** heyongli has joined #openstack-infra | 02:33 | |
| bardia | clarkb: yeah didn't have enough hdd for m1.large | 02:33 |
| bardia | clarkb: thanks for your hel | 02:33 |
| bardia | help* | 02:33 |
| *** bpokorny_ has joined #openstack-infra | 02:33 | |
| *** mtanino has quit IRC | 02:34 | |
| *** bpokorny has quit IRC | 02:37 | |
| *** bpokorny_ has quit IRC | 02:38 | |
| *** barnaby has quit IRC | 02:39 | |
| *** heyongli has quit IRC | 02:42 | |
| *** Daisy has quit IRC | 02:42 | |
| *** heyongli has joined #openstack-infra | 02:44 | |
| *** rfolco has quit IRC | 02:44 | |
| *** otter768 has quit IRC | 02:44 | |
| *** sridhar_ram1 has quit IRC | 02:45 | |
| *** inteq has joined #openstack-infra | 02:45 | |
| openstackgerrit | Liuqing Jing proposed openstack-infra/project-config: Add Evoque client project https://review.openstack.org/242309 | 02:45 |
| *** mahatic has joined #openstack-infra | 02:45 | |
| *** sabeen1 has joined #openstack-infra | 02:49 | |
| *** salv-orlando has quit IRC | 02:49 | |
| *** rebase has quit IRC | 02:51 | |
| *** barnaby has joined #openstack-infra | 02:52 | |
| *** heyongli has quit IRC | 02:53 | |
| *** heyongli has joined #openstack-infra | 02:54 | |
| jesusaurus | jhesketh: why would it be better to build a new generator class? | 02:54 |
| *** btully has quit IRC | 02:55 | |
| *** wenlock has joined #openstack-infra | 03:00 | |
| *** heyongli has quit IRC | 03:00 | |
| *** Daisy has joined #openstack-infra | 03:04 | |
| jhesketh | jesusaurus: well not necessarily a whole new class, but a generator so that we can stream the contents as they are read rather than reading them all and then passing them to the wsgi app | 03:05 |
| *** btully has joined #openstack-infra | 03:05 | |
| jesusaurus | jhesketh: but you need to read the entire contents in order to return only the last X bytes | 03:05 |
| jhesketh | jesusaurus: for that use case, yes, but in the case of reading the first X bytes, they could be done in parts | 03:06 |
| *** Daisy has quit IRC | 03:08 | |
| *** rguillebert has quit IRC | 03:08 | |
| jesusaurus | jhesketh: so you think it would be better if the type of generator created happened after the logic of determining what kind of range should be returned? | 03:10 |
| *** EricGonczer_ has quit IRC | 03:13 | |
| jhesketh | jesusaurus: that might make it a bit easier... but you could just solve it for the general case and it shouldn't be any less efficient | 03:14 |
| jhesketh | consider if somebody wants the bytes from A -> X-1 where X is the last byte | 03:14 |
| jhesketh | there is no way to tell they wanted nearly all of them, so you may as well start scrolling through until you are ready to yield and continue from there | 03:14 |
| jhesketh | oh, I see the issue, needing to determine the negatives... yes, handling that case differently would probably be best | 03:15 |
| jhesketh | it could, however, be optimised by looking at the size of the file on disk or from swift | 03:15 |
| jhesketh | which I believe is passed through | 03:15 |
| clarkb | you cant | 03:16 |
| clarkb | because we add so much junk | 03:16 |
| jhesketh | ah yes, you're right | 03:16 |
| clarkb | and its line based | 03:16 |
| clarkb | which swidt and disk dont count for you | 03:17 |
| clarkb | so ypu have to iterate through regardless | 03:18 |
| *** salv-orlando has joined #openstack-infra | 03:19 | |
| clarkb | I dont thinkwe get better performance changing it | 03:19 |
| *** baoli has joined #openstack-infra | 03:19 | |
| *** rebase has joined #openstack-infra | 03:19 | |
| *** rebase has quit IRC | 03:19 | |
| *** tnarg_ has quit IRC | 03:20 | |
| jhesketh | clarkb: we do in the case where it's not a negative amount requested | 03:20 |
| jhesketh | we can start streaming the results rather than bufferring it all and stop once we reach either the end or the amount of bytes requested | 03:20 |
| clarkb | arent they equivalent? | 03:21 |
| *** sabeen1 has quit IRC | 03:21 | |
| *** bardia has quit IRC | 03:21 | |
| jhesketh | you have to wait for osloganalyze to compute the end before you start receiving a respons | 03:21 |
| *** otter768 has joined #openstack-infra | 03:22 | |
| clarkb | ok so you want the appemds changed to yields? | 03:22 |
| jhesketh | right, rather than building a list, yield it out | 03:23 |
| *** boris-42 has joined #openstack-infra | 03:24 | |
| jesusaurus | but then we would need to special case bytes=-5 rather than a general solution, and only to optimize the first byte received, the last byte would still be pretty much the same | 03:29 |
| clarkb | jesusaurus you could put the yielding in an inner method that is returned if not negative index else convert to list and count backwards | 03:30 |
| clarkb | so a fairly small special case but yes would be special | 03:30 |
| *** Daisy has joined #openstack-infra | 03:32 | |
| *** baoli has quit IRC | 03:36 | |
| *** sdake has quit IRC | 03:36 | |
| *** Daisy has quit IRC | 03:37 | |
| *** salv-orlando has quit IRC | 03:40 | |
| *** skolekonov has quit IRC | 03:48 | |
| *** mohankumar has joined #openstack-infra | 03:48 | |
| *** skolekonov has joined #openstack-infra | 03:50 | |
| *** aukhan has joined #openstack-infra | 03:54 | |
| *** Daisy has joined #openstack-infra | 03:54 | |
| *** mohankumar has quit IRC | 03:56 | |
| *** baoli has joined #openstack-infra | 03:57 | |
| *** Daisy has quit IRC | 03:57 | |
| *** Sree has joined #openstack-infra | 03:58 | |
| *** links has joined #openstack-infra | 03:59 | |
| *** tlian has quit IRC | 04:10 | |
| *** tiswanso has quit IRC | 04:14 | |
| *** denisra has quit IRC | 04:18 | |
| *** denisra has joined #openstack-infra | 04:18 | |
| *** fawadkhaliq has joined #openstack-infra | 04:20 | |
| *** FallenPegasus has quit IRC | 04:22 | |
| *** rlandy has quit IRC | 04:28 | |
| *** otter768 has quit IRC | 04:29 | |
| *** sdake has joined #openstack-infra | 04:35 | |
| *** wenlock_ has joined #openstack-infra | 04:35 | |
| *** FallenPegasus has joined #openstack-infra | 04:38 | |
| *** salv-orlando has joined #openstack-infra | 04:40 | |
| *** wuhg has joined #openstack-infra | 04:43 | |
| *** salv-orlando has quit IRC | 04:49 | |
| *** amotoki has joined #openstack-infra | 04:51 | |
| *** baoli has quit IRC | 04:51 | |
| *** sdake has quit IRC | 04:53 | |
| *** sdake has joined #openstack-infra | 04:58 | |
| *** vilobhmm11 has joined #openstack-infra | 04:59 | |
| *** Ramanjaneya has joined #openstack-infra | 04:59 | |
| *** ybathia has joined #openstack-infra | 05:01 | |
| *** vilobhmm11 has quit IRC | 05:02 | |
| *** ybathia_ has joined #openstack-infra | 05:02 | |
| *** Liuqing has quit IRC | 05:05 | |
| *** ybathia has quit IRC | 05:06 | |
| *** ybathia_ is now known as ybathia | 05:06 | |
| *** Liuqing has joined #openstack-infra | 05:07 | |
| *** btully has quit IRC | 05:09 | |
| *** mohankumar has joined #openstack-infra | 05:14 | |
| *** vilobhmm11 has joined #openstack-infra | 05:20 | |
| *** wenlock_ has quit IRC | 05:25 | |
| *** asettle has quit IRC | 05:27 | |
| *** mohankumar has quit IRC | 05:27 | |
| *** mrmartin has joined #openstack-infra | 05:29 | |
| *** mohankumar has joined #openstack-infra | 05:31 | |
| *** nadya has joined #openstack-infra | 05:39 | |
| *** heyongli has joined #openstack-infra | 05:40 | |
| *** heyongli has quit IRC | 05:44 | |
| *** heyongli has joined #openstack-infra | 05:44 | |
| *** nikhil_k has joined #openstack-infra | 05:48 | |
| *** nikhil has quit IRC | 05:48 | |
| *** mahatic_ has joined #openstack-infra | 05:50 | |
| *** btully has joined #openstack-infra | 05:53 | |
| *** mahatic has quit IRC | 05:54 | |
| *** wenlock1 has joined #openstack-infra | 05:54 | |
| *** heyongli has quit IRC | 05:54 | |
| *** heyongli has joined #openstack-infra | 05:55 | |
| *** bdemers has quit IRC | 05:57 | |
| *** sdake has quit IRC | 06:00 | |
| *** sdake has joined #openstack-infra | 06:00 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum https://review.openstack.org/242341 | 06:02 |
| *** BharatK has joined #openstack-infra | 06:03 | |
| *** oomichi_ has quit IRC | 06:03 | |
| *** heyongli has quit IRC | 06:05 | |
| *** heyongli has joined #openstack-infra | 06:05 | |
| *** bdemers has joined #openstack-infra | 06:07 | |
| *** hdd has joined #openstack-infra | 06:08 | |
| *** legoktm has left #openstack-infra | 06:09 | |
| *** nadya has quit IRC | 06:12 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum https://review.openstack.org/242341 | 06:14 |
| *** heyongli has quit IRC | 06:15 | |
| *** heyongli has joined #openstack-infra | 06:15 | |
| *** mrmartin has quit IRC | 06:16 | |
| *** vilobhmm111 has joined #openstack-infra | 06:18 | |
| *** vilobhmm11 has quit IRC | 06:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Update vitrage-specs https://review.openstack.org/241899 | 06:21 |
| *** heyongli has quit IRC | 06:21 | |
| *** heyongli has joined #openstack-infra | 06:22 | |
| *** unicell has joined #openstack-infra | 06:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/237894 | 06:24 |
| *** heyongli has quit IRC | 06:25 | |
| *** heyongli has joined #openstack-infra | 06:25 | |
| *** harlowja_at_home has joined #openstack-infra | 06:26 | |
| *** heyongli has quit IRC | 06:27 | |
| *** yonglihe has joined #openstack-infra | 06:27 | |
| *** salv-orlando has joined #openstack-infra | 06:29 | |
| *** otter768 has joined #openstack-infra | 06:30 | |
| *** openstackgerrit has quit IRC | 06:31 | |
| *** FallenPegasus has quit IRC | 06:32 | |
| *** openstackgerrit has joined #openstack-infra | 06:32 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum https://review.openstack.org/242341 | 06:34 |
| *** otter768 has quit IRC | 06:34 | |
| *** yonglihe has quit IRC | 06:35 | |
| *** yonglihe has joined #openstack-infra | 06:36 | |
| *** harlowja_at_home has quit IRC | 06:37 | |
| *** david-lyle has joined #openstack-infra | 06:40 | |
| *** salv-orlando has quit IRC | 06:41 | |
| *** tonytan4ever has quit IRC | 06:44 | |
| *** peterlisak has joined #openstack-infra | 06:44 | |
| *** david-lyle has quit IRC | 06:45 | |
| *** yonglihe has quit IRC | 06:46 | |
| *** yonglihe has joined #openstack-infra | 06:46 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: [WIP] Add new pipeline for magnum https://review.openstack.org/242341 | 06:46 |
| *** mahatic_ has quit IRC | 06:47 | |
| *** mahatic has joined #openstack-infra | 06:49 | |
| *** fedexo has joined #openstack-infra | 06:51 | |
| *** gildub has quit IRC | 06:53 | |
| *** vilobhmm111 has quit IRC | 06:54 | |
| *** yonglihe has quit IRC | 06:56 | |
| *** yonglihe has joined #openstack-infra | 06:56 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 06:58 |
| *** jamielennox is now known as jamielennox|away | 06:59 | |
| *** baoli has joined #openstack-infra | 07:03 | |
| *** unicell1 has joined #openstack-infra | 07:03 | |
| *** unicell has quit IRC | 07:04 | |
| *** yonglihe has quit IRC | 07:06 | |
| *** yonglihe has joined #openstack-infra | 07:06 | |
| *** ybathia has quit IRC | 07:07 | |
| *** baoli has quit IRC | 07:08 | |
| *** gampel has joined #openstack-infra | 07:10 | |
| *** Liuqing has quit IRC | 07:10 | |
| *** e0ne has joined #openstack-infra | 07:10 | |
| *** Liuqing has joined #openstack-infra | 07:11 | |
| *** sdake has quit IRC | 07:14 | |
| *** sdake has joined #openstack-infra | 07:14 | |
| *** yonglihe has quit IRC | 07:16 | |
| *** yonglihe has joined #openstack-infra | 07:17 | |
| *** salv-orlando has joined #openstack-infra | 07:17 | |
| *** fawadkhaliq has quit IRC | 07:19 | |
| *** salv-orlando has quit IRC | 07:19 | |
| *** salv-orlando has joined #openstack-infra | 07:20 | |
| *** e0ne has quit IRC | 07:21 | |
| *** eil397 has joined #openstack-infra | 07:25 | |
| *** andersonvom has quit IRC | 07:26 | |
| *** mohankumar has quit IRC | 07:26 | |
| *** sdake has quit IRC | 07:26 | |
| *** yonglihe has quit IRC | 07:27 | |
| *** yonglihe has joined #openstack-infra | 07:27 | |
| *** sdake has joined #openstack-infra | 07:27 | |
| *** wenlock1 has quit IRC | 07:30 | |
| *** rossella_s has joined #openstack-infra | 07:33 | |
| *** fawadkhaliq has joined #openstack-infra | 07:34 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Add job for the ovs mechanism driver with native mode https://review.openstack.org/221143 | 07:35 |
| *** Daisy has joined #openstack-infra | 07:36 | |
| *** someara2 has joined #openstack-infra | 07:36 | |
| *** yonglihe has quit IRC | 07:37 | |
| *** abregman has joined #openstack-infra | 07:37 | |
| *** yonglihe has joined #openstack-infra | 07:37 | |
| *** k4n0 has joined #openstack-infra | 07:39 | |
| *** sdake_ has joined #openstack-infra | 07:40 | |
| *** someara2 has quit IRC | 07:41 | |
| *** someara2 has joined #openstack-infra | 07:41 | |
| *** sdake has quit IRC | 07:42 | |
| *** someara2 has quit IRC | 07:45 | |
| *** someara2 has joined #openstack-infra | 07:46 | |
| *** yonglihe has quit IRC | 07:47 | |
| *** yonglihe has joined #openstack-infra | 07:47 | |
| *** shardy has joined #openstack-infra | 07:49 | |
| *** fawadkhaliq has quit IRC | 07:52 | |
| *** cody-somerville has joined #openstack-infra | 07:55 | |
| *** Guest65445 is now known as tdurakov | 07:56 | |
| tdurakov | clarkb, hi, are you there? | 07:56 |
| *** yonglihe has quit IRC | 07:57 | |
| *** yonglihe has joined #openstack-infra | 07:58 | |
| *** fawadkhaliq has joined #openstack-infra | 07:58 | |
| *** ihrachys has joined #openstack-infra | 07:58 | |
| *** mrmartin has joined #openstack-infra | 07:59 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 08:03 |
| *** devvesa has joined #openstack-infra | 08:03 | |
| *** someara2 has quit IRC | 08:04 | |
| *** eil397 has quit IRC | 08:06 | |
| *** yonglihe has quit IRC | 08:08 | |
| *** yonglihe has joined #openstack-infra | 08:08 | |
| *** matrohon has joined #openstack-infra | 08:11 | |
| *** fedexo has quit IRC | 08:13 | |
| *** yonglihe has quit IRC | 08:18 | |
| *** yonglihe has joined #openstack-infra | 08:18 | |
| *** HeOS has quit IRC | 08:19 | |
| *** placko has joined #openstack-infra | 08:20 | |
| *** mohankumar has joined #openstack-infra | 08:23 | |
| *** gampel has quit IRC | 08:24 | |
| *** kiran-r has joined #openstack-infra | 08:27 | |
| *** mohankumar has quit IRC | 08:28 | |
| *** yonglihe has quit IRC | 08:28 | |
| *** yonglihe has joined #openstack-infra | 08:28 | |
| *** otter768 has joined #openstack-infra | 08:31 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-graphite: Add support for apache >= 2.4 https://review.openstack.org/232991 | 08:32 |
| *** sdake_ has quit IRC | 08:33 | |
| *** binbincong has quit IRC | 08:34 | |
| *** btully has quit IRC | 08:34 | |
| *** abregman has quit IRC | 08:35 | |
| *** otter768 has quit IRC | 08:36 | |
| *** zigo has quit IRC | 08:37 | |
| *** arxcruz has joined #openstack-infra | 08:37 | |
| *** Daisy has quit IRC | 08:38 | |
| *** ZZelle has quit IRC | 08:38 | |
| *** yonglihe has quit IRC | 08:38 | |
| *** ZZelle has joined #openstack-infra | 08:38 | |
| *** Daisy has joined #openstack-infra | 08:38 | |
| *** yonglihe has joined #openstack-infra | 08:39 | |
| *** aeng has quit IRC | 08:42 | |
| *** zigo has joined #openstack-infra | 08:42 | |
| *** openstackstatus has quit IRC | 08:42 | |
| *** Daisy has quit IRC | 08:43 | |
| *** jlanoux has joined #openstack-infra | 08:44 | |
| *** binbincong has joined #openstack-infra | 08:45 | |
| *** hashar-away is now known as hashar | 08:45 | |
| *** yonglihe has quit IRC | 08:48 | |
| openstackgerrit | Merged openstack-infra/project-config: Change acl for freezer* projects https://review.openstack.org/242025 | 08:49 |
| *** yonglihe has joined #openstack-infra | 08:49 | |
| *** jordanP has joined #openstack-infra | 08:51 | |
| *** ifarkas has joined #openstack-infra | 08:51 | |
| *** Daisy has joined #openstack-infra | 08:52 | |
| *** Daisy has quit IRC | 08:54 | |
| *** kengo_sakai has joined #openstack-infra | 08:55 | |
| *** sergmelikyan has joined #openstack-infra | 08:57 | |
| *** yonglihe has quit IRC | 08:58 | |
| *** fhubik has joined #openstack-infra | 08:59 | |
| *** yonglihe has joined #openstack-infra | 08:59 | |
| *** Ala has joined #openstack-infra | 08:59 | |
| *** dgonzalez has quit IRC | 09:03 | |
| *** dgonzalez has joined #openstack-infra | 09:03 | |
| *** nadya has joined #openstack-infra | 09:07 | |
| *** yonglihe has quit IRC | 09:07 | |
| openstackgerrit | Merged openstack-infra/project-config: Run Ironic grenade jobs with IPA https://review.openstack.org/241426 | 09:11 |
| *** links has quit IRC | 09:11 | |
| *** derekh has joined #openstack-infra | 09:11 | |
| *** Liuqing has quit IRC | 09:12 | |
| *** sorantis has joined #openstack-infra | 09:13 | |
| *** nadya has quit IRC | 09:13 | |
| sorantis | Hi! | 09:13 |
| sorantis | Could you guys help me with updating gerrit group members? | 09:14 |
| sorantis | the project I’ve created is https://review.openstack.org/#/admin/projects/openstack/kingbird | 09:14 |
| sorantis | kingbird-core and kingbird-release are the groups. | 09:14 |
| sorantis | Could you please add the users [sorantis, joehuang] to both? | 09:15 |
| *** bauwser has joined #openstack-infra | 09:15 | |
| *** asselin_ has joined #openstack-infra | 09:18 | |
| *** jistr has joined #openstack-infra | 09:19 | |
| yolanda | hi sorantis, sure | 09:20 |
| sorantis | ah, great! I was just going to write an email :) | 09:21 |
| sorantis | thanks! | 09:21 |
| *** achanda has quit IRC | 09:21 | |
| yolanda | sorantis, do you have the change for project creation? | 09:21 |
| yolanda | so i can verify | 09:21 |
| sorantis | you mean this one? https://review.openstack.org/#/c/235253/ | 09:21 |
| yolanda | sorantis, as project creation is done by Dimitri Mazmanov, i'd normally add him to that group | 09:23 |
| sorantis | that’s me :) | 09:23 |
| yolanda | ah ok :) | 09:23 |
| sorantis | https://launchpad.net/~sorantis | 09:23 |
| yolanda | done | 09:24 |
| sorantis | super! thank you! | 09:25 |
| sorantis | I still have only +1/-1 though | 09:26 |
| sorantis | oh | 09:26 |
| sorantis | got it now | 09:26 |
| *** e0ne has joined #openstack-infra | 09:27 | |
| *** fhubik is now known as fhubik_brb | 09:28 | |
| *** HeOS has joined #openstack-infra | 09:30 | |
| *** bdemers has quit IRC | 09:30 | |
| yolanda | ah, you need to refresh cache | 09:32 |
| *** camunoz has quit IRC | 09:33 | |
| *** sergmelikyan has quit IRC | 09:35 | |
| *** Sree has quit IRC | 09:36 | |
| *** Sree has joined #openstack-infra | 09:37 | |
| *** cdent has joined #openstack-infra | 09:38 | |
| *** mgrosz has joined #openstack-infra | 09:46 | |
| *** mgrosz has quit IRC | 09:47 | |
| *** fhubik_brb is now known as fhubik | 09:49 | |
| SamYaple | o/ yolanda how are you | 09:58 |
| *** zhenguo has quit IRC | 09:58 | |
| *** rguillebert has joined #openstack-infra | 10:00 | |
| *** nijaba has quit IRC | 10:01 | |
| *** nadya has joined #openstack-infra | 10:03 | |
| *** someara2 has joined #openstack-infra | 10:03 | |
| asselin_ | anyone know how to get past this error? OpenStackCloudException: Error fetching image list: HTTPMultipleChoices (HTTP 300) Requested version of OpenStack Images API is not available. (Inner Exception: ) | 10:08 |
| asselin_ | http://paste.openstack.org/show/478167/ | 10:08 |
| *** kiran-r has quit IRC | 10:08 | |
| *** fawadkhaliq has quit IRC | 10:10 | |
| asselin_ | mordred, ^^ ? | 10:11 |
| *** mfalatic has joined #openstack-infra | 10:13 | |
| *** jyuso1 has quit IRC | 10:15 | |
| *** mfalatic_ has quit IRC | 10:15 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Rename project networking-bagpipe(-l2) https://review.openstack.org/237936 | 10:16 |
| *** jyuso1 has joined #openstack-infra | 10:16 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move #openstack-ceilometer to #openstack-telemetry https://review.openstack.org/242422 | 10:18 |
| *** julim_ has joined #openstack-infra | 10:19 | |
| *** tdurakov has quit IRC | 10:20 | |
| *** Guest13749 has joined #openstack-infra | 10:20 | |
| *** tdurakov has joined #openstack-infra | 10:21 | |
| *** julim has quit IRC | 10:22 | |
| openstackgerrit | John Garbutt proposed openstack-infra/project-config: Add tempest-dsvm-lxc as experimental https://review.openstack.org/226565 | 10:27 |
| *** otherwiseguy has quit IRC | 10:28 | |
| *** tdurakov has quit IRC | 10:29 | |
| *** aukhan has quit IRC | 10:30 | |
| *** aukhan has joined #openstack-infra | 10:30 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Disable py34 for networking-cisco kilo and juno https://review.openstack.org/241994 | 10:30 |
| *** otherwiseguy has joined #openstack-infra | 10:30 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move #openstack-ceilometer to #openstack-telemetry https://review.openstack.org/242422 | 10:31 |
| *** sergmelikyan has joined #openstack-infra | 10:31 | |
| *** otter768 has joined #openstack-infra | 10:32 | |
| *** someara2 has quit IRC | 10:34 | |
| *** ldnunes has joined #openstack-infra | 10:36 | |
| *** otter768 has quit IRC | 10:36 | |
| asselin_ | mordred, nevermind figured it out: needed profile:hp in clouds.yaml. | 10:36 |
| *** Daisy has joined #openstack-infra | 10:37 | |
| *** bdemers has joined #openstack-infra | 10:37 | |
| *** ihrachys has quit IRC | 10:37 | |
| *** ihrachys has joined #openstack-infra | 10:38 | |
| *** sergmelikyan has quit IRC | 10:40 | |
| mordred | asselin_: ++ | 10:41 |
| mordred | asselin_: dude. where are you? I feel like you should not be awake | 10:41 |
| *** Daisy has quit IRC | 10:41 | |
| asselin_ | mordred, jet lag | 10:41 |
| asselin_ | I should be awake | 10:42 |
| asselin_ | should not | 10:42 |
| mordred | jeblair, ruagair: my only preference for cauth was that it was what our enovance friends were using in software-factory already | 10:42 |
| asselin_ | mordred, any tips for this? OpenStackCloudException: Image creation failed: 403 Forbidden: Access was denied to this resource. (HTTP 403) (Inner Exception: ) | 10:43 |
| *** dzamboni has joined #openstack-infra | 10:43 | |
| mordred | asselin_: you have to request image uploads be turned on on your hp cloud account | 10:44 |
| mordred | asselin_: it's not enabled by default | 10:44 |
| mordred | jeblair, clarkb, ruagair: "I'm looking for additional contributers to take this project over! Email me (I'm bamuller on gmail) if you can help out. Thanks!" | 10:44 |
| *** someara2 has joined #openstack-infra | 10:44 | |
| asselin_ | mordred, cool thanks | 10:44 |
| mordred | jeblair, clarkb, ruagair: so, if libapache-mod-auth-openid works, we might want to adopt it | 10:44 |
| *** someara2 has quit IRC | 10:45 | |
| *** shardy has quit IRC | 10:46 | |
| mordred | however, libapache-mod-auth-openid looks great | 10:47 |
| *** yamamoto has quit IRC | 10:48 | |
| *** someara2 has joined #openstack-infra | 10:48 | |
| *** nadya has quit IRC | 10:49 | |
| *** jamielennox|away is now known as jamielennox | 10:49 | |
| *** Sree has quit IRC | 10:51 | |
| *** someara2 has quit IRC | 10:51 | |
| daemontool | Hi all, how can I add myself or other the this group? https://review.openstack.org/#/admin/groups/1172,members | 10:52 |
| *** someara2 has joined #openstack-infra | 10:54 | |
| *** electrofelix has joined #openstack-infra | 10:57 | |
| *** kozhukalov_ has joined #openstack-infra | 10:58 | |
| *** gildub has joined #openstack-infra | 10:59 | |
| *** pc_m has joined #openstack-infra | 11:01 | |
| *** sergmelikyan has joined #openstack-infra | 11:02 | |
| *** aysyd has joined #openstack-infra | 11:04 | |
| *** aukhan has quit IRC | 11:05 | |
| *** sorantis has quit IRC | 11:05 | |
| *** aysyd has quit IRC | 11:08 | |
| *** aysyd has joined #openstack-infra | 11:09 | |
| *** jaosorior has joined #openstack-infra | 11:10 | |
| *** jamielennox is now known as jamielennox|away | 11:11 | |
| *** wenlock has quit IRC | 11:14 | |
| *** sergmelikyan has quit IRC | 11:14 | |
| *** someara2 has quit IRC | 11:14 | |
| *** achanda has joined #openstack-infra | 11:22 | |
| *** achanda has quit IRC | 11:26 | |
| *** mwagner has quit IRC | 11:27 | |
| *** dims has joined #openstack-infra | 11:27 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Set ZUUL_BRANCH and ZUUL_REF https://review.openstack.org/242449 | 11:30 |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** yamamoto_ has joined #openstack-infra | 11:33 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 11:34 |
| *** yamamoto has quit IRC | 11:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Let os-client-config handle session creation https://review.openstack.org/240734 | 11:37 |
| ihrachys | hi folks. I need some advice on ZUUL_BRANCH variable. I see that it's passed in gate lbaas jobs for Liberty but not in periodic stable jobs. That results in the latter jobs to fetch master neutron and run against it (and fail). I wonder whether there is some reason the var is not passed into periodic jobs, and if not, how can I set it? | 11:43 |
| *** pbourke has joined #openstack-infra | 11:45 | |
| daemontool | anyone around can help me understand how can I add members to this group? https://review.openstack.org/#/admin/groups/1172,members | 11:50 |
| *** Liuqing has joined #openstack-infra | 11:54 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Bump upper-constraint for os-client-config https://review.openstack.org/242160 | 11:59 |
| *** baoli has joined #openstack-infra | 12:05 | |
| *** dprince has joined #openstack-infra | 12:06 | |
| *** fhubik is now known as fhubik_brb | 12:08 | |
| *** baoli has quit IRC | 12:09 | |
| *** rfolco has joined #openstack-infra | 12:09 | |
| *** gampel has joined #openstack-infra | 12:11 | |
| *** Daisy has joined #openstack-infra | 12:13 | |
| *** Daisy has quit IRC | 12:14 | |
| *** sorantis has joined #openstack-infra | 12:17 | |
| *** gampel has quit IRC | 12:18 | |
| *** pc_m has quit IRC | 12:20 | |
| *** pc_m has joined #openstack-infra | 12:20 | |
| *** Sree has joined #openstack-infra | 12:22 | |
| *** aysyd has quit IRC | 12:23 | |
| *** aysyd has joined #openstack-infra | 12:24 | |
| *** achanda has joined #openstack-infra | 12:24 | |
| *** smarcet has joined #openstack-infra | 12:24 | |
| *** rossella_s has quit IRC | 12:25 | |
| *** k4n0 has quit IRC | 12:27 | |
| *** dingyichen has quit IRC | 12:29 | |
| *** achanda has quit IRC | 12:30 | |
| *** wuhg has quit IRC | 12:31 | |
| *** someara2 has joined #openstack-infra | 12:31 | |
| *** otter768 has joined #openstack-infra | 12:32 | |
| *** zigo has quit IRC | 12:36 | |
| *** zigo has joined #openstack-infra | 12:37 | |
| *** abregman has joined #openstack-infra | 12:37 | |
| *** otter768 has quit IRC | 12:37 | |
| *** weihan has joined #openstack-infra | 12:38 | |
| *** fhubik_brb is now known as fhubik | 12:39 | |
| *** rossella_s has joined #openstack-infra | 12:41 | |
| *** Sree has quit IRC | 12:43 | |
| anteaya | daemontool: first a gerrit admin has to add the owner of the patch that created that group | 12:43 |
| *** Sree has joined #openstack-infra | 12:43 | |
| anteaya | daemontool: then the first person in the group can add others | 12:44 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 12:44 |
| sdague | greghaynes / fungi ^^^ - still not passing tests, but it's at a different failure now | 12:45 |
| *** abregman has quit IRC | 12:45 | |
| *** thorst has joined #openstack-infra | 12:45 | |
| anteaya | sdague: morning | 12:45 |
| anteaya | so next week I was thinking of convening the first patch of folks following the learning how to debug the gate thread | 12:46 |
| anteaya | sdague: was thinking of booking the sprint channel for two hours next thursday | 12:46 |
| anteaya | sdague: might you have a window next thursday at all? | 12:46 |
| anteaya | s/first patch/first batch | 12:47 |
| *** tdurakov has joined #openstack-infra | 12:47 | |
| sdague | honestly, I don't know. I'm still unpacking what I signed up for at summit. | 12:47 |
| anteaya | fair enough | 12:47 |
| anteaya | well ideally I would pick a time that works for you, mriedem, and mtreinish | 12:48 |
| anteaya | not that you have to commit to the whole thing, but at least not conflicting with other things | 12:49 |
| tdurakov | sdague, hi | 12:49 |
| anteaya | sdague: if you can let me know what works for you, that would be great | 12:49 |
| anteaya | wasn't going to post to the list until next week | 12:49 |
| tdurakov | sdague, working on live-migration job for nova, need help with local env | 12:49 |
| *** trown|outttypeww is now known as trown | 12:50 | |
| *** weihan has quit IRC | 12:51 | |
| *** kengo_sakai has quit IRC | 12:53 | |
| anteaya | fungi clarkb this patch for today's rename was respun: https://review.openstack.org/#/c/237936/ it looks fine to me, I have pinged mestery in -neutron to ask him to review | 12:56 |
| anteaya | sorry I don't have time and attention to update the etherpad to include this patch | 12:57 |
| openstackgerrit | Alexey Deryugin proposed openstack-infra/project-config: Add missing project openstack/yaql to zuul layout https://review.openstack.org/242469 | 12:57 |
| *** boris-42 has quit IRC | 12:58 | |
| *** sorantis has quit IRC | 12:59 | |
| *** Ramanjaneya has quit IRC | 12:59 | |
| *** gordc has joined #openstack-infra | 13:00 | |
| anteaya | sdague: I worded that poorly, you don't have to commit to anything, but it would be great for me if you could attend | 13:01 |
| *** gildub has quit IRC | 13:02 | |
| *** EricGonczer_ has joined #openstack-infra | 13:03 | |
| *** kgiusti has joined #openstack-infra | 13:06 | |
| *** mwagner has joined #openstack-infra | 13:06 | |
| *** thorst has quit IRC | 13:07 | |
| *** baoli has joined #openstack-infra | 13:07 | |
| *** jyuso1 has quit IRC | 13:08 | |
| *** sorantis has joined #openstack-infra | 13:10 | |
| *** spzala has quit IRC | 13:13 | |
| *** alivigni has joined #openstack-infra | 13:18 | |
| *** kozhukalov_ has quit IRC | 13:18 | |
| *** tlian has joined #openstack-infra | 13:21 | |
| *** wenlock has joined #openstack-infra | 13:25 | |
| openstackgerrit | Alexey Deryugin proposed openstack-infra/project-config: Add missing project openstack/yaql to zuul layout https://review.openstack.org/242469 | 13:27 |
| *** dkranz has joined #openstack-infra | 13:28 | |
| *** Liuqing_ has joined #openstack-infra | 13:28 | |
| *** wenlock has quit IRC | 13:29 | |
| *** Liuqing has quit IRC | 13:31 | |
| *** nadya has joined #openstack-infra | 13:31 | |
| *** edmondsw has joined #openstack-infra | 13:32 | |
| *** thorst has joined #openstack-infra | 13:32 | |
| *** kozhukalov_ has joined #openstack-infra | 13:33 | |
| *** xyang1 has joined #openstack-infra | 13:37 | |
| freerunner | Hey folks! Looks like when https://review.openstack.org/#/c/236719/ was committed, AJaeger forget about openstack/yaql repository. Here the patch, which add yaql back to zuul layout. https://review.openstack.org/242469 . <--- fungi clarkb sdague anteaya | 13:37 |
| *** peterlisak has quit IRC | 13:40 | |
| *** peterlisak has joined #openstack-infra | 13:41 | |
| *** baoli_ has joined #openstack-infra | 13:41 | |
| *** ericksonsantos has quit IRC | 13:41 | |
| anteaya | freerunner: thank you, +2 | 13:42 |
| *** baoli has quit IRC | 13:44 | |
| *** cloudtrainme has joined #openstack-infra | 13:45 | |
| *** EricGonc_ has joined #openstack-infra | 13:46 | |
| *** ericksonsantos has joined #openstack-infra | 13:47 | |
| *** peterlisak has quit IRC | 13:47 | |
| *** peterlisak13 has joined #openstack-infra | 13:47 | |
| *** regXboi has joined #openstack-infra | 13:47 | |
| *** spzala has joined #openstack-infra | 13:48 | |
| *** inteq has quit IRC | 13:49 | |
| *** EricGonczer_ has quit IRC | 13:50 | |
| *** pradk has joined #openstack-infra | 13:50 | |
| *** Shrews has quit IRC | 13:50 | |
| *** salv-orlando has quit IRC | 13:51 | |
| *** salv-orlando has joined #openstack-infra | 13:51 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: multinode and grenade sanity checking. https://review.openstack.org/242489 | 13:54 |
| anteaya | mestery: thank you | 13:54 |
| *** e0ne has quit IRC | 13:55 | |
| *** Shrews has joined #openstack-infra | 13:55 | |
| tristanC | mordred: in all fairness, cauth is still being used and developed, we are looking for supporting more auth backends | 13:56 |
| mordred | tristanC: yah. that it has active developers is one of the nice things | 13:56 |
| *** e0ne has joined #openstack-infra | 13:56 | |
| mordred | ruagair: perhaps also connecting with tristanC about cauth directly is useful/worthwhile? | 13:56 |
| tristanC | mordred: and other improvement like supporting same user's mail from different backend like github and launchpad mail conflicts | 13:57 |
| tristanC | ruagair: mordred: well sure it's opensource and user story are more than welcome fwiw | 13:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing project openstack/yaql to zuul layout https://review.openstack.org/242469 | 13:58 |
| *** rossella_s has quit IRC | 13:59 | |
| *** rossella_s has joined #openstack-infra | 14:00 | |
| *** tdurakov has quit IRC | 14:00 | |
| SamYaple | can I get someone to ack https://review.openstack.org/#/c/241878/ ? adding the devstack-logs publisher for kolla | 14:00 |
| *** tdurakov has joined #openstack-infra | 14:02 | |
| *** tiswanso has joined #openstack-infra | 14:03 | |
| *** kushal has joined #openstack-infra | 14:04 | |
| *** ddieterly has quit IRC | 14:05 | |
| pabelanger | ilyashakhat: ping. Was looking to get some info about the system requirements for stackalytics.com. Specifically, how much RAM are you running | 14:07 |
| *** akshai has joined #openstack-infra | 14:07 | |
| mhu | tristanC, moravec ruagair : cauth might be a bit overkill for what you want to do, indeed. If an application doesn't support your auth protocol of choice, chances are you can simply delegate this to apache with the right mod and use REMOTE_USER | 14:08 |
| mhu | tristanC, mordred ruagair : our use case with software factory was that we didn't want to configure each service we ship for the auth, depending on whether or not they support our auth protocol of choice | 14:09 |
| mhu | tristanC, mordred ruagair so instead, each service will use REMOTE_USER + cauth, cauth holding the auth config for every service in one place | 14:10 |
| * anteaya is traveling the rest of the day | 14:11 | |
| mhu | tristanC, mordred ruagair which is convenient when people deploying Software Factory want to choose the auth protocol(s) they want to offer | 14:12 |
| *** amitgandhinz has joined #openstack-infra | 14:13 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 14:13 |
| *** Piet has quit IRC | 14:14 | |
| mordred | mhu: indeed. whereas pluggable backend auth strategies are not very important for infra itself, as there is one and only one auth | 14:15 |
| *** sdake has joined #openstack-infra | 14:16 | |
| *** hdd has quit IRC | 14:17 | |
| daemontool | anteaya, ty | 14:17 |
| *** rlandy has joined #openstack-infra | 14:17 | |
| *** mriedem has joined #openstack-infra | 14:18 | |
| daemontool | yolanda, or anyone can add me as admin to thie group https://review.openstack.org/#/admin/groups/1172,members as per: https://review.openstack.org/#/c/242025/ | 14:18 |
| *** salv-orlando has quit IRC | 14:22 | |
| *** salv-orlando has joined #openstack-infra | 14:26 | |
| *** placko has quit IRC | 14:30 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/project-config: Add a job around using tripleo with docker https://review.openstack.org/238475 | 14:32 |
| *** jsavak has joined #openstack-infra | 14:32 | |
| *** Sree has quit IRC | 14:33 | |
| *** otter768 has joined #openstack-infra | 14:33 | |
| *** nadya has quit IRC | 14:33 | |
| *** ddieterly has joined #openstack-infra | 14:36 | |
| *** btully has joined #openstack-infra | 14:36 | |
| *** bswartz has joined #openstack-infra | 14:36 | |
| *** tdurakov has quit IRC | 14:36 | |
| *** yfried has quit IRC | 14:36 | |
| *** jsavak has quit IRC | 14:37 | |
| *** otter768 has quit IRC | 14:38 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/project-config: Add a job around using tripleo with docker https://review.openstack.org/238475 | 14:39 |
| yolanda | hi daemontool, done | 14:39 |
| matrohon | hi | 14:40 |
| matrohon | can anyone help me to understand why some patches don't change the bug status in launchpad? | 14:41 |
| *** zz_gondoi is now known as gondoi | 14:41 | |
| matrohon | this is the patch : https://review.openstack.org/#/c/242437/ | 14:41 |
| *** jsavak has joined #openstack-infra | 14:42 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/gear: inline helps for interactive commands https://review.openstack.org/66591 | 14:44 |
| *** achanda has joined #openstack-infra | 14:46 | |
| *** achanda has quit IRC | 14:51 | |
| *** alaski is now known as lascii | 14:52 | |
| ddieterly | can we get someone to review https://review.openstack.org/#/c/240600/ | 14:53 |
| *** wenlock has joined #openstack-infra | 14:53 | |
| *** Sree has joined #openstack-infra | 14:53 | |
| daemontool | yolanda, thank you | 14:54 |
| ddieterly | alternatively, explain the process that we need to go thru to get patches to openstack-infra/project-config reviewed | 14:54 |
| derekh | Hi, the tripleo periodic jobs that are now running fail because ZUUL_BRANCH isn't set (I think), if anybody has a chance can you take a look at this patch that I think will fix it https://review.openstack.org/#/c/242449/ | 14:57 |
| *** burgerk has joined #openstack-infra | 14:57 | |
| *** wenlock has quit IRC | 14:57 | |
| *** Sree has quit IRC | 14:58 | |
| fungi | matrohon: lack of sufficient permissions. you need to add the "OpenStack Infra (hudson-openstack)" user to https://launchpad.net/~bgpvpn-bugs/+members | 14:59 |
| *** larainema has joined #openstack-infra | 14:59 | |
| fungi | only bug supervisors can reassign a bug | 14:59 |
| matrohon | fungi : done, thanks | 15:00 |
| *** Piet has joined #openstack-infra | 15:01 | |
| *** wenlock has joined #openstack-infra | 15:03 | |
| *** tdurakov has joined #openstack-infra | 15:03 | |
| matrohon | fungi : do I need to wait so that the hudson-openstack user in the bug-supervisor team is able to reassign bug? | 15:08 |
| matrohon | fungi : I just updated https://review.openstack.org/#/c/242437/ to test the change | 15:08 |
| *** skolekonov has quit IRC | 15:10 | |
| *** amitgandhinz has quit IRC | 15:15 | |
| *** Liuqing_ has quit IRC | 15:18 | |
| *** amotoki has quit IRC | 15:19 | |
| *** tongli has joined #openstack-infra | 15:20 | |
| *** mtanino has joined #openstack-infra | 15:21 | |
| *** nadya has joined #openstack-infra | 15:22 | |
| *** wenlock_ has joined #openstack-infra | 15:22 | |
| *** wenlock has quit IRC | 15:24 | |
| fungi | matrohon: it only leaves a comment in the bug for patchset #1 (so as to avoid spamming the bug on changes with lots of patchsets) | 15:25 |
| matrohon | fungi : ok make sense | 15:25 |
| *** wenlock has joined #openstack-infra | 15:26 | |
| *** tjones has joined #openstack-infra | 15:27 | |
| ilyashakhat | pabelanger: ping | 15:28 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: DO NOT MERGE -- CI performance debugging https://review.openstack.org/242542 | 15:29 |
| *** sfinucan has joined #openstack-infra | 15:31 | |
| *** sileht has quit IRC | 15:32 | |
| *** dansmith is now known as superdan | 15:33 | |
| *** tonytan4ever has joined #openstack-infra | 15:37 | |
| *** ajmiller has joined #openstack-infra | 15:39 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Correct ACLs for Neutron sub-projects https://review.openstack.org/242196 | 15:44 |
| *** e0ne has quit IRC | 15:46 | |
| *** FallenPegasus has joined #openstack-infra | 15:48 | |
| *** ddieterl_ has joined #openstack-infra | 15:49 | |
| *** cloudtrainme has quit IRC | 15:51 | |
| *** sridhar_ram has joined #openstack-infra | 15:52 | |
| *** Liuqing has joined #openstack-infra | 15:52 | |
| *** ayoung has joined #openstack-infra | 15:53 | |
| *** ddieterly has quit IRC | 15:53 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Update priorities for the Mitaka cycle https://review.openstack.org/241331 | 15:53 |
| *** Liuqing has quit IRC | 15:55 | |
| *** Sree has joined #openstack-infra | 15:55 | |
| *** vilobhmm11 has joined #openstack-infra | 15:57 | |
| fungi | sdague: the problem with DISTRO_NAME in {} on 164927 was that the builder was being used in a template so needed {{DISTRO_NAME}} to keep it from looking like a replacement parameter | 15:58 |
| greghaynes | Oops | 15:59 |
| *** samuelBartel has joined #openstack-infra | 15:59 | |
| *** Sree has quit IRC | 16:01 | |
| *** amitgandhinz has joined #openstack-infra | 16:02 | |
| *** jsavak has quit IRC | 16:02 | |
| *** sdake_ has joined #openstack-infra | 16:03 | |
| matrohon | fungi : actually what I don't understand is why some changes update launchpad correctly (242143), and others don't (242437) | 16:03 |
| *** freyes has quit IRC | 16:04 | |
| *** jsavak has joined #openstack-infra | 16:04 | |
| matrohon | fungi, I assume that it's because in 242143, the Closes-bug statement was correct since the first patch | 16:04 |
| fungi | matrohon: because the script makes a set of api calls to leave a comment, adjust the bug status and reassign it. of those, reassignment at least requires being a bug supervisor, so if the bug isn't already assigned to the correct user then the other api calls are also aborted | 16:04 |
| *** freyes has joined #openstack-infra | 16:05 | |
| *** woodster_ has joined #openstack-infra | 16:05 | |
| greghaynes | fungi: not sure if you saw my new rev / comment on https://review.openstack.org/#/c/165240/ | 16:06 |
| fungi | matrohon: it's not a particularly robust script, and basically dies in flames the moment it encounters a launchpad api error | 16:06 |
| *** sdake has quit IRC | 16:06 | |
| matrohon | fungi : ok, hope it will works fine now, thanks | 16:07 |
| fungi | greghaynes: while my puppet-fu too is weak, i believe that's simply adding the authorized key so that the jenkins master can connect and run jobs on it | 16:07 |
| greghaynes | ah | 16:07 |
| fungi | but not setting up keys to allow that slave to authenticate to the mirror servers | 16:08 |
| greghaynes | so we need another key | 16:08 |
| greghaynes | Is there somewhere else we do that? | 16:08 |
| *** jaypipes is now known as leakypipes | 16:09 | |
| *** fedexo has joined #openstack-infra | 16:09 | |
| fungi | greghaynes: on proposal.slave.o.o we install an ssh keypair to be able to authenticate to gerrit so it can push patches | 16:10 |
| greghaynes | fungi: awesome, ty | 16:10 |
| fungi | that's the most similar configuration we have that comes to mind anyway | 16:11 |
| *** yamamoto_ has quit IRC | 16:12 | |
| *** hemnafk is now known as hemna | 16:12 | |
| *** ifarkas has quit IRC | 16:14 | |
| *** achanda has joined #openstack-infra | 16:20 | |
| *** hashar has quit IRC | 16:20 | |
| *** tdurakov has quit IRC | 16:22 | |
| greghaynes | fungi: Looks like that works by just referring to a keypair in hiera, is that something infra rooters generate by hand and instll? | 16:23 |
| greghaynes | into the hiera db | 16:23 |
| *** sorantis has quit IRC | 16:25 | |
| *** derekh has quit IRC | 16:26 | |
| *** freyes has quit IRC | 16:26 | |
| *** mriedem is now known as mriedem_away | 16:27 | |
| *** sdake_ is now known as sdake | 16:28 | |
| *** zxiiro has quit IRC | 16:29 | |
| *** ihrachys has quit IRC | 16:29 | |
| *** luqas has quit IRC | 16:29 | |
| *** zxiiro has joined #openstack-infra | 16:30 | |
| fungi | greghaynes: yep. we just use ssh-keygen to create the rsa keypair and then stash the two values in hiera | 16:30 |
| greghaynes | fungi: makes sense, easy enough to do | 16:30 |
| fungi | also it would be good if we could also puppet the known_hosts entries for the mirror servers onto the mirror slaves | 16:30 |
| *** hdd has joined #openstack-infra | 16:30 | |
| *** sabeen has joined #openstack-infra | 16:31 | |
| fungi | so that we don't have to go manually accept all of them by hand | 16:31 |
| greghaynes | ah, yea. I'll loo at that | 16:31 |
| greghaynes | look, even | 16:31 |
| *** cloudtrainme has joined #openstack-infra | 16:31 | |
| fungi | codesearch.o.o for known_hosts will probably turn up some examples of where we already do that | 16:32 |
| mordred | codesearch is the best | 16:32 |
| *** bpokorny has joined #openstack-infra | 16:33 | |
| *** otter768 has joined #openstack-infra | 16:34 | |
| fungi | thanks again, taron! | 16:35 |
| *** vilobhmm11 has quit IRC | 16:35 | |
| *** jaosorior has quit IRC | 16:35 | |
| *** vilobhmm11 has joined #openstack-infra | 16:36 | |
| fungi | did we ever announce that service officially? | 16:36 |
| *** cloudtrainme has quit IRC | 16:36 | |
| mordred | unsure? | 16:37 |
| fungi | i think it wrapped up during the summit so maybe we haven't made a splash with it yet | 16:37 |
| *** tnarg has joined #openstack-infra | 16:37 | |
| jeblair | don't believe so | 16:37 |
| *** HenryG has quit IRC | 16:37 | |
| *** e0ne has joined #openstack-infra | 16:38 | |
| fungi | i'll let taron have the honor of posting the availability announcement to the -infra and -dev mailing lists if interested | 16:38 |
| *** sridhar_ram has quit IRC | 16:38 | |
| jeblair | ++ | 16:39 |
| fungi | happy to proof-read a draft and make suggestions too | 16:39 |
| *** HenryG has joined #openstack-infra | 16:39 | |
| *** otter768 has quit IRC | 16:39 | |
| *** jsavak has quit IRC | 16:39 | |
| *** sridhar_ram has joined #openstack-infra | 16:39 | |
| *** jsavak has joined #openstack-infra | 16:40 | |
| *** matrohon has quit IRC | 16:41 | |
| *** sileht has joined #openstack-infra | 16:41 | |
| greghaynes | fungi: hrm, for the known_hosts thing, we would need to know the host key beforehand | 16:42 |
| fungi | greghaynes: we already do | 16:42 |
| greghaynes | fungi: oh? | 16:42 |
| fungi | those servers exist | 16:42 |
| greghaynes | well then :) | 16:43 |
| *** achanda has quit IRC | 16:45 | |
| pabelanger | pleia2: just need to run my daughter to school, but it looks like the 15GB flavor will work for stackalytics.o.o. Have some free time today to launch it? | 16:46 |
| pabelanger | ilyashakhat: pong | 16:46 |
| pabelanger | ilyashakhat: seen your email | 16:46 |
| fungi | greghaynes: ssh-keyscan mirror.dfw.openstack.org (et cetera) | 16:47 |
| greghaynes | fungi: Yea, the existing solution pulls it out of hiera (I am guessing that is better in case we switch the host to a new place?) | 16:47 |
| *** freyes has joined #openstack-infra | 16:48 | |
| fungi | greghaynes: i'm torn on things like host key hashes and public keys going into our secret hiera. up to you for now. when we have a public/private hiera split of course it makes a lot of sense to keep data like that in yaml in git | 16:48 |
| *** dangers_away is now known as dangers | 16:48 | |
| greghaynes | Yep, usually there is a some kind of public environment for these kinds of tools... | 16:49 |
| fungi | in an ideal world, the secret hiera would just be private keys, passwords, et cetera | 16:49 |
| *** vilobhmm11 has quit IRC | 16:49 | |
| *** vilobhmm11 has joined #openstack-infra | 16:49 | |
| *** nadya has quit IRC | 16:50 | |
| sdague | fungi: ok, gotcha. Well hopefully you and greghaynes have it all sorted now | 16:51 |
| fungi | sdague: yeah, it's very close now--thanks for hacking on it! | 16:51 |
| greghaynes | Yep, should be close | 16:52 |
| *** samuelBartel has quit IRC | 16:52 | |
| *** gyee has joined #openstack-infra | 16:52 | |
| greghaynes | This thought had never occured to me until you pointed it out yesterday, but now that we have the transitive deps defined in requirements there is actually 0 reason for pypi-mirror to exist | 16:53 |
| *** vilobhmm11 has quit IRC | 16:54 | |
| *** sridhar_ram has quit IRC | 16:54 | |
| *** tjones has quit IRC | 16:54 | |
| *** achanda has joined #openstack-infra | 16:56 | |
| greghaynes | So it might be time to let that project die... again. | 16:56 |
| *** sridhar_ram has joined #openstack-infra | 16:56 | |
| *** tjones has joined #openstack-infra | 16:56 | |
| *** Sree has joined #openstack-infra | 16:57 | |
| *** ilyashakhat has quit IRC | 16:57 | |
| sdague | gah, rax images without libxml2 in them .... | 16:57 |
| *** fhubik has quit IRC | 16:57 | |
| fungi | greghaynes: well, it's not just that we have resolved lists of transitive deps, but also that pip's wheel integration/caching has gotten better | 16:57 |
| fungi | if this was still circa pip 1.5/1.6 i don't think things would be a ssimple | 16:57 |
| fungi | sdague: in all regions or just one that you can find? i'll delete the current images to roll back to working ones | 16:58 |
| greghaynes | fungi: hah, yea | 16:58 |
| sdague | fungi: well, it's in one, and it's triggered on a subnode | 16:58 |
| sdague | but it made me realize we needed to fix devstack anyway | 16:59 |
| *** sputnik13 has joined #openstack-infra | 16:59 | |
| sdague | https://review.openstack.org/#/c/242559/ | 16:59 |
| *** sputnik13 has quit IRC | 16:59 | |
| fungi | sdague: oh, yeah subnode means multi-node means devstack means we don't normally preinstall devstack dependencies on those images | 16:59 |
| sdague | because we're installing python-lxml, then uninstalling it all the time | 17:00 |
| fungi | so if libxml2 was installed on them previously, i think that was an oversight | 17:00 |
| sdague | right, well, we need it | 17:00 |
| sdague | n-cpu needs it because of the libvirt driver | 17:00 |
| sdague | but, anyway, we can fix it in devstack. upper constraints means that python-lxml is no longer useful to install as a package | 17:01 |
| fungi | oh, sure, i don't question devstack needing it, just that devstack should be installing it | 17:01 |
| sdague | in liberty and beyond | 17:01 |
| *** Sree has quit IRC | 17:01 | |
| sdague | yep, I'm fine with that | 17:01 |
| sdague | it's a little unfortunate, as it killed - https://review.openstack.org/#/c/242489/ | 17:01 |
| sdague | or more importantly, died early enough that we never ran it correctly | 17:01 |
| mordred | sdague: I keep meaning to go back to the pach of "never install python-* from distro packages ever" in devstack- are we still installing things from packages? | 17:02 |
| sdague | mordred: a few things | 17:02 |
| sdague | https://review.openstack.org/#/c/242559/ cleans up most of them | 17:02 |
| fungi | so if devstack was relying on libxml2 existing preinstalled on our images, just saying that's a devstack bug and i don't know why we had libxml2 preinstalled nor what caused it to stop being preinstalled, though i can try to look into it | 17:02 |
| sdague | fungi: yep, I'm completely ok with that | 17:02 |
| sdague | the patch is pushed that will fix it | 17:02 |
| *** jsavak has quit IRC | 17:02 | |
| fungi | i guess it also needs backporting to devstack stable branches | 17:03 |
| sdague | I don't need any investigation on this one, it was just a "gah" | 17:03 |
| sdague | I backported to liberty | 17:03 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 17:03 |
| EmilienM | I have a job which is hanging for a while now: https://jenkins07.openstack.org/job/gate-puppet-openstack_extras-puppet-unit-latest-dsvm-centos7/8/console | 17:03 |
| mordred | sdague: cool | 17:03 |
| EmilienM | can someone kill it ? | 17:03 |
| sdague | pre liberty the packages should kick it up | 17:03 |
| sdague | because we don't have upper constraints | 17:03 |
| fungi | ahh, right-o | 17:04 |
| mordred | sdague: wow: files/debs/nova:python-m2crypto | 17:04 |
| sdague | yeh | 17:04 |
| mordred | we haven't depended on m2crypto in like 3 years | 17:04 |
| *** sputnik13 has joined #openstack-infra | 17:04 | |
| *** wenlock__ has joined #openstack-infra | 17:04 | |
| *** unicell1 has quit IRC | 17:04 | |
| sdague | well, it's removed now :) | 17:04 |
| *** jlanoux has quit IRC | 17:04 | |
| *** Hal1 has quit IRC | 17:05 | |
| EmilienM | actually don't kill the job, but Zuul-Cloner looks very slow to clone our modules | 17:05 |
| fungi | yeah, i thought m2crypto was abandoned upstream and an acknowledged security risk. amusing it hung around in the list that long | 17:05 |
| *** unicell has joined #openstack-infra | 17:05 | |
| *** eil397 has joined #openstack-infra | 17:05 | |
| mordred | sdague: neat! | 17:05 |
| mordred | sdague: oh - I didn't follow your link before :) | 17:05 |
| EmilienM | fungi: have you noticed any issue with zuul-cloner recently ? See https://jenkins07.openstack.org/job/gate-puppet-openstack_extras-puppet-unit-latest-dsvm-centos7/8/console how it's so slow | 17:05 |
| fungi | EmilienM: it's possible we have an overloaded git server or two | 17:05 |
| sdague | mordred: ... squirel! | 17:06 |
| fungi | or an overloaded zuul merger perhaps | 17:06 |
| mordred | sdague: is there an equv patch for rpms? | 17:06 |
| EmilienM | fungi: ack - just wanted to make sure you're aware about that | 17:06 |
| sdague | mordred: there is not yet, because I didn't want to go through all the analysis on what -dev things had to get added | 17:06 |
| fungi | EmilienM: yep, thanks... i'll start going through cacti graphs to see if i spot an obvious problem | 17:06 |
| sdague | I will totally +2 one if you make it | 17:06 |
| EmilienM | fungi: I'm looking now | 17:07 |
| *** wenlock__ has quit IRC | 17:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Let os-client-config handle session creation https://review.openstack.org/240734 | 17:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove designate support https://review.openstack.org/242095 | 17:08 |
| mordred | fungi, Shrews, jeblair: ^^ 242095 is up for discussion and I could go either way on it - designate support literally does not work at all currently | 17:10 |
| fungi | sdague: mordred: it's possible the list is mostly similar to what's in the bindep fallback list in project-config | 17:10 |
| mordred | the unit tests work | 17:10 |
| mordred | but it's a great example of the fact that the new rule needs to be "new API functions must come with functional tests" | 17:10 |
| sdague | fungi: it is | 17:10 |
| *** mfalatic has quit IRC | 17:11 | |
| sdague | I mostly wanted to get enough in to make these tests work, and if anyone else wants to optimize the rpm space, happy hacking | 17:11 |
| mordred | but I could also see sticking to the letter of the law in terms of api contracts and still considering removing those bad, even though it is impossible for anyone to have ever used them | 17:11 |
| clarkb | mordred does designate even have >1 cloud deployment? | 17:12 |
| clarkb | couldnt users use the designate client? | 17:12 |
| mordred | clarkb: no. the designate client constructor was the thing that was broken | 17:13 |
| EmilienM | fungi: i don't see issue on git servers | 17:13 |
| EmilienM | fungi: established tcp connections were high, but not a big deal | 17:13 |
| clarkb | more specificallt I see shade as useful when multiple clouds need dealing with | 17:13 |
| mordred | clarkb: and yes, vexxhost has it deployed. however, they have v1 deployed, and we _also_ don't have a design for how to deal with the v1/v2 conceptual breakage | 17:13 |
| fungi | EmilienM: yep, that's common | 17:13 |
| mordred | clarkb: yah | 17:13 |
| mordred | so - there are 2 problems a) it doesn't work b) we only support v2 in the current code-base, and there are zero clouds with v2 deployed | 17:14 |
| mordred | with "support" being "you can list zones" | 17:14 |
| greghaynes | fungi: mordred I had a wierd thought - when we start doing ansible to run puppet everywhere... any reason to not have an ansible play to copy the ansible node's known hosts' file to everyone since it will by definition know who all the hosts are | 17:15 |
| fungi | EmilienM: your example run used zm01 and it looks not at all loaded either | 17:15 |
| *** maiteb has joined #openstack-infra | 17:15 | |
| mordred | greghaynes: nope. no reason at all | 17:15 |
| mordred | greghaynes: and we already do ansible to run puppet everywhere | 17:15 |
| greghaynes | orly | 17:16 |
| mordred | greghaynes: so we could do it now if we desired | 17:16 |
| EmilienM | fungi: I don't find any issue in our graphs, I continue to dig | 17:16 |
| greghaynes | I think that would remove all this known_hosts wierdness | 17:16 |
| mordred | greghaynes: yah - the ongoing work is to move from running puppet agent to running puppet apply | 17:16 |
| fungi | greghaynes: yeah, a global /etc/ssh/ssh_known_hosts file is useful and simplifies lots of stuff | 17:16 |
| fungi | i used to do that in other large operations where i worked | 17:16 |
| greghaynes | sweet | 17:16 |
| *** arxcruz has quit IRC | 17:16 | |
| fungi | keeping it in git is even more useful, because you can then point your sysadmins(/users too in some cases) to a list they can check against or even just reuse | 17:17 |
| Shrews | mordred: yeah, i'd like to see it removed if it doesn't work. and ++ to requiring functional tests for new things | 17:18 |
| fungi | of course, publishing sshfp records could be even _more_ useful, but only in some future utopia where you have dnssec working everywhere | 17:18 |
| greghaynes | fungi: Yea, So maybe we copy the ansible node's in to git, then use that | 17:18 |
| *** sergmelikyan has joined #openstack-infra | 17:18 | |
| fungi | greghaynes: i would fully support that but for one problem--the list you're talking about is probably hashed. we want the normal ssh-keyscan output instead | 17:19 |
| Shrews | mordred: i think we should give the author a heads-up notice about it, though | 17:19 |
| mordred | rcarrillocruz: ^^ | 17:19 |
| fungi | greghaynes: however, a quick loop over our servers with ssh-keyscan (passing through a grep -v ^# to filter out comment lines) would be pretty quick to build what we need though | 17:19 |
| greghaynes | fungi: aye. Welp, sounds like a script to generate a good source of truth, put that in git, then an ansible or puppet module to convert that to the format we want | 17:19 |
| greghaynes | :) | 17:20 |
| mordred | I have a questoin about server creation flow with that | 17:20 |
| mordred | currently we use launch_node.py | 17:20 |
| mordred | which is manual | 17:20 |
| mordred | which means that adding the host to the file after launching and submitting a change to git is a step the rooter who is launching can perform | 17:20 |
| mordred | however, once we have ansible-launch-node in place | 17:20 |
| mordred | we have an issue that it'll be an ansible robot creating servers | 17:21 |
| mordred | so how do we get keys back into the git source? | 17:21 |
| *** e0ne has quit IRC | 17:21 | |
| greghaynes | we can have a bot propose the change on launch_node | 17:21 |
| mordred | we could - but in this case when I say "bot" I mean "ansible cron job on puppetmaster.o.o" | 17:21 |
| mordred | which would mean we'd have puppetmaster submitting patches to gerrit | 17:22 |
| mordred | which I suppose is fine | 17:22 |
| fungi | launch_node.py doesn't add the host key to ~root/.ssh/known_hosts (it's a manual step right now anyway), and only prevents subsequent puppet runs until such time as the global known_hosts in git is updated | 17:22 |
| mordred | right | 17:22 |
| mordred | but ansible-launch-node will | 17:22 |
| fungi | ahh, yeah that does then get trickier | 17:22 |
| greghaynes | Yea, I think its not actually a technical issue, its more of a question of how we trust the data and how much auto approval we want | 17:22 |
| nibalizer | why is having a hashed known_hosts a problem? | 17:22 |
| fungi | nibalizer: makes it harder to match visually | 17:22 |
| mordred | right. well, we can't _actually_ trust the data at all until my patch to nova is written, approved, landed and deployed | 17:22 |
| mordred | but, it's as good as we've got currently | 17:23 |
| fungi | nibalizer: the hashed version obscures the hostname | 17:23 |
| greghaynes | So my answer would be that it is going to be manual until the trusted path problem goes away, which is unrelated to the automation tooling | 17:23 |
| fungi | and for reasons that are irrelevant to our public-facing infrastructure (we don't care if people get their hands on a list of our servers' dns names) | 17:23 |
| nibalizer | we could turn off hashing in ~/.ssh/config | 17:24 |
| greghaynes | nibalizer: I would think that is something we want if were going to be making those files global | 17:24 |
| nibalizer | we'd have to start from zero with a new known_hosts though | 17:24 |
| fungi | yep, which is ~= to the work required to loop ssh-keyscan over that list | 17:24 |
| nibalizer | fungi: something about running ssh-keyscan in a loop doesn't sit well with me, seems like we're basically giving up on host verification at that point, and depending only on dns | 17:26 |
| fungi | nibalizer: huh? we already tofu these servers | 17:26 |
| nibalizer | assuming its in a cronjob, i mean | 17:26 |
| *** krtaylor has quit IRC | 17:26 | |
| greghaynes | That would be a one time | 17:26 |
| nibalizer | ok | 17:26 |
| fungi | oh, no i meant once to populate the initial list | 17:26 |
| greghaynes | and we can also verify it matches the existing known_hosts | 17:26 |
| nibalizer | ah okay im on board then | 17:27 |
| fungi | perhaps with some effort (there is an ssh-keygen option to hash a known hosts file we could use to validate the initial list) | 17:27 |
| *** Ala has quit IRC | 17:28 | |
| *** vponomaryov has left #openstack-infra | 17:28 | |
| pleia2 | pabelanger: yep, top of my list after I clean up the fantastic mess my coffee make made \o/ | 17:29 |
| *** peterlisak13 has quit IRC | 17:29 | |
| fungi | also if we ever get to the point of having an openstack dns api for openstack.org (or run our own master nameserver) then whatever automation we have to propose entries to the global known_hosts file on new server creation can also propose sshfp additions to our zone file | 17:31 |
| greghaynes | ooo | 17:31 |
| *** tdurakov has joined #openstack-infra | 17:31 | |
| *** trown is now known as trown|lunch | 17:31 | |
| pleia2 | s/make/maker | 17:32 |
| *** sputnik13 has quit IRC | 17:32 | |
| *** sputnik13 has joined #openstack-infra | 17:32 | |
| fungi | clarkb: which one public cloud provider were you thinking of that has designate access for tenants? | 17:32 |
| clarkb | fungi: HPCloud | 17:32 |
| mordred | fungi: vexxhost also runs it | 17:32 |
| clarkb | its not going to exist for long but it does exist now | 17:32 |
| fungi | clarkb: got it. thought maybe you knew of one which wasn't about to go away | 17:33 |
| mordred | fungi: I can check with mohammed and see if he's going to upgrade to v2 any time soon | 17:33 |
| fungi | so with vexxhost, at least there will still be one known designate instance out there | 17:33 |
| mordred | fungi: if he is, I'd LOVE to suggest that we move our dns there | 17:33 |
| mordred | but I do not think we should jump on v1 - v2 supports importing zone files directly | 17:34 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: multinode and grenade sanity checking. https://review.openstack.org/242489 | 17:35 |
| *** tdurakov has quit IRC | 17:37 | |
| *** jordanP has quit IRC | 17:37 | |
| clarkb | isn't v2 brand new and stillin development though? | 17:37 |
| mordred | dunno | 17:38 |
| fungi | yeah, i find that the battle-tested bind zone file format is easy to parse and update with scripts and also by hand | 17:38 |
| mordred | Kiall: ^^ ? | 17:38 |
| mordred | fungi: you know - we could just run dns servers and use ansible launch-node to update them as part of the process | 17:38 |
| fungi | mordred: yep, i even alluded to that earlier. but having our zonefiles in git might still be nice | 17:39 |
| mordred | fungi: indeed | 17:39 |
| *** Swami has joined #openstack-infra | 17:39 | |
| *** krtaylor has joined #openstack-infra | 17:40 | |
| pleia2 | mordred: did you create a statalytics.o.o once? from 2014-06-08T17:08:20Z | 17:40 |
| mordred | fungi: although again that would give us a chicken-and-egg in terms of fully automated launch-node | 17:40 |
| mordred | pleia2: yes | 17:40 |
| mordred | pleia2: please feel free to delete/re-create | 17:40 |
| pleia2 | mordred: thanks, on it :) | 17:40 |
| fungi | mordred: not entirely chicken-and-egg there... puppet would run once during launch, we just wouldn't get continual puppeting until the host key update was approved | 17:41 |
| pleia2 | we need a 15G node for this, so I'm ust going to delete and make a new one | 17:41 |
| *** unicell has quit IRC | 17:42 | |
| fungi | mordred: also we could decree it acceptable to just single-core approve those updates, so whoever approves the server creation change can watch for and approve the known hosts update | 17:42 |
| fungi | and/or dns record addition | 17:42 |
| mordred | right - it's the dns record addition I was more worried about | 17:42 |
| mordred | it's "land this change, watch for a folow up patch, land that patch" | 17:43 |
| *** dzamboni has quit IRC | 17:43 | |
| mordred | which seems clunky when most of the info in the files is generated info ... maybe we could have supplimental/overlay files in git | 17:43 |
| clarkb | anteaya: fungi I am reviewing those project rename changes now | 17:44 |
| clarkb | this is another place I feel the lack of tree tabs | 17:44 |
| fungi | the alternative, i guess is to simply propose those updates into gerrit as record keeping, and have the source of truth be the zone file on the master nameserver and the /etc/ssh/ssh_known_hosts on the launch master | 17:44 |
| clarkb | so the openstack health chnage changes the names of the groups | 17:44 |
| *** mfalatic has joined #openstack-infra | 17:44 | |
| fungi | clarkb: yeah, i ctrl-click all the urls and then go switching through all the resultant tabs in my sea of other tabs | 17:45 |
| greghaynes | clarkb: why dont you have tree tabs? | 17:45 |
| *** nadya has joined #openstack-infra | 17:45 | |
| clarkb | greghaynes: ff 42 broke the plgin | 17:45 |
| clarkb | anteaya: fungi we should add a step to rename the group via gerrits API so we don't end up with two sets of groups | 17:45 |
| fungi | at least pentadactyl lets me switch between relative or arbitrary tabs with simple keystrokes | 17:45 |
| greghaynes | clarkb: oh. I should hold off on upgrading | 17:46 |
| clarkb | fungi: the problem is they have no organization | 17:46 |
| clarkb | with tree tabs all these changes would hang below the etherpad | 17:46 |
| fungi | right, that is my main challenge as well | 17:46 |
| clarkb | so I can group logical sets of tabs together easily | 17:46 |
| fungi | oh, it's smart trees? that's useful | 17:46 |
| clarkb | then when work is done I close etherpad and all leaf tabs go away too | 17:46 |
| fungi | so you don't have to manually group the tabs into trees yourself | 17:46 |
| * fungi didn't know such magic existed | 17:47 | |
| clarkb | no by default opening a new tab from within a tab creates a child relationship | 17:47 |
| clarkb | except its all broken on ff42 so :( | 17:47 |
| clarkb | I will update etherpad with group rename step | 17:47 |
| pabelanger | pleia2: cool, and back | 17:48 |
| clarkb | oh wait its already there I was just blind | 17:48 |
| clarkb | nevermind | 17:48 |
| fungi | clarkb: so you're saying the group rename commands in step 12 are incomplete? | 17:48 |
| fungi | ahh, yep. cool | 17:48 |
| fungi | anteaya's done great work on this | 17:48 |
| pleia2 | pabelanger: great, just cleaning up our old stackalytics instance, should be getting to new one momentarily | 17:48 |
| *** daniloramalho has joined #openstack-infra | 17:49 | |
| *** daniloramalho is now known as dramalho | 17:50 | |
| pleia2 | <insert monthly complaint about web based dns interface for deleting entries here> | 17:50 |
| greghaynes | Maybe we could add that to the irc bot. | 17:51 |
| fungi | pleia2: that discussion was so 10 minutes ago | 17:51 |
| pleia2 | fungi :) | 17:51 |
| mordred | fungi: that discussion is always | 17:51 |
| jroll | clarkb: this last line implies a job timeout, right? http://logs.openstack.org/60/240260/4/check/gate-tempest-dsvm-ironic-agent_ssh/35548f1/logs/syslog.txt.gz#_Nov__6_15_46_10 | 17:51 |
| fungi | we've met our quota for the month | 17:51 |
| *** pradk has quit IRC | 17:52 | |
| jroll | clarkb: that seems to be the only relevant difference between success and fail for bug 1393099 | 17:52 |
| openstack | bug 1393099 in OpenStack-Gate "test_baremetal_server_ops fails with timeout waiting for provision_state to change" [Undecided,Confirmed] https://launchpad.net/bugs/1393099 | 17:52 |
| clarkb | jroll: not necessarily, the clenaup could've happened before the timeout | 17:52 |
| clarkb | jroll: thats a generic end of job cleanup runs regardless of timeout or not | 17:52 |
| jroll | clarkb: oh god I can't read, sorry | 17:53 |
| fungi | clarkb: the only addition i can see to the current etherpad is that the networking-bagpipe-l2 -> networking-bagpipe change is now in order and needs to get entries in the various cut-n-paste sections | 17:53 |
| *** sputnik13 has quit IRC | 17:53 | |
| jroll | clarkb: though it seems to happen during a tftp operation, which means we weren't done here yet | 17:53 |
| *** vilobhmm11 has joined #openstack-infra | 17:54 | |
| fungi | clarkb: i'll get those added here shortly and start working on rebasing the 5 changes into a series in preparation for the maintenance in a couple hours | 17:54 |
| clarkb | jroll: if you look at the console log tempest appears to fail cleanly | 17:54 |
| *** jistr has quit IRC | 17:54 | |
| fungi | once you're done reviewing that is | 17:54 |
| clarkb | fungi: I am through all of them but bigpipe | 17:55 |
| clarkb | fungi: they look fine so far | 17:55 |
| clarkb | the one quetion I did have was attci | 17:55 |
| clarkb | are we still atticing? | 17:55 |
| *** fedexo has quit IRC | 17:55 | |
| clarkb | because two akanda repos go to the attic | 17:55 |
| *** vilobhmm11 has left #openstack-infra | 17:55 | |
| *** peterlisak13 has joined #openstack-infra | 17:56 | |
| fungi | clarkb: i would rather not attic but it's already written up at this point and we haven't revisited that topic on the ml post stackforge retirement | 17:56 |
| jroll | clarkb: mmm yeah. my other theory is these dropped dhcp packets, but it eventually succeeds and that also happens on successful jobs http://logs.openstack.org/60/240260/4/check/gate-tempest-dsvm-ironic-agent_ssh/35548f1/logs/syslog.txt.gz#_Nov__6_15_46_06 | 17:56 |
| *** sputnik13 has joined #openstack-infra | 17:56 | |
| jroll | so I think that's a red herring | 17:56 |
| *** cdent has quit IRC | 17:56 | |
| clarkb | fungi: and agreed looks like we just need to update the steps to include bagpipe | 17:57 |
| clarkb | which is not confusing at all ocmpared to bigpipe | 17:57 |
| fungi | heh, i miss old f5s | 17:57 |
| *** Sree has joined #openstack-infra | 17:57 | |
| fungi | my fingers still remember the bigpipe command | 17:58 |
| *** tonytan4ever has quit IRC | 17:58 | |
| fungi | wow, actually looks like bigpipe is still a thing in their cli | 17:58 |
| clarkb | in other news nodepool seems to be struggling a bit (lots of building nodes and not relaly keeping up with demand), I will look at logs there to see what is going on | 17:59 |
| fungi | thanks | 17:59 |
| fungi | having that clear up before we need to push changes through and restore the queue at the end of our maintenance would be stellar | 18:00 |
| *** tonytan4ever has joined #openstack-infra | 18:00 | |
| clarkb | Unknown Error HTTP 500 :) | 18:01 |
| clarkb | the best error | 18:01 |
| mordred | it's my favorite | 18:01 |
| clarkb | thats all in hpcloud | 18:01 |
| fungi | clarkb: floating ip leak | 18:01 |
| fungi | says smart money | 18:01 |
| *** electrofelix has quit IRC | 18:01 | |
| clarkb | ya thats one possibility | 18:01 |
| mordred | s/Unknown Error/Our load balancer has a sad/ | 18:01 |
| clarkb | I am going to see if I can nova show one of them before it goes away | 18:01 |
| *** EricGonc_ has quit IRC | 18:01 | |
| *** Sree has quit IRC | 18:02 | |
| fungi | i'm getting a list of stale unassociated fips | 18:02 |
| fungi | should be able to clean them up in a few minutes if there are any | 18:02 |
| clarkb | nova show shows the VM as active | 18:02 |
| clarkb | so no errors there | 18:02 |
| clarkb | I will dig up my delete fips script | 18:03 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Treat openstack_health normally https://review.openstack.org/242586 | 18:03 |
| mordred | yah. well, that was the symptom of the db timeout thing from before | 18:03 |
| *** peterlisak13 has quit IRC | 18:03 | |
| mordred | create took longer than the load balancer timeout | 18:03 |
| clarkb | venv/bin/neutron floatingip-list | grep -v '10\.0\.' | sed -e '1,3d' -e '$d' | cut -d'|' -f 2 | xargs -n 1 -P 1 venv/bin/neutron floatingip-delete | 18:03 |
| mordred | so the lb threw a 500 at us | 18:03 |
| fungi | clarkb: i already have mine going (well, the discovery pass anyway) | 18:03 |
| mordred | which made us mark it as failed | 18:03 |
| mordred | and then we deleted it | 18:03 |
| fungi | nova floating-ip-list | grep ' - ' | cut -d' ' -f4 | sort > fip-cruft-1 ; sleep 300 ; nova floating-ip-list | grep ' - ' | cut -d' ' -f4 | sort > fip-cruft-2 ; diff -U1000 fip-cruft-{1,2} | grep '^ ' > fip-cruft ; wc -l fip-cruft{,-1,-2} | 18:03 |
| mordred | which put more stress on the already stressed db | 18:03 |
| nibalizer | https://review.openstack.org/#/c/242586/ will fix install_modules.sh once openstack-health is renamed to openstack_health | 18:04 |
| clarkb | heh no module named oslo.utils | 18:04 |
| mordred | create server times are slow - but not as bad as they were | 18:04 |
| mordred | list servers time is at a vvery large spike | 18:05 |
| clarkb | --force-reinstall will hopefully fix my virutalenv | 18:05 |
| clarkb | yup that worked | 18:06 |
| mordred | fungi: you should learn about the neutron floatingip commands | 18:06 |
| fungi | i should | 18:06 |
| clarkb | 338 potentially leaked floating IPs | 18:06 |
| mordred | fungi: they have a better output format selection | 18:06 |
| pabelanger | pleia2: great | 18:06 |
| clarkb | mordred: eh | 18:06 |
| clarkb | mordred: more like completely undocumented and terrible | 18:06 |
| fungi | clarkb: is that comparing passes spaced some time apart? | 18:06 |
| clarkb | mordred: and to get feature parity with novaclient you have to RTFS | 18:06 |
| clarkb | fungi: no that was a single point in time | 18:06 |
| mordred | clarkb: what??? | 18:07 |
| fungi | i've found that they can show up not associated during provisioning | 18:07 |
| clarkb | mordred: neutronclient allows you to select columns | 18:07 |
| mordred | yah | 18:07 |
| fungi | and also sometimes briefly during cleanup | 18:07 |
| clarkb | mordred: its completely undocumented | 18:07 |
| *** dramalho has quit IRC | 18:07 | |
| mordred | it's not | 18:07 |
| mordred | it's totally documented | 18:07 |
| clarkb | mordred: the columns are not | 18:07 |
| fungi | my diff has 333 entries | 18:07 |
| clarkb | mordred: so to figure out what you can actually display you have to rtfs | 18:07 |
| fungi | spaced 5 minutes apart | 18:07 |
| clarkb | mordred: and by default you get less info than what novaclient shows | 18:07 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 18:08 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add unit tests for subunit2sql.shell.main https://review.openstack.org/242587 | 18:08 |
| clarkb | I spent a good hour figuring this out one day and probably drank too much after | 18:08 |
| fungi | heh | 18:08 |
| greghaynes | mordred: clarkb somewhat relavent - on https://review.openstack.org/#/c/235165/1/nodepool/provider_manager.py my thinking was that in a case like this if we did FIP creation first it would be a lot less work on a failue | 18:08 |
| clarkb | "why can't I just .... OH! that would've been helpful to actually tell someone" | 18:08 |
| greghaynes | The shade deal does help, but still it would be good to do the cheap work first and fail fast | 18:08 |
| mordred | clarkb: ok. I stand corrected. I have no idea why I knew about it | 18:09 |
| mordred | OH | 18:09 |
| greghaynes | oh, actually, we cant because the shade thing attaches | 18:09 |
| mordred | it's documented in the openstackclient help | 18:09 |
| *** mriedem_away is now known as mriedem | 18:09 | |
| mordred | and they both use cliff | 18:09 |
| *** nadya has quit IRC | 18:09 | |
| *** bharathm has joined #openstack-infra | 18:10 | |
| *** abregman has joined #openstack-infra | 18:11 | |
| fungi | wow, pre-purchasing internet access on a cruise ship is only roughly as usurious as for an international flight | 18:11 |
| mordred | clarkb, fungi: http://paste.openstack.org/show/478208/ | 18:11 |
| clarkb | yes its cliffy but highly coupled to the api version too | 18:11 |
| *** achanda has quit IRC | 18:12 | |
| fungi | (on a per diem basis anyway) | 18:12 |
| clarkb | so its not really a portable thing regardless | 18:12 |
| mordred | there are the docs, fwiw | 18:12 |
| clarkb | mordred: where? | 18:12 |
| mordred | clarkb: that was the output of openstack ip floating list -h | pastebinit | 18:12 |
| *** unicell has joined #openstack-infra | 18:12 | |
| mordred | I'm not sure why that help doesn't get outputted elsewhere | 18:12 |
| greghaynes | I just got reminded - did we ever follow up and check if the paste.o.o robots.txt had any effect? | 18:12 |
| clarkb | I don't care about openstack | 18:12 |
| mordred | clarkb: it's the same help | 18:12 |
| clarkb | oh I see in your paste | 18:13 |
| mordred | neutron just doesn't display it | 18:13 |
| clarkb | ya those docs are uesless | 18:13 |
| clarkb | the -c is outputed in neutron help floatingip-list | 18:13 |
| clarkb | but no information on what a valid column is is documented | 18:13 |
| clarkb | thats the problem | 18:13 |
| clarkb | its great I can select columns, I have no idea what columns I can select | 18:13 |
| purp | Howdy, folks. Pardon my ignorance, but if I want to get Stackalytics to properly assign @hpe.com email addresses to HP(E), who should I talk to? | 18:14 |
| clarkb | and it is actually dependent on what the api returns to you raw in the api | 18:14 |
| clarkb | which means its not portable | 18:14 |
| fungi | purp: there's a data file in the openstack/stackalytics repo, you just propose an update to it through gerrit | 18:14 |
| fungi | i'll see if i can find it | 18:14 |
| purp | fungi: I've got that. | 18:15 |
| purp | fungi: etc/default_data.json, yes? | 18:15 |
| *** abregman has quit IRC | 18:16 | |
| mordred | yup | 18:16 |
| clarkb | fungi: looking at a diff over time of fips it looks like we aren't really seeing any old ones go active | 18:16 |
| fungi | looks like that's the one | 18:16 |
| clarkb | maybe I am not waiting long enough | 18:16 |
| clarkb | my diff just shows a bunch of new fips on the more recent listing | 18:16 |
| clarkb | none of the old fips go away in the more recent listing | 18:17 |
| *** tonytan_brb has joined #openstack-infra | 18:17 | |
| *** abregman has joined #openstack-infra | 18:17 | |
| *** tonytan_brb has quit IRC | 18:17 | |
| nibalizer | purp: there is a 'companies' blob in there where I think you can add HPE domains to the hp company | 18:17 |
| fungi | clarkb: on my diff i see 5 drop out in the second pass 5 minute slater | 18:18 |
| purp | fungi: thanks! | 18:18 |
| clarkb | I see none drop out but that may be due to different timing | 18:18 |
| clarkb | I will rerun a new listing in a bit to see if that changes | 18:18 |
| purp | nibalizer: I'll go sniffing. Guess we need to decide how we want to handle the split here. | 18:18 |
| *** sputnik13 has quit IRC | 18:18 | |
| clarkb | my diff arrows all point > that way | 18:18 |
| purp | fungi nibalizer: is there someone I should chat with on Stackalytics team who would care how we handle attribution around this? | 18:19 |
| *** niedbalski has quit IRC | 18:19 | |
| fungi | clarkb: looks like 15.126.218.123,139,147,172,183 went active between my passes | 18:19 |
| mordred | greghaynes: so, we could do that, although there is a problem we'd have to solve which is right now we don't have a GREAT answer for "will I need a floating ip in the future if I decide to create a server" | 18:19 |
| mordred | greghaynes: we could grow one of course, since it's just programming | 18:19 |
| nibalizer | purp: I don't know anyone on the stackalytics team, sorry | 18:19 |
| *** tonytan4ever has quit IRC | 18:19 | |
| mordred | greghaynes: but at the moment the quetion is "hey, I've got this server, do I need to gt a floating ip for it?" | 18:20 |
| mordred | purp: ilya doesn't seem to be in channel at the moment | 18:20 |
| *** nadya has joined #openstack-infra | 18:20 | |
| purp | I'll poke in #openstack-stackalytics and see what I get. Thanks fungi nibalizer mordred for guidance! | 18:21 |
| *** e0ne has joined #openstack-infra | 18:21 | |
| *** cdent has joined #openstack-infra | 18:21 | |
| pleia2 | if someone has a couple minutes, I could use a few eyes on the error I'm getting when I try to bring up this server: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /opt/system-config/production/manifests/site.pp:152 on node stackalytics.openstack.org | 18:21 |
| *** tjones has quit IRC | 18:21 | |
| pleia2 | that line looks fine to me, so it could be red herring because puppet | 18:21 |
| pleia2 | my eyes can't see the issue though | 18:21 |
| *** niedbalski has joined #openstack-infra | 18:22 | |
| *** sputnik13 has joined #openstack-infra | 18:22 | |
| fungi | clarkb: oh, it's possible the ones i'm seeing drop off my list were in the midst of being deleted during the first pass | 18:22 |
| fungi | rather than switching from disassociated to associated | 18:23 |
| pabelanger | pleia2: looks like issue getting remote data from puppetmaster.o.o | 18:23 |
| *** dramalho has joined #openstack-infra | 18:23 | |
| fungi | since i'm just filtering out all associated fips i don't see any difference between associated and deleted | 18:23 |
| *** cdent has quit IRC | 18:24 | |
| *** tjones has joined #openstack-infra | 18:25 | |
| *** bardia has joined #openstack-infra | 18:25 | |
| *** salv-orlando has quit IRC | 18:25 | |
| *** salv-orlando has joined #openstack-infra | 18:26 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 18:26 |
| pleia2 | pabelanger: it does manage to get some data earlier in the run (and this is all initiated from the puppetmaster), so I'm not sure what would cause it to fall over here, and know exactly where it's failing | 18:26 |
| *** devvesa has quit IRC | 18:27 | |
| pleia2 | could be one of the changes we recently made, but I wasn't involved with those enough to speculate | 18:28 |
| pleia2 | and I launched codesearch.o.o ok a week ago | 18:28 |
| *** abregman has quit IRC | 18:29 | |
| nibalizer | pleia2: ill look at that | 18:29 |
| clarkb | puppet is trying to run tge empty? method against nilclass and failing. without proper stacltrace hard to know wht | 18:29 |
| *** abregman has joined #openstack-infra | 18:29 | |
| nibalizer | ya that sounds like a catalog compilation | 18:29 |
| * nibalizer runs puppet agent --test on stackalytics | 18:29 | |
| jeblair | formating error in stackalytics.openstack.org.yaml ? | 18:29 |
| jeblair | no indedentation for the private key | 18:30 |
| jeblair | though it's a yaml text block | 18:30 |
| jeblair | stackalytics_ssh_private_key_contents: | | 18:30 |
| jeblair | -----BEGIN RSA PRIVATE KEY----- | 18:30 |
| jeblair | should have a couple spaces at the start of the second and subsequent lines | 18:30 |
| nibalizer | pleia2: did stackalytics.openstack.org make it into dns yet? | 18:31 |
| jeblair | pleia2: did you use hieraedit.py? | 18:31 |
| pleia2 | jeblair: I did not | 18:31 |
| jeblair | nibalizer: that happens last | 18:31 |
| *** ybathia has joined #openstack-infra | 18:31 | |
| jeblair | pleia2: ah, i'd recommend that :) | 18:31 |
| pleia2 | nibalizer: no, the run deletes the server upon failure, I can email you the full output | 18:31 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate: DO NOT MERGE: test config for gate-tempest-dsvm-cells-neutron https://review.openstack.org/235485 | 18:31 |
| pleia2 | jeblair: thanks, I didn't know about it :) | 18:31 |
| nibalizer | sounds like jeblair has a likely hypothesis | 18:31 |
| jeblair | pleia2: i manually updated the file; try again? | 18:32 |
| *** dramalho has quit IRC | 18:32 | |
| pleia2 | jeblair: trying, thank you | 18:33 |
| *** achanda has joined #openstack-infra | 18:34 | |
| *** trown|lunch is now known as trown | 18:35 | |
| *** otter768 has joined #openstack-infra | 18:35 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 18:35 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add unit tests for subunit2sql.shell.main https://review.openstack.org/242587 | 18:35 |
| *** achanda has quit IRC | 18:35 | |
| pleia2 | jeblair: that seemed to be it, thanks for the assist, I'll use hieraedit.py next time | 18:36 |
| jeblair | pleia2: w00t | 18:36 |
| *** ilyashakhat has joined #openstack-infra | 18:37 | |
| *** yamamoto has joined #openstack-infra | 18:38 | |
| *** cloudtrainme has joined #openstack-infra | 18:38 | |
| *** HeOS has quit IRC | 18:39 | |
| *** otter768 has quit IRC | 18:40 | |
| SpamapS | # TODO: make this more ansibley | 18:41 |
| * SpamapS smiles | 18:42 | |
| pabelanger | SpamapS: # NOTE(pabelanger): Not complete until Chef hooks added | 18:43 |
| mordred | ewww | 18:43 |
| SpamapS | crap | 18:43 |
| *** yamamoto has quit IRC | 18:43 | |
| SpamapS | does that mean when the next ansible thing comes out we'll all be hating on that lovely ansibley comment too? | 18:43 |
| mordred | yup | 18:43 |
| SpamapS | ossum | 18:43 |
| *** maiteb has quit IRC | 18:45 | |
| *** wenlock__ has joined #openstack-infra | 18:45 | |
| SpamapS | so... I want to pull in some code I have pushed up to github and test running it as part of a pre/post test hook. Anybody have thoughts on how I might do that? | 18:45 |
| pleia2 | pabelanger: server is up, but we have some debugging to do | 18:45 |
| pabelanger | pleia2: sure | 18:46 |
| *** hdd has quit IRC | 18:47 | |
| *** abregman has quit IRC | 18:47 | |
| eantyshev | jeblair, pleia2: Hello! I have an improvement for hieraedit.py, hope it will make it more usable for human-viewed yaml: https://review.openstack.org/239479 | 18:48 |
| *** achanda has joined #openstack-infra | 18:48 | |
| *** wenlock__ has quit IRC | 18:49 | |
| clarkb | fungi: were you running a delete fip script then? | 18:49 |
| clarkb | fungi: wasn't clear if you were but I can in a few minute sif not | 18:50 |
| fungi | clarkb: i can fire it now. didn't want to step on your toes | 18:50 |
| fungi | it's running | 18:50 |
| *** sabeen has quit IRC | 18:51 | |
| clarkb | thanks | 18:51 |
| *** ildikov has joined #openstack-infra | 18:51 | |
| fungi | looks like 237936 is renaming a group too so i'll add it to the group renames list | 18:52 |
| fungi | appears to be dropping the old networking-bagpipe-l2-release group entirely in favor of neutron-release | 18:53 |
| fungi | so just the core group needs renaming | 18:53 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 18:53 |
| *** pradk has joined #openstack-infra | 18:54 | |
| clarkb | fungi: maybe rename the release group for completeness if though it will be unused? | 18:54 |
| *** cloudtrainme has quit IRC | 18:55 | |
| fungi | clarkb: better to just empty it, switch it to owned by administrator and hide it | 18:55 |
| *** sputnik13 has quit IRC | 18:55 | |
| clarkb | wfm | 18:55 |
| fungi | i'd love to get around to writing a script which can check our list of visible groups against all acls for direct (or indirect through arbitrary levels of inclusion) reference and then clean up any which have been orphaned | 18:56 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add 'testing' node to site.pp https://review.openstack.org/241497 | 18:58 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Remove global puppetmaster setting https://review.openstack.org/241502 | 18:58 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Switch ansible to applyprep and puppetapply https://review.openstack.org/241503 | 18:58 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Role to prep system for puppet apply https://review.openstack.org/241501 | 18:58 |
| *** mrmartin has quit IRC | 18:58 | |
| *** Sree has joined #openstack-infra | 18:58 | |
| *** mrmartin has joined #openstack-infra | 18:59 | |
| *** sputnik13 has joined #openstack-infra | 18:59 | |
| mordred | fungi: ++ | 18:59 |
| clarkb | just don't forget All-Projects | 19:00 |
| fungi | heh | 19:00 |
| fungi | indeed, it counts as an acl too | 19:00 |
| *** cloudtrainme has joined #openstack-infra | 19:01 | |
| *** bpokorny_ has joined #openstack-infra | 19:02 | |
| *** Sree has quit IRC | 19:02 | |
| clarkb | anything else need doing in the next hour for rename prep? | 19:04 |
| fungi | not that i can think of. i've updated the etherpad if you want to check that and am working on the rebases now | 19:04 |
| clarkb | kk | 19:05 |
| *** bpokorny has quit IRC | 19:05 | |
| clarkb | etherpad updates look good | 19:06 |
| clarkb | do we want an explicit step for "disabling" that release group? | 19:06 |
| fungi | meh, i'll just make a note in step 12 | 19:07 |
| *** sputnik13 has quit IRC | 19:07 | |
| clarkb | jeblair: you said ovh was looking better with even less oversubscription, does that mean we can start adding some load to it now? or do we still have api side stuff to wait on? | 19:08 |
| *** e0ne has quit IRC | 19:08 | |
| openstackgerrit | Václav Hejral proposed openstack-infra/elastic-recheck: Add query for nova bug 1512955 https://review.openstack.org/242598 | 19:08 |
| openstack | bug 1512955 in OpenStack Compute (nova) "Race condition in nova/neutron when booting instance with xen driver" [Undecided,In progress] https://launchpad.net/bugs/1512955 - Assigned to huan (huan-xie) | 19:08 |
| jeblair | clarkb: i believe we're now waiting for them to see if they have room for us :/ | 19:08 |
| greghaynes | fungi: for gra1 we dont have a AAAA - do they lack ipv6 or was that an oversight? | 19:09 |
| greghaynes | for pypi.gra1.o.o | 19:09 |
| jeblair | greghaynes: they lack ipv6 | 19:09 |
| greghaynes | ah | 19:09 |
| *** sputnik13 has joined #openstack-infra | 19:10 | |
| fungi | yeah, i would make some joke about the 1990s calling and... well anyway no ipv6 there (yet anyway) | 19:12 |
| *** e0ne has joined #openstack-infra | 19:12 | |
| clarkb | in fairness the routers I had from the ealry 2000s couldn't ipv6 | 19:16 |
| clarkb | and they had uptime of like 6 years | 19:17 |
| fungi | lucky you--must not have been cisco ;) | 19:17 |
| clarkb | they were :) | 19:17 |
| *** tonytan4ever has joined #openstack-infra | 19:17 | |
| clarkb | running CatOS + IOS hybrid mode | 19:17 |
| *** cloudtrainme has quit IRC | 19:18 | |
| fungi | lead-lined housings then? ours used to crash regularly and tac would blame solar flares/cosmic rays | 19:18 |
| clarkb | fungi: I have no idea but they did eventually crash at around the 6 year mark | 19:18 |
| fungi | 6500 series? | 19:18 |
| clarkb | had a pair of them and they died a week apart | 19:19 |
| clarkb | ya | 19:19 |
| clarkb | pretty sure if we went back to look at change control we would've seen they were installed a week apart | 19:19 |
| clarkb | actually they were in an old bunker type building, maybe that protected them from cosmic rays | 19:19 |
| fungi | we had dozens of them, fully-populated, and one or two would pop every month (different ones each time) | 19:19 |
| clarkb | the other hting you have to worry about is fan try dying and having half your cables strung in front of the tray so you couldn't hot swap it | 19:20 |
| fungi | heh, yes. always have to fight the spaghetti | 19:20 |
| clarkb | there was a bug with hot swapping modules iirc | 19:20 |
| fungi | harlowja: i'm curious, in your zk+openjdk debunking, you never addressed the previously quoted alarming comments from the zk source code about only oracle's jdk being trusted to work correctly... stale code comments which never got cleaned up? | 19:21 |
| clarkb | where the chassis would restart power if you didn't seat the new module in the correct amount of time | 19:21 |
| harlowja | fungi that was not zk | 19:21 |
| clarkb | so ya they rebooted a lot but those 2 were special I guess | 19:21 |
| harlowja | that was cassandra... | 19:21 |
| fungi | oh! | 19:21 |
| harlowja | ya | 19:21 |
| harlowja | not zk | 19:21 |
| fungi | harlowja: you are correct--i'm confusing the two threads | 19:21 |
| harlowja | u aren't the only one afaik | 19:21 |
| harlowja | i should have mentioned that in the thread | 19:21 |
| fungi | all java apps look the same to me ;) | 19:21 |
| openstackgerrit | Stephen Lowrie proposed openstack-infra/project-config: Add tempest-dsvm-lxc as experimental https://review.openstack.org/226565 | 19:22 |
| harlowja | but feel free to respond, saying 'and btw the comment blahblah not working was cassanddra' | 19:22 |
| harlowja | it wasn't zk..... | 19:22 |
| harlowja | fungi since u had that misconception (u aren't alone) please do respond if u want :) | 19:22 |
| openstackgerrit | Glauco Oliveira proposed openstack-infra/subunit2sql: Add db migration to align column size to 255 https://review.openstack.org/203251 | 19:22 |
| harlowja | if u want/can | 19:23 |
| greghaynes | I think SpamapS was AFK from that session when that point came up, but he mentioned that juju did a lot of openjdk +zk deploys | 19:23 |
| fungi | harlowja: yep, added to my backlog for today | 19:23 |
| greghaynes | as another data point | 19:23 |
| harlowja | greghaynes yup, feel free to respond to ;) | 19:23 |
| harlowja | i mentioned that greghaynes in the DLM spec | 19:23 |
| harlowja | *the juju usage | 19:23 |
| harlowja | but more 'testimonals' are welcome | 19:23 |
| greghaynes | ah, awesome | 19:23 |
| harlowja | fungi thx | 19:23 |
| *** hdd has joined #openstack-infra | 19:24 | |
| *** achanda has quit IRC | 19:24 | |
| *** mahatic has quit IRC | 19:25 | |
| *** cloudtrainme has joined #openstack-infra | 19:25 | |
| *** sabeen has joined #openstack-infra | 19:26 | |
| *** ilyashakhat has quit IRC | 19:28 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Moving compass-install from stackforge to openstack namespace https://review.openstack.org/241788 | 19:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Rename puppet project 'puppet-openstack_health' https://review.openstack.org/230421 | 19:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Move networking-bigswitch from stackforge to openstack https://review.openstack.org/237753 | 19:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Rename project networking-bagpipe(-l2) https://review.openstack.org/237936 | 19:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Rename projects akanda-* -> astara-* https://review.openstack.org/241328 | 19:28 |
| *** ilyashakhat has joined #openstack-infra | 19:28 | |
| fungi | clarkb: rebased ^ and made sure the only one with a merge conflict is last in the stack for easier re-reviewing | 19:28 |
| *** wenlock__ has joined #openstack-infra | 19:28 | |
| *** abregman has joined #openstack-infra | 19:29 | |
| *** achanda has joined #openstack-infra | 19:29 | |
| fungi | looks like gerrit properly preserved votes for the rest of them | 19:29 |
| *** sabeen2 has joined #openstack-infra | 19:32 | |
| jroll | harlowja: more anecdotal data for you, rackspace's cloud monitoring team uses openjdk7 + zk for locking at scale without much issue | 19:32 |
| harlowja | jroll press reply button and send that ;) | 19:32 |
| *** wenlock__ has quit IRC | 19:33 | |
| fungi | clarkb: you okay with #status notice Gerrit will be offline at 20:00-20:15 UTC today (starting 25 minutes from now) for scheduled project rename maintenance | 19:33 |
| SpamapS | ZK+OpenJDK should be fine | 19:33 |
| *** Piet has quit IRC | 19:33 | |
| fungi | also i'll go resurrect statusbot again | 19:33 |
| SpamapS | We really do need ti dissambiguate Cassandra+OpenJDK from Zookeeper+OpenJDK | 19:34 |
| *** smarcet has quit IRC | 19:34 | |
| *** openstackstatus has joined #openstack-infra | 19:35 | |
| *** ChanServ sets mode: +v openstackstatus | 19:35 | |
| *** sabeen has quit IRC | 19:35 | |
| harlowja | SpamapS fungi says he'll send email about that ;) | 19:36 |
| harlowja | seems people are still having that misconception somehow... | 19:36 |
| * fungi is plenty misconceptiony | 19:36 | |
| *** akshai has quit IRC | 19:37 | |
| *** cloudtrainme has quit IRC | 19:37 | |
| mordred | yah. also, the difference between them is that zk is abstractable so can be replaced for our needs by consul if someone wants | 19:38 |
| clarkb | fungi: ya lgtm | 19:38 |
| mordred | cassandra on the other hand - you can't really abstract that | 19:38 |
| fungi | #status notice Gerrit will be offline at 20:00-20:15 UTC today (starting 20 minutes from now) for scheduled project rename maintenance | 19:39 |
| openstackstatus | fungi: sending notice | 19:39 |
| *** nadya has quit IRC | 19:40 | |
| -openstackstatus- NOTICE: Gerrit will be offline at 20:00-20:15 UTC today (starting 20 minutes from now) for scheduled project rename maintenance | 19:40 | |
| openstackstatus | fungi: finished sending notice | 19:41 |
| pleia2 | I'm around, so lmk if you want me to pitch in with anything | 19:42 |
| SpamapS | harlowja: thanks for your email about openjdk+zk.. now.. time to rewrite it in rust. | 19:43 |
| harlowja | SpamapS np, haha | 19:43 |
| harlowja | i did what i could, there will be crazies that will still say stuff, but meh, those people will always be there | 19:44 |
| harlowja | crazies will be crazy, ha | 19:44 |
| SpamapS | craycrayz be craycray | 19:44 |
| harlowja | :) | 19:44 |
| mordred | craycraybecraycrayyolo | 19:44 |
| harlowja | new meme ftw | 19:44 |
| harlowja | lol | 19:44 |
| mordred | somebody go register craycraybecraycrayyolo.io | 19:44 |
| harlowja | ha | 19:45 |
| mordred | it'll get a pile of VC I'm sure | 19:45 |
| SpamapS | #mahcraycraybaebecraycraytoday | 19:45 |
| harlowja | hahaha | 19:45 |
| SpamapS | I'm going to go flog myself now for that | 19:45 |
| SpamapS | 100 lashes for going millenial | 19:45 |
| clarkb | mordred: cray might want to invest | 19:45 |
| SpamapS | craybecraycray | 19:46 |
| fungi | indeed. cray be craycray | 19:46 |
| fungi | at least these days | 19:46 |
| harlowja | haha | 19:46 |
| *** akshai has joined #openstack-infra | 19:46 | |
| harlowja | logo => craycray person next to old cray cray computer | 19:47 |
| harlowja | *with hipster in there somewhere to | 19:47 |
| *** daemontool has quit IRC | 19:47 | |
| mordred | to be truly hipster, do a DEC logo but add a wink or a beard or someting | 19:48 |
| fungi | we can start a trend of dressing as hipsters and posing next to old supercomputers | 19:48 |
| *** salv-orl_ has joined #openstack-infra | 19:48 | |
| harlowja | fungi +2 | 19:48 |
| fungi | i call dibs on the wang photo | 19:48 |
| harlowja | :-/ | 19:48 |
| mordred | fungi: so many jokes. | 19:48 |
| docaedo | fungi: best line of the day | 19:48 |
| harlowja | i'm ok with giving someone else the wang photo, i don't need to run for congress/president or something | 19:49 |
| *** mestery has quit IRC | 19:50 | |
| harlowja | annnd, i will say no more, ha | 19:50 |
| clarkb | fungi: midcycle at the living computer museum? | 19:50 |
| fungi | clarkb: that would be awesome | 19:50 |
| fungi | we'd take that place over | 19:50 |
| fungi | i think they do rent it out for organized functions | 19:50 |
| clarkb | ooh | 19:50 |
| harlowja | that'd be sweet | 19:50 |
| harlowja | good idea | 19:50 |
| clarkb | http://www.livingcomputermuseum.org/Visit-Us/Plan-an-Event.aspx lets do it | 19:51 |
| *** salv-orlando has quit IRC | 19:51 | |
| clarkb | dibs on the lisa running xenix | 19:51 |
| fungi | yikes | 19:51 |
| * fungi puts the xenix down and slowly backs away | 19:51 | |
| fungi | that os was scart | 19:52 |
| fungi | scary | 19:52 |
| *** peterlisak13 has joined #openstack-infra | 19:52 | |
| harlowja | the history museum in mountain view could work also, they probably rent rooms to | 19:52 |
| fungi | (scart was something else entirely) | 19:52 |
| clarkb | harlowja: but do they let you hack on the pdp11? | 19:52 |
| harlowja | unsure | 19:52 |
| clarkb | on a teletype! | 19:52 |
| *** mrmartin has quit IRC | 19:52 | |
| *** e0ne has quit IRC | 19:54 | |
| pleia2 | harlowja: they do, but only after hours (I've been to a few events there) | 19:54 |
| *** pradk has quit IRC | 19:54 | |
| harlowja | cool | 19:54 |
| *** dangers is now known as dangers_away | 19:54 | |
| pleia2 | not hack on the pdp11, I meant renting rooms ;) | 19:54 |
| harlowja | oh | 19:55 |
| *** ilyashakhat has quit IRC | 19:55 | |
| harlowja | well we can at least take craycray pictures next to a cray cray | 19:55 |
| harlowja | and publish a calendar | 19:55 |
| harlowja | lol | 19:55 |
| pleia2 | they do live demos of the difference engine though! | 19:55 |
| harlowja | ya, that thing is noisy | 19:55 |
| *** cloudtrainme has joined #openstack-infra | 19:55 | |
| harlowja | *from what i remember | 19:55 |
| pleia2 | yeah | 19:55 |
| pleia2 | computers are still noisy | 19:56 |
| pleia2 | :) | 19:56 |
| samueldmq | is our current system able to un-do workflow+1 ? | 19:56 |
| nibalizer | fungi: clarkb ++ | 19:56 |
| fungi | they'll also demo the card sorter | 19:56 |
| clarkb | samueldmq: yes | 19:56 |
| clarkb | just vote 0 | 19:56 |
| samueldmq | clarkb: https://review.openstack.org/#/c/148995/ | 19:56 |
| clarkb | samueldmq: there is no workflow on that change | 19:56 |
| samueldmq | clarkb: looks like it got merged ? jenkins just posted a 'check' mark on that, it's weird | 19:56 |
| clarkb | its not merged | 19:56 |
| *** peterlisak13 has quit IRC | 19:57 | |
| clarkb | status is review in progresss | 19:57 |
| samueldmq | clarkb: there was, and it was removed, but jenkins just posted a ceck mark there | 19:57 |
| clarkb | jenkins will continue to run tests | 19:57 |
| samueldmq | clarkb: great! just wanted to check | 19:57 |
| clarkb | so jenkins reported but nothing merged | 19:57 |
| *** tonytan4ever has quit IRC | 19:57 | |
| clarkb | fungi: pleia2 how do we want to split up the work? | 19:58 |
| samueldmq | clarkb: thanks for clarifying | 19:58 |
| *** dramalho has joined #openstack-infra | 19:58 | |
| clarkb | fungi: pleia2 I am happy to do whatever bits including github mouse clicking | 19:58 |
| fungi | clarkb: pleia2: i'm happy to pick up the dregs of whatever you two don't want | 19:58 |
| fungi | heh | 19:58 |
| nibalizer | I can do whatever | 19:58 |
| fungi | we're all so indecisive | 19:58 |
| * pleia2 hops on the etherpad | 19:58 | |
| clarkb | nibalizer: have you done the gerrity and zuul things yet? | 19:59 |
| fungi | i'll send the status alert | 19:59 |
| clarkb | nibalizer: those are good to do if not | 19:59 |
| nibalizer | I have not done anything | 19:59 |
| nibalizer | I just opened laptop | 19:59 |
| *** Sree has joined #openstack-infra | 19:59 | |
| clarkb | nibalizer: I mean with previous renames | 19:59 |
| *** ilyashakhat has joined #openstack-infra | 19:59 | |
| nibalizer | i've done the zuul capture queue things | 19:59 |
| nibalizer | ive not done the gerritdb thigns | 19:59 |
| *** abregman is now known as abregman|nearby | 19:59 | |
| pleia2 | oh, I was just signing up for the gerrit stuff | 19:59 |
| * clarkb nominates nibalizer for gerritdb and indexing and related items | 20:00 | |
| fungi | #status alert Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance | 20:00 |
| openstackstatus | fungi: sending alert | 20:00 |
| * pleia2 hands off to nibalizer | 20:00 | |
| fungi | did we land zaro's documentation update to parallelize the lucene reindex? | 20:00 |
| fungi | it should go a lot faster if we do that | 20:00 |
| clarkb | pleia2: you cna have it :) | 20:01 |
| pleia2 | ok, I've got zuul things and git renames | 20:01 |
| fungi | stopping puppet now | 20:01 |
| clarkb | I will do github since I mentiond that I don't mind doing the mouse clicking | 20:01 |
| pleia2 | nibalizer: I added you to all the review.o.o stuff | 20:01 |
| nibalizer | woot | 20:01 |
| clarkb | nibalizer: you have etherpad link? | 20:02 |
| clarkb | looks like you are in the user list | 20:02 |
| -openstackstatus- NOTICE: Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance | 20:02 | |
| *** ChanServ changes topic to "Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance" | 20:02 | |
| nibalizer | clarkb: i do | 20:02 |
| pleia2 | fungi: good to do zuul? | 20:02 |
| fungi | yep, it's stopped now | 20:02 |
| pleia2 | k | 20:03 |
| *** bharathm has quit IRC | 20:03 | |
| pleia2 | zuul is stopped | 20:03 |
| pleia2 | nibalizer: you're up | 20:03 |
| nibalizer | kk | 20:03 |
| *** Sree has quit IRC | 20:04 | |
| pleia2 | nibalizer: let me know when you start step 7 and I'll begin the git renames :) | 20:04 |
| openstackstatus | fungi: finished sending alert | 20:04 |
| nibalizer | gerrit stopped | 20:05 |
| *** bpokorny_ has quit IRC | 20:05 | |
| fungi | hrm, looks like we approved the documentation update which switches the rename process to the ansible playbook | 20:05 |
| clarkb | uh | 20:05 |
| clarkb | I didn't realize we had a working playbook yet | 20:05 |
| clarkb | or that it had been tested | 20:06 |
| fungi | i guess we can give it a shot next time | 20:06 |
| clarkb | last I head it was in the "known to not work" state | 20:06 |
| *** sputnik13 has quit IRC | 20:06 | |
| * mordred hadn't spent a ton of mental energy on the playbook because I was hoping we were almost done with renames for our lives | 20:06 | |
| nibalizer | Rows matched: 0 Changed: 0 Warnings: 0 | 20:07 |
| nibalizer | just means no one is actually watching that project ya? | 20:07 |
| clarkb | nibalizer: yes | 20:07 |
| nibalizer | which puppet-openstack-health I can believei | 20:07 |
| fungi | jhesketh and yolanda must think it's working because they +2'd and approved it | 20:07 |
| nibalizer | the real work is in the 'changes' table ya? | 20:07 |
| *** jroll is now known as tjroll | 20:07 | |
| nibalizer | Rows matched: 10 Changed: 10 Warnings: 0 | 20:07 |
| nibalizer | is more 'work done' | 20:07 |
| fungi | nibalizer: yep, looks good | 20:07 |
| fungi | nibalizer: mainly keep an eye out for warnings/errors | 20:07 |
| nibalizer | kk db done | 20:08 |
| nibalizer | moving git repos not | 20:08 |
| nibalizer | now* | 20:08 |
| *** mmmpork is now known as mmmborkborkbork | 20:08 | |
| * pleia2 adds DONE notes to etherpad | 20:08 | |
| clarkb | I am going to go ahead and start github things now | 20:09 |
| fungi | anyway, see http://git.openstack.org/cgit/openstack-infra/system-config/tree/doc/source/gerrit.rst?id=9b44bbaede66655e9f0089a8516584d4ef6de671#n484 for the new and faster reindex command line | 20:09 |
| *** _ody has quit IRC | 20:09 | |
| fungi | i'll update the etherpad | 20:09 |
| *** mwagner has quit IRC | 20:09 | |
| *** bhunter71 has joined #openstack-infra | 20:10 | |
| fungi | it's basically the old command just append --threads 4 | 20:10 |
| pleia2 | cool | 20:10 |
| fungi | zaro's tested it on a dump of our production data and it works well, basically a 4x speedup | 20:10 |
| pleia2 | when threading works like it should \o/ | 20:10 |
| fungi | puppet's already updated to do this on the corresponding exec too | 20:10 |
| nibalizer | moves completed | 20:10 |
| nibalizer | starting reindexes | 20:11 |
| fungi | hopefully this should now complete in just a few minutes rather than 10-20 | 20:11 |
| *** ilyashakhat has quit IRC | 20:11 | |
| *** sputnik13 has joined #openstack-infra | 20:11 | |
| nibalizer | java jar ir jaring | 20:12 |
| nibalizer | is jaring | 20:12 |
| *** bpokorny has joined #openstack-infra | 20:12 | |
| pleia2 | ok, doing the git{01-08} moves | 20:12 |
| *** cloudtrainme has quit IRC | 20:12 | |
| pabelanger | jroovie | 20:12 |
| *** _ody has joined #openstack-infra | 20:12 | |
| clarkb | github slow | 20:13 |
| fungi | slow github is slow | 20:13 |
| *** salv-orl_ has quit IRC | 20:13 | |
| clarkb | I had to change gerrit's github perms after ownership transfer | 20:14 |
| *** dramalho has quit IRC | 20:14 | |
| *** ldnunes has quit IRC | 20:14 | |
| pleia2 | git moves all done | 20:15 |
| *** achanda has quit IRC | 20:15 | |
| *** dramalho has joined #openstack-infra | 20:15 | |
| nibalizer | ~33% | 20:15 |
| pleia2 | ooh, that is speedy :) | 20:15 |
| clarkb | github is done | 20:17 |
| EmilienM | where can I patch to have openstackstatus user connected on #puppet-openstack ? | 20:17 |
| clarkb | I should write a patch to the docs to add the gerrit write perms to github step | 20:17 |
| *** asilenkov_ has joined #openstack-infra | 20:17 | |
| samueldmq | is gerrit down ? | 20:18 |
| pleia2 | EmilienM: system-config manifests/site.pp | 20:18 |
| EmilienM | samueldmq: yes, maintainance | 20:18 |
| pabelanger | samueldmq: topic | 20:18 |
| EmilienM | pleia2: ack thx | 20:18 |
| fungi | some day someone will move the statusbot config to a file in project-config | 20:19 |
| samueldmq | EmilienM: pabelanger nice, sorry didn't notice, got confused with tzs :) | 20:19 |
| pleia2 | samueldmq: we're always UTC :) | 20:19 |
| fungi | and also the meetbot config for that matter | 20:19 |
| pleia2 | eavesdrop in system-config does make me a sad panda | 20:20 |
| fungi | i'm more pandasad over the giant one-line list of channels in the global site manifest for statusbot | 20:21 |
| *** Piet has joined #openstack-infra | 20:21 | |
| pleia2 | it is pretty awful | 20:22 |
| docaedo | which bot logs channels? | 20:22 |
| clarkb | docaedo: meetbot | 20:22 |
| fungi | e.g. the openstack_project::eavesdrop class in system-config | 20:22 |
| samueldmq | pleia2: nice, now I learned :) | 20:22 |
| pleia2 | samueldmq: I put a UTC clock in the panel on my desktop to help my brain | 20:23 |
| docaedo | clarkb: thanks, I need to see if I missed something because I don't think #openstack-app-catalog is logged | 20:23 |
| clarkb | docaedo: is the 'openstack' nick in channel? | 20:23 |
| EmilienM | pleia2: that's smart | 20:23 |
| clarkb | if yes then it should be logged | 20:23 |
| pleia2 | docaedo: http://eavesdrop.openstack.org/irclogs/%23openstack-app-catalog/ | 20:23 |
| docaedo | clarkb: no, it's not | 20:23 |
| *** rfolco has quit IRC | 20:23 | |
| clarkb | docaedo: then ya I think you need to update the system-config class that fungi pointed out | 20:24 |
| fungi | i just set all my clocks to utc except the ones my wife will complain about | 20:24 |
| pleia2 | docaedo: it's logged ^^ | 20:24 |
| clarkb | oh it is logged | 20:24 |
| samueldmq | pleia2: hm, good idea, looks to be very useful, thanks for sharing | 20:24 |
| docaedo | pleia2: yes I see that, thanks! I was wrong :) | 20:24 |
| docaedo | pleia2: yay! | 20:24 |
| nibalizer | ~66% | 20:24 |
| clarkb | nibalizer: so I guess its not much faster :/ | 20:24 |
| clarkb | oh well | 20:24 |
| pleia2 | yeah | 20:24 |
| fungi | nibalizer: this was with --threads 4 ? | 20:25 |
| *** sputnik13 has quit IRC | 20:25 | |
| nibalizer | yes | 20:25 |
| fungi | zaro: ^ i guess the parallel reindex isn't appreciably faster in production after all | 20:26 |
| clarkb | I wonder if there is a fixed overhead we don't really get past | 20:26 |
| clarkb | or maybe writing to the index is serial? | 20:26 |
| clarkb | and so 1 or 4 threads they are all synchronizing through writes | 20:26 |
| purp | Hi, smart infra folks. I'm here to be clueless and beg help again. | 20:27 |
| *** glauco_ has quit IRC | 20:27 | |
| *** sputnik13 has joined #openstack-infra | 20:27 | |
| purp | Trying to git review and get an ssh error I don't know how to resolve: http://paste.openstack.org/show/HsVVPoQTos5beNanMUSO/ | 20:27 |
| fungi | purp: we're wrapping up a maintenance window for our code review/ci system. will look in a moment | 20:27 |
| purp | No rush. | 20:27 |
| purp | I suspect mine might be related. =] | 20:28 |
| clarkb | well git revie wwon't work right now | 20:28 |
| clarkb | gerrit is off | 20:28 |
| fungi | yeah, the outage is running a few minutes over what i projected unfortunately | 20:28 |
| clarkb | its all in the index | 20:28 |
| nibalizer | Reindexed 238640 changes in 1040.3s (229.4/s) | 20:29 |
| fungi | w00t | 20:29 |
| fungi | looks like we're clear to start gerrit and ok the alert | 20:29 |
| nibalizer | am i good to proceed win #10 ? | 20:29 |
| fungi | yep | 20:30 |
| pleia2 | yep | 20:30 |
| pleia2 | and once gerrit has its head on straight I'll start up zuul | 20:30 |
| fungi | pleia2: make sure you wait a moment after starting zuul before reenqueuing | 20:30 |
| pleia2 | fungi: ok | 20:31 |
| fungi | helps to check the gearman status to make sure all the common node types have registered | 20:31 |
| pleia2 | fungi: ok, how do I do that? | 20:31 |
| *** tjones has quit IRC | 20:31 | |
| *** ildikov has quit IRC | 20:31 | |
| fungi | otherwise you end up with a bunch of the reenqueued changes bailing out with jobs NOT_REGISTERED | 20:31 |
| pleia2 | right | 20:31 |
| nibalizer | gerrit started | 20:32 |
| fungi | echo status|nc -q 3 localhost 4730|grep <stuff> | 20:32 |
| clarkb | I will rename groups now | 20:32 |
| nibalizer | pleia2: you're good to start zuul i think | 20:32 |
| pleia2 | starting zuul | 20:32 |
| *** kushal has quit IRC | 20:32 | |
| *** tongli has quit IRC | 20:33 | |
| pleia2 | fungi: stuff would be job names? | 20:33 |
| *** pc_m has quit IRC | 20:33 | |
| fungi | pleia2: node labels. i'd use -c to just get counts and make sure they're nonzero | 20:33 |
| fungi | pleia2: like `echo status|nc -q 3 localhost 4730|grep -c bare-centos6` | 20:33 |
| pleia2 | ok cool | 20:34 |
| asilenkov_ | you guys should replace that ugly 503 with sexy announce )) | 20:34 |
| pabelanger | asilenkov_: ha ha, working on something now | 20:34 |
| clarkb | group renaming and hiding is all done | 20:34 |
| pabelanger | asilenkov_: thought the same | 20:34 |
| pleia2 | looking good | 20:35 |
| fungi | pleia2: awesome, i'll clear the alert | 20:35 |
| fungi | #status ok | 20:35 |
| openstackstatus | fungi: sending ok | 20:35 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: site.pp: add openstackstatus to #puppet-openstack https://review.openstack.org/242621 | 20:35 |
| fungi | and i'll go ahead and start approving our project-config change series now | 20:35 |
| *** aysyd has quit IRC | 20:35 | |
| clarkb | fungi: I had preivously -2d some of them but all of them lgtm | 20:36 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 20:36 | |
| *** hrubi has quit IRC | 20:36 | |
| *** otter768 has joined #openstack-infra | 20:36 | |
| pleia2 | fungi: hang on, check.sh is finishing | 20:36 |
| fungi | pleia2: it shouldn't interfere | 20:36 |
| pleia2 | ok | 20:36 |
| fungi | these are going straight into the gate pipeline anyway | 20:37 |
| *** dims is now known as dimsum__ | 20:37 | |
| *** tnarg_ has joined #openstack-infra | 20:37 | |
| nibalizer | neat | 20:37 |
| openstackstatus | fungi: finished sending ok | 20:37 |
| nibalizer | im gonna go offline for a bit | 20:37 |
| *** hrubi has joined #openstack-infra | 20:38 | |
| pleia2 | ok, all done with #11 | 20:38 |
| fungi | all 5 project-config changes are in the gate pipeline now | 20:38 |
| fungi | nibalizer: i'm a little chicken-and-egg confused on 242586 | 20:39 |
| fungi | there's a review comment saying it shouldn't merge until the rename is complete, but a comment in the etherpad saying that it needs to be merged before puppet is reenabled (yet puppet does necessary bits to complete the rename) | 20:39 |
| clarkb | fungi: I think update modules will just fail until we get that in | 20:39 |
| clarkb | so merge that change, then enable puppet | 20:40 |
| fungi | except that change won't pass tests until the rename is complete | 20:40 |
| *** tnarg has quit IRC | 20:40 | |
| *** nadya has joined #openstack-infra | 20:40 | |
| clarkb | oh heh | 20:40 |
| *** otter768 has quit IRC | 20:40 | |
| clarkb | is it not complete enough to work | 20:41 |
| clarkb | it just needs to be able to clone/fetch from that location | 20:41 |
| fungi | oh, true it's already hopefully able to clone from the new name on our git server farm. i'll recheck it now | 20:41 |
| clarkb | ya I think it may work | 20:42 |
| *** bauwser is now known as bauzas | 20:42 | |
| *** wenlock has quit IRC | 20:42 | |
| fungi | gate-infra-puppet-apply-dsvm-f21: NOR_REGISTERED | 20:43 |
| clarkb | oh and thats voting | 20:43 |
| fungi | er, NOT_... | 20:43 |
| fungi | yep | 20:43 |
| pleia2 | :\ | 20:43 |
| pleia2 | I didn't check f21 | 20:43 |
| fungi | so we need to get devstack-f21 nodes built first and i'll recheck it again | 20:43 |
| *** dramalho has quit IRC | 20:43 | |
| *** Guest88047 is now known as jdg | 20:43 | |
| *** jdg is now known as jgriffith | 20:44 | |
| notmorgan | . | 20:44 |
| *** dramalho has joined #openstack-infra | 20:44 | |
| *** cloudtrainme has joined #openstack-infra | 20:44 | |
| clarkb | I don't see any building right now | 20:44 |
| notmorgan | whoopse. wrong channel | 20:44 |
| *** tjones has joined #openstack-infra | 20:45 | |
| fungi | clarkb: i see one in a used state for the past hour | 20:45 |
| *** nadya has quit IRC | 20:45 | |
| clarkb | fungi: we can probably delte that one since we restarted nodepool | 20:45 |
| *** wenlock_ has quit IRC | 20:45 | |
| clarkb | er zuul | 20:45 |
| fungi | we may need to wait for a second demand pulse before it notices it needs one | 20:45 |
| fungi | yeah, good point, that job's reporting nowhere now | 20:45 |
| clarkb | and there won't be extra demand until we register the jobs :) | 20:46 |
| clarkb | min-ready is important | 20:46 |
| fungi | okay, it's deleted | 20:46 |
| fungi | do we not have a min-ready of at least 1 for those? | 20:46 |
| clarkb | we probably do but thats used to determine ratio for readyness vs the other labels | 20:46 |
| fungi | yeah, we do have min-ready: 1 | 20:46 |
| clarkb | so it may be a lower priority | 20:46 |
| fungi | yep, great point. we jumped up to max quota when the changes all got reenqued | 20:47 |
| fungi | clarkb: also, all the same apply jobs failed on that change again anyway, so it seems to need more than just the manual rename steps completed | 20:48 |
| clarkb | ok let me go look at logs to see if thats fixable | 20:48 |
| clarkb | er I don't see them failing yet | 20:49 |
| clarkb | just the NOT_REGISTERED | 20:49 |
| *** kushal has joined #openstack-infra | 20:49 | |
| clarkb | are we looking at the same change in zuul status/ | 20:50 |
| clarkb | also I had brie and toast and ginger green tea for this project rename | 20:51 |
| fungi | oh, i bet i was looking at the wrong change | 20:51 |
| clarkb | every project rename should come with food | 20:51 |
| openstackgerrit | Merged openstack-infra/project-config: Rename projects akanda-* -> astara-* https://review.openstack.org/241328 | 20:51 |
| pleia2 | I have winter squash soup | 20:52 |
| fungi | clarkb: yeah, sorry. the not registered job has it failing, but it hasn't gotten any other nodes assigned yet either | 20:52 |
| *** daemontool has joined #openstack-infra | 20:52 | |
| *** mwagner has joined #openstack-infra | 20:52 | |
| *** salv-orlando has joined #openstack-infra | 20:53 | |
| *** gordc has quit IRC | 20:53 | |
| clarkb | https://review.openstack.org/#/admin/groups/638,info I updated the description there too | 20:53 |
| fungi | that previously used devstack-f21 node is sitting in delete state for 7 minutes and counting. nodepool still doesn't think it has sufficient incentive to provision another one yet due to the backlog demand | 20:53 |
| clarkb | but no members, not visible, and admin owned should do it | 20:53 |
| fungi | looks like all 5 project-config changes succeeded, so now we just need to wait for zuul to finish flushing pending events/results | 20:54 |
| openstackgerrit | Merged openstack-infra/project-config: Rename puppet project 'puppet-openstack_health' https://review.openstack.org/230421 | 20:54 |
| openstackgerrit | Merged openstack-infra/project-config: Rename project networking-bagpipe(-l2) https://review.openstack.org/237936 | 20:54 |
| openstackgerrit | Merged openstack-infra/project-config: Move networking-bigswitch from stackforge to openstack https://review.openstack.org/237753 | 20:55 |
| openstackgerrit | Merged openstack-infra/project-config: Moving compass-install from stackforge to openstack namespace https://review.openstack.org/241788 | 20:55 |
| *** dramalho has quit IRC | 20:55 | |
| fungi | that's all of them | 20:55 |
| *** cloudtrainme has quit IRC | 20:55 | |
| fungi | amd yjere | 20:56 |
| *** dramalho has joined #openstack-infra | 20:56 | |
| *** harlowja_ has joined #openstack-infra | 20:56 | |
| fungi | grrr | 20:56 |
| fungi | there's a devstack-f21 node building now | 20:56 |
| *** harlowja has quit IRC | 20:56 | |
| *** dramalho has quit IRC | 20:56 | |
| fungi | once/if it registers those jobs, i'll just enqueue 242586 straight into the gate and takes our chances | 20:56 |
| *** sputnik13 has quit IRC | 20:57 | |
| fungi | otherwise it's going to be an hour or more to get sufficient nodes to rerun check | 20:57 |
| clarkb | would be nice to see the other jobs run and pass, it will all come down to how fast we can boot an f21 node | 20:57 |
| clarkb | oh right the gate wins | 20:57 |
| pleia2 | so would it help to do a more thorough gearman check of *all* the nodes? | 20:57 |
| *** BobBall is now known as BobBall_AWOL | 20:57 | |
| *** sputnik13 has joined #openstack-infra | 20:57 | |
| fungi | probably. just nobody's written/documented one afaik | 20:57 |
| clarkb | or stop testing our stuff on fedora | 20:58 |
| pleia2 | clarkb: hee | 20:58 |
| clarkb | is that only for the slave manifest? | 20:58 |
| clarkb | I doubt we would ever run services on fedora | 20:58 |
| clarkb | must be for the slave manifest | 20:58 |
| fungi | part of the challenge is that zuul doesn't know what all the possible node labels are, only nodepool does, so we need to marry nodepool configuration with a command run on zuul | 20:58 |
| *** burgerk has quit IRC | 20:58 | |
| * pleia2 nods | 20:59 | |
| fungi | okay, that f21 node went ready. i'll go aheand and enqueue | 20:59 |
| *** ZZelle_ has joined #openstack-infra | 21:00 | |
| *** Sree has joined #openstack-infra | 21:00 | |
| *** alivigni has quit IRC | 21:00 | |
| *** openstackgerrit has quit IRC | 21:01 | |
| fungi | and it's in the gate now | 21:01 |
| fungi | lookit that f21 job go! | 21:01 |
| pleia2 | yay | 21:02 |
| *** openstackgerrit has joined #openstack-infra | 21:02 | |
| *** kdas_ has joined #openstack-infra | 21:02 | |
| *** alivigni has joined #openstack-infra | 21:02 | |
| fungi | once it merges i'll unleash the puppets | 21:02 |
| clarkb | now I want to play little big planet | 21:03 |
| *** mriedem has left #openstack-infra | 21:03 | |
| *** mriedem has joined #openstack-infra | 21:04 | |
| *** kushal has quit IRC | 21:04 | |
| *** bhunter71 has quit IRC | 21:04 | |
| clarkb | f21 job passed | 21:04 |
| clarkb | so I think having the git repos renamed was sufficient | 21:04 |
| fungi | yep, trusty now too | 21:05 |
| *** Sree has quit IRC | 21:05 | |
| fungi | should be in the clear as soon as we get a bare-centos6 and devstack-centos7 in the gate | 21:05 |
| *** kdas_ has quit IRC | 21:05 | |
| *** kushal has joined #openstack-infra | 21:05 | |
| *** alivigni has quit IRC | 21:06 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: site.pp: add openstackstatus to #puppet-openstack https://review.openstack.org/242621 | 21:06 |
| *** rlandy has quit IRC | 21:09 | |
| *** cloudtrainme has joined #openstack-infra | 21:10 | |
| *** thorst has quit IRC | 21:11 | |
| pleia2 | pabelanger: wow, loading data into stackalytics isn't exactly quick :) maybe your instructions should be "never let this server turn off" | 21:12 |
| *** thorst has joined #openstack-infra | 21:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add CRUD methods for keystone groups https://review.openstack.org/242633 | 21:12 |
| fungi | pleia2: we're using a trove instance for this, right? | 21:12 |
| *** rlandy has joined #openstack-infra | 21:13 | |
| clarkb | just one more job queued | 21:13 |
| pleia2 | fungi: no, it doesn't use mysql, it stores stuff in RAM (yay! no wait) | 21:13 |
| fungi | oh! no wonder it's slow | 21:13 |
| * pleia2 nods | 21:13 | |
| fungi | (and needs an insanely large server) | 21:13 |
| pleia2 | right | 21:13 |
| pleia2 | it may be worthy of some refactoring by someone clever if we ever have time now that it will come live with us | 21:14 |
| clarkb | and last job started | 21:15 |
| openstackgerrit | Merged openstack-infra/system-config: Treat openstack_health normally https://review.openstack.org/242586 | 21:16 |
| *** thorst has quit IRC | 21:16 | |
| clarkb | jeblair: what do you think about removing that zuul job retry logic if it looks like a jenkins fail? Now that we have 8 masters and are restarting them weekly we seem to have much healthier jenkinses | 21:16 |
| clarkb | jeblair: just thinking about it due to the devstack-gate change that got stuck recently | 21:16 |
| clarkb | fungi: ^ | 21:16 |
| clarkb | I think we are ready to run puppet now | 21:16 |
| fungi | yup, starting | 21:16 |
| fungi | uncommented in crontab and manually started | 21:20 |
| fungi | fatal: repository 'https://git.openstack.org/openstack-infra/puppet-openstack-health/' not found | 21:20 |
| clarkb | fungi: oh! | 21:21 |
| clarkb | it doesn't reexec itself | 21:21 |
| clarkb | so I think if you rerun now it will be fine | 21:21 |
| clarkb | so it always runs a version behind | 21:21 |
| clarkb | er thats not quite a correct statement | 21:22 |
| pleia2 | confirmed it does exist on the filesystems | 21:22 |
| *** thorst has joined #openstack-infra | 21:22 | |
| clarkb | it updates the version but it does not reexec so will continue on the old version if it was updated | 21:22 |
| *** abregman|nearby has quit IRC | 21:22 | |
| clarkb | pleia2: did you see japanese force awakens trailer? | 21:22 |
| *** rfolco has joined #openstack-infra | 21:22 | |
| pleia2 | clarkb: nope | 21:22 |
| *** sergmelikyan has quit IRC | 21:22 | |
| *** signed8bit has joined #openstack-infra | 21:22 | |
| pleia2 | I did manage to get tickets for the 17th though \o/ | 21:22 |
| clarkb | pleia2: let me get a link | 21:23 |
| fungi | also we didn't accept the ssh key for codesearch.o.o yet so i did that just now | 21:23 |
| clarkb | https://www.youtube.com/watch?v=SdAUiyeJMFQ | 21:23 |
| pleia2 | fungi: oops, thanks | 21:23 |
| clarkb | 36 seconds is a pretty neat shot | 21:23 |
| pleia2 | need to for stackalytics.o.o too | 21:23 |
| clarkb | fungi: so you restarted it and it was happy? | 21:23 |
| *** alivigni has joined #openstack-infra | 21:24 | |
| fungi | pleia2: i went ahead and deleted/accepted the stackalytics.o.o key now too | 21:24 |
| fungi | clarkb: er, no, looks like it's still trying to hit the old clone url | 21:25 |
| pleia2 | fungi: thanks | 21:25 |
| fungi | also it complains "destination path '/etc/puppet/modules/apps_site' already exists and is not an empty directory." | 21:25 |
| clarkb | fungi: oh | 21:25 |
| pleia2 | clarkb: bb-8 makes me so happy :) | 21:25 |
| clarkb | fungi: I bet the repo we have on disk needs to have its remotes updated | 21:25 |
| clarkb | fungi: or we can delete it and script should reclone | 21:25 |
| fungi | ahh | 21:25 |
| fungi | pypi.sjc1.openstack.org is new as well, no accepted ssh host key yet | 21:27 |
| clarkb | pleia2: I am afraid to do a release night showing, I have to line up babysitters (grandparents) too | 21:27 |
| pleia2 | clarkb: I was on a plane when tickets when on sale, but we got lucky to be on a list for $company-rented-theater showing | 21:28 |
| pleia2 | otherwise it's impossible in the geek land that is san francisco | 21:28 |
| fungi | clarkb: yeah, looks like we had a stale remote there, so removed now and will see if it reclones correctly | 21:28 |
| clarkb | I wonder if I can watch it in a theater that sells beer | 21:28 |
| clarkb | fungi: ok | 21:28 |
| fungi | also blew away apps_site since it was erroring too | 21:29 |
| fungi | looks like it recloned them both sanely now | 21:29 |
| pabelanger | pleia2: IKR! We'll be doing daily data dumps from memcached, so its really the first import that takes a long, long time (ago?) | 21:30 |
| clarkb | fungi: yay | 21:30 |
| pabelanger | pleia2: but it is working as expected | 21:30 |
| pleia2 | pabelanger: right, that helps :) | 21:30 |
| pabelanger | okay, time for a bus ride with the kids. Will be back later tonight | 21:31 |
| fungi | puppet is disabled on groups.o.o... will try to remember to ask mrmartin if that can be undone next time he's around | 21:32 |
| fungi | disabled on logstash.o.o too clarkb? | 21:32 |
| fungi | is that still needed? | 21:32 |
| *** abregman|nearby has joined #openstack-infra | 21:32 | |
| clarkb | fungi: if those changes to fix kibana havne't merged yet we need it | 21:32 |
| clarkb | I will check | 21:32 |
| fungi | k | 21:33 |
| fungi | Error: /Package[faraday]: Provider gem is not functional on this host | 21:33 |
| openstackgerrit | Tushar Gohad proposed openstack/requirements: Bump PyECLib version to 1.1.1 https://review.openstack.org/238771 | 21:33 |
| fungi | looks like we're failing to upgrade elasticsearch | 21:34 |
| clarkb | looks like they merged, I will do a noop run and enable if looks good | 21:34 |
| fungi | yeah, looks like puppet-elasticsearch is failing to apply somewhere because gem is broken | 21:36 |
| clarkb | fungi: which host? | 21:37 |
| fungi | i can't tell from the log. will check puppetboard | 21:38 |
| regXboi | clarkb, fungi: are multinode jobs having issues? | 21:38 |
| fungi | hrm, well the failure didn't report into puppetdb | 21:38 |
| fungi | regXboi: let me guess, jobs reporting a status of NOT_REGISTERED | 21:38 |
| regXboi | fungi: not sure yet - still looking through logs | 21:38 |
| fungi | regXboi: oh, if there are logs then that's not it | 21:39 |
| *** tonytan4ever has joined #openstack-infra | 21:39 | |
| fungi | so probably nothing related to our ci maintenance in that case | 21:39 |
| jeblair | clarkb: i don't think we should remove the retry -- it actually gets triggered constantly. instead, i think we should put a counter on it; max 3x or even 1x. | 21:39 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Move irc bot from #akanda->#openstack-astara https://review.openstack.org/242642 | 21:39 |
| clarkb | jeblair: that seems reasonable too | 21:39 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/system-config: Move eavesdrop bot from #akanda->#openstack-astara https://review.openstack.org/242643 | 21:40 |
| clarkb | regXboi: the only thing I know of as being broken is shelve | 21:40 |
| clarkb | regXboi: otherwise last I saw the test env itself was working fine | 21:40 |
| clarkb | fungi: I am running puppet on logstsah.o.o looks fine so will enable when done | 21:40 |
| fungi | awesome | 21:41 |
| regXboi | fungi, clarkb: ok - I happened to poke my nose at the neutron multinode jobs and they are headed off to 100% failure rate - looking at a sample now to understand what broke things | 21:41 |
| clarkb | regXboi: do you have a link? | 21:41 |
| *** kushal has quit IRC | 21:42 | |
| regXboi | clarkb: to the graph or to a sample job? | 21:42 |
| clarkb | to the job that is failing | 21:42 |
| regXboi | http://logs.openstack.org/67/234067/5/check/gate-tempest-dsvm-neutron-multinode-full/2a859d9/ | 21:42 |
| regXboi | I'm seeing a return value of 2 in the console | 21:43 |
| clarkb | regXboi: http://logs.openstack.org/67/234067/5/check/gate-tempest-dsvm-neutron-multinode-full/2a859d9/logs/subnode-2/devstacklog.txt.gz#_2015-11-06_08_03_32_033 | 21:43 |
| clarkb | its the lxml compile fail issue | 21:43 |
| clarkb | I think sdague was fixing that in devstack | 21:43 |
| regXboi | clarkb: ok, let me do a logstash query and see if that covers all of what I'm seeing | 21:44 |
| sdague | clarkb: yes | 21:44 |
| sdague | clarkb: https://review.openstack.org/#/c/242559/ | 21:44 |
| * regXboi goes and reads | 21:45 | |
| sdague | clarkb: you want to ack that? I'll self approve as our ds cores are sparse this week | 21:45 |
| clarkb | let me read it | 21:45 |
| *** ZZelle_ has quit IRC | 21:46 | |
| *** rlandy has quit IRC | 21:46 | |
| clarkb | sdague: is rhel based stuff ok? | 21:46 |
| fungi | okay, ansible puppet run completed | 21:46 |
| sdague | rhel based stuff is a further fix | 21:46 |
| nibalizer | fungi: ah sorry, didn't mean to leave you hanging on that one | 21:46 |
| nibalizer | clarks analysis was correct | 21:46 |
| sdague | but that's not impacting any jobs atm | 21:46 |
| sdague | af | 21:46 |
| clarkb | sdague: ok | 21:46 |
| sdague | afaik | 21:47 |
| fungi | nibalizer: no worries. 'twas sorted | 21:47 |
| clarkb | sdague: last question, you remove it from keystone but don't put the libs there, is that because keystone has actually cleaned up their dep on it now? | 21:47 |
| *** sridhar_ram has quit IRC | 21:47 | |
| sdague | it's in base | 21:47 |
| fungi | working on step 17 now | 21:47 |
| sdague | sorry, in general | 21:47 |
| clarkb | sdague: right but you added it to nova | 21:47 |
| clarkb | should it be removed from nova completely too? | 21:47 |
| sdague | probably wouldn't hurt | 21:48 |
| sdague | it's all addative today | 21:48 |
| sdague | so also being in nova doesn't hurt anything | 21:48 |
| regXboi | does logstash.o.o stash devstack logs? | 21:48 |
| clarkb | regXboi: yes | 21:48 |
| clarkb | sdague: ok | 21:49 |
| clarkb | regXboi: though it may not be getting the subnode logs proprely | 21:49 |
| sdague | there are follow ups, but I'm done in about 5 minutes for the week. So if you are good with what's there I'll approve it | 21:49 |
| clarkb | regXboi: in any case all of those fails are likely due to this bug | 21:49 |
| sdague | otherwise we can fix it next week | 21:49 |
| clarkb | sdague: ya I +1'd | 21:49 |
| sdague | coolio | 21:49 |
| sdague | ok approved | 21:50 |
| regXboi | clarkb: It looks like it may not be getting the sub logs in an obvious fashion | 21:50 |
| *** hdd has quit IRC | 21:50 | |
| regXboi | but that's for another day | 21:50 |
| sdague | yeh, I would not be surprised if subnodes aren't being indexed | 21:50 |
| clarkb | regXboi: its only indexing 3 of the subnode logs all nova | 21:51 |
| *** tjones has left #openstack-infra | 21:51 | |
| regXboi | clarkb: that would explain it then | 21:51 |
| fungi | step 17 completed. working on step 18 now | 21:51 |
| *** rlandy has joined #openstack-infra | 21:51 | |
| *** rfolco has quit IRC | 21:52 | |
| *** alivigni has quit IRC | 21:52 | |
| *** peterlisak13 has joined #openstack-infra | 21:53 | |
| clarkb | jenkins restarts will being in about 3 hours | 21:53 |
| clarkb | I am going to check and see if any have leaked since last restarts | 21:54 |
| clarkb | (useful to know if we are properly restarting leaky masters or if we just avoid it altogether) | 21:54 |
| regXboi | clarkb: if I'm still around when 242559 merges, I'll see about running a recheck on one of these patches to see if that patch did the trick | 21:54 |
| clarkb | regXboi: you cna also depends-on that change | 21:54 |
| regXboi | clarkb: yeah, but it's not my patch I'm going to recheck | 21:55 |
| clarkb | ooh 02 has properly leaked | 21:55 |
| clarkb | all your threads are belong to jenkins | 21:56 |
| regXboi | :) | 21:56 |
| clarkb | 04 as well | 21:56 |
| *** dprince has quit IRC | 21:57 | |
| *** peterlisak13 has quit IRC | 21:58 | |
| clarkb | btw the rolling restarts means we can do plugin upgrades unattended now too assuming they don't need config updates | 21:58 |
| *** bswartz has quit IRC | 21:58 | |
| *** rossella_s has quit IRC | 22:00 | |
| *** rossella_s has joined #openstack-infra | 22:00 | |
| *** Sree has joined #openstack-infra | 22:00 | |
| clarkb | jesusaurus: ok I think I am all done with rename related things, were you planning to get to os-loganalyze today? | 22:01 |
| clarkb | let me know if I can help with anything | 22:01 |
| fungi | (and assuming the plugin upgrades don't break something causing jenkins to fail to restart) | 22:02 |
| *** barnaby has quit IRC | 22:04 | |
| *** Sree has quit IRC | 22:05 | |
| *** shashank_hegde has joined #openstack-infra | 22:05 | |
| *** achanda has joined #openstack-infra | 22:08 | |
| *** rlandy has quit IRC | 22:11 | |
| *** lascii is now known as alaski | 22:13 | |
| jesusaurus | clarkb: im probably not going to get to that today | 22:16 |
| *** boris-42 has joined #openstack-infra | 22:21 | |
| *** mmmborkborkbork is now known as mmmpork | 22:22 | |
| openstackgerrit | Merged openstack-infra/project-config-example: Fix doc error for PUBLISH_HOST https://review.openstack.org/242249 | 22:23 |
| *** tiswanso has quit IRC | 22:23 | |
| *** shashank_hegde has quit IRC | 22:25 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstack_health: Update .gitreview for project rename https://review.openstack.org/242651 | 22:26 |
| *** August1914 has joined #openstack-infra | 22:26 | |
| *** achanda has quit IRC | 22:27 | |
| *** shashank_hegde has joined #openstack-infra | 22:27 | |
| *** sfinucan has quit IRC | 22:29 | |
| *** smarcet has joined #openstack-infra | 22:31 | |
| *** dtardivel has quit IRC | 22:33 | |
| *** signed8bit has quit IRC | 22:33 | |
| *** smarcet has quit IRC | 22:35 | |
| *** otter768 has joined #openstack-infra | 22:37 | |
| *** sputnik13 has quit IRC | 22:37 | |
| *** mriedem has quit IRC | 22:37 | |
| *** sputnik13 has joined #openstack-infra | 22:38 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder: Fix diskimage-builder image size https://review.openstack.org/242273 | 22:38 |
| EmilienM | infra puppet apply jobs are broken: http://logs.openstack.org/21/242621/2/check/gate-infra-puppet-apply-bare-trusty/507412b/console.html#_2015-11-06_21_32_17_913 | 22:38 |
| EmilienM | I guess you already know, just in case of | 22:38 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_health: Update .gitreview for project rename https://review.openstack.org/242651 | 22:39 |
| *** kgiusti has quit IRC | 22:39 | |
| *** w_ has joined #openstack-infra | 22:40 | |
| clarkb | EmilienM: I think they are fixed now | 22:40 |
| openstackgerrit | Alberto Gireud proposed openstack-infra/shade: Fix incorrect variable name https://review.openstack.org/242679 | 22:40 |
| *** doug-fish has quit IRC | 22:40 | |
| *** otter768 has quit IRC | 22:42 | |
| fungi | i've got step 18 wrapped up, so i'm going to switch to friday evening chores. i'm around if anybody needs me | 22:42 |
| *** olaph has quit IRC | 22:43 | |
| pleia2 | thanks fungi | 22:43 |
| nibalizer | thanks | 22:43 |
| EmilienM | clarkb: ack | 22:47 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: site.pp: add openstackstatus to #puppet-openstack https://review.openstack.org/242621 | 22:47 |
| *** TheJulia has quit IRC | 22:48 | |
| *** amitgandhinz has quit IRC | 22:51 | |
| *** shashank_hegde has quit IRC | 22:51 | |
| *** thorst has quit IRC | 22:54 | |
| *** thorst has joined #openstack-infra | 22:54 | |
| bookwar | what is the reason for Zuul to combine changes with same change id but to two different branches (usually cherry-picks) in one integrated queue? Is it bug of feature? | 22:55 |
| clarkb | bookwar: you can't do upgrade testing otherwise | 22:56 |
| *** yamamoto has joined #openstack-infra | 22:56 | |
| *** abregman_ has joined #openstack-infra | 22:57 | |
| *** thorst has quit IRC | 22:58 | |
| *** abregman|nearby has quit IRC | 23:00 | |
| bookwar | yes, sounds like a reason, but i am not sure is it really useful in my current case. Is there an option to disable it if i don't use upgrades test? (i mean in Zuul as a local service, not the one in Openstack Infra) | 23:00 |
| clarkb | bookwar: there isn't an option to disable it. Is there something specific yo uare trying to do? | 23:01 |
| clarkb | is it a problem that they are in the same queue? | 23:01 |
| *** Sree has joined #openstack-infra | 23:01 | |
| bookwar | clarkb: yes, i have long tests in stable branches, and i have much smaller coverage in master | 23:02 |
| bookwar | so my master fixes are get delayed by those huge tests running for stable | 23:02 |
| bookwar | ideally master should be tested and merged before it is cherry-picked.. | 23:03 |
| bookwar | may be i should just promote this policy more, but option to disable grouping by change-id would be nice | 23:04 |
| clarkb | bookwar: its not actually grouping by change id | 23:04 |
| clarkb | bookwar: its grouping because they are the same project | 23:04 |
| clarkb | or it is grouping because they share a job | 23:05 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Only install modules from source https://review.openstack.org/242683 | 23:05 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Remove 'removed' modules, they have been removed https://review.openstack.org/242684 | 23:05 |
| bookwar | there is no intersections in jobs | 23:05 |
| bookwar | but project yes is the same as one patch is the cherrypick for another | 23:06 |
| *** Sree has quit IRC | 23:06 | |
| clarkb | yup thats why thne | 23:07 |
| *** regXboi has quit IRC | 23:07 | |
| clarkb | the change id isn't related | 23:07 |
| clarkb | I am not sure if anyone has looked into adding that to zuul but I think jeblair is wanting to slow down zuul dev so that zuulv3 can be built | 23:07 |
| *** achanda has joined #openstack-infra | 23:08 | |
| *** achanda has quit IRC | 23:08 | |
| bookwar | we've just started to dive into Zuul v.2, i'll take a look at v3 version plans, but for now we'll need a workaround | 23:10 |
| bookwar | thanks anyway | 23:10 |
| *** thorst has joined #openstack-infra | 23:13 | |
| *** kozhukalov_ has quit IRC | 23:15 | |
| *** trown is now known as trown|outttypeww | 23:15 | |
| *** sabeen2 has quit IRC | 23:16 | |
| *** thorst has quit IRC | 23:17 | |
| *** _denisra has quit IRC | 23:18 | |
| *** tnarg_ has quit IRC | 23:18 | |
| *** yamamoto has quit IRC | 23:19 | |
| *** tnarg has joined #openstack-infra | 23:19 | |
| *** tnarg_ has joined #openstack-infra | 23:20 | |
| *** gondoi is now known as zz_gondoi | 23:22 | |
| *** tnarg has quit IRC | 23:24 | |
| *** xyang1 has quit IRC | 23:26 | |
| *** achanda has joined #openstack-infra | 23:29 | |
| *** cloudtrainme has quit IRC | 23:31 | |
| *** salv-orlando has quit IRC | 23:33 | |
| *** tonytan4ever has quit IRC | 23:33 | |
| *** ddieterly has joined #openstack-infra | 23:34 | |
| *** ddieterly has quit IRC | 23:34 | |
| *** ddieterly has joined #openstack-infra | 23:35 | |
| *** yamamoto has joined #openstack-infra | 23:35 | |
| *** thorst has joined #openstack-infra | 23:35 | |
| *** ddieterl_ has quit IRC | 23:36 | |
| *** achanda has quit IRC | 23:37 | |
| *** thorst has quit IRC | 23:40 | |
| *** ddieterly has quit IRC | 23:41 | |
| *** otter768 has joined #openstack-infra | 23:49 | |
| *** jtomasek has quit IRC | 23:51 | |
| *** dimsum__ has quit IRC | 23:52 | |
| *** yamamoto has quit IRC | 23:54 | |
| *** Swami_ has joined #openstack-infra | 23:56 | |
| *** Swami has quit IRC | 23:56 | |
| *** rfolco has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!