| *** dkranz has joined #openstack-infra | 00:00 | |
| *** Adri2000 has joined #openstack-infra | 00:01 | |
| *** ^d has joined #openstack-infra | 00:03 | |
| *** ^d has joined #openstack-infra | 00:03 | |
| *** ^d has quit IRC | 00:07 | |
| *** erfanian has joined #openstack-infra | 00:12 | |
| *** odyssey4me4 has quit IRC | 00:14 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Update git documentation with HAProxy details https://review.openstack.org/43385 | 00:15 |
|---|---|---|
| pleia2 | clarkb: ^^ it's not particularly thorough (intentionally) so please lmk if there are points which we should expand upon | 00:16 |
| *** CaptTofu has joined #openstack-infra | 00:17 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for emailing results via SMTP https://review.openstack.org/42645 | 00:17 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Separate reporters from triggers https://review.openstack.org/42643 | 00:17 |
| clarkb | will do. my dsl modem is being cranky | 00:17 |
| *** Ryan_Lane has quit IRC | 00:18 | |
| *** dims has quit IRC | 00:18 | |
| openstackgerrit | Jim Branen proposed a change to openstack/requirements: Allow use of hp3parclient 2.0 https://review.openstack.org/42991 | 00:23 |
| dkranz | clarkb: Did something happen to the grizzly/folsom bitrot jobs? | 00:24 |
| dkranz | clarkb: I don't see them in jenkins. | 00:24 |
| jeblair | dkranz: they've been moved to jenkins01 and jenkins02 | 00:24 |
| dkranz | jeblair: OK, thanks :) | 00:25 |
| jeblair | dkranz: if you want to see the latest runs, you'd have to look at both of them and combine (sorry) | 00:25 |
| jeblair | dkranz: but they do still email | 00:25 |
| dkranz | jeblair: OK. I'm looking at a recent stable/grizzly failure. | 00:25 |
| clarkb | ugh modem wont power up | 00:27 |
| jeblair | clarkb: maybe it's a sign that you should spend the evening on the couch without doing code review :) | 00:27 |
| pleia2 | +1 | 00:28 |
| clarkb | maybe. I think the heat may have done bad things to it | 00:28 |
| dkranz | jeblair: Well, it seems stable/grizzly is broken for real. Saw the same error I was chasing in the latest failure. | 00:28 |
| jeblair | dkranz: are you on openstack-stable-maint? | 00:29 |
| jeblair | mailing list | 00:29 |
| dkranz | jeblair: No | 00:29 |
| jeblair | dkranz: i think there was a thread | 00:29 |
| dkranz | jeblair: OK, I'll look. Thanks. | 00:29 |
| jeblair | dkranz: http://lists.openstack.org/pipermail/openstack-stable-maint/2013-August/001188.html | 00:29 |
| *** ftcjeff has joined #openstack-infra | 00:30 | |
| jeblair | dkranz: morganfainberg may be able to provide the latest on keystoneclient | 00:30 |
| *** HenryG has joined #openstack-infra | 00:32 | |
| *** sarob has quit IRC | 00:35 | |
| *** sarob has joined #openstack-infra | 00:35 | |
| morganfainberg | dkranz: this is a tempest test? | 00:35 |
| *** dims has joined #openstack-infra | 00:35 | |
| morganfainberg | jeblair: question to you as well. | 00:36 |
| *** nati_ueno has quit IRC | 00:36 | |
| *** dina_belova has joined #openstack-infra | 00:37 | |
| *** sarob has quit IRC | 00:38 | |
| jeblair | morganfainberg: ? | 00:38 |
| morganfainberg | the bitrot keystone error | 00:39 |
| morganfainberg | dkranz: was talking about | 00:39 |
| morganfainberg | is that tempest? | 00:39 |
| morganfainberg | trying to figure out if there is something i can do to see what changed. | 00:40 |
| *** mrodden has joined #openstack-infra | 00:41 | |
| *** fifieldt has joined #openstack-infra | 00:41 | |
| jeblair | morganfainberg: oh i don't know, sorry. | 00:41 |
| morganfainberg | jeblair: ah, no worries then. | 00:41 |
| *** sarob has joined #openstack-infra | 00:42 | |
| *** dina_belova has quit IRC | 00:42 | |
| morganfainberg | jeblair: i'm just trying to see what is different so i can reproduce it identify what changed. it might be an issue with the client, it might be a slight drift that the test is incorrectly identifying. | 00:42 |
| morganfainberg | but no real impact. | 00:42 |
| morganfainberg | i'll wait for dkranz. | 00:43 |
| jeblair | morganfainberg: did you see that ml thread i linked? it described a failure, but i don't know if it was the same as what dkranz is seeing. | 00:43 |
| morganfainberg | yeah it might also be related to the netaddr issue | 00:43 |
| *** pcrews has joined #openstack-infra | 00:44 | |
| *** mrodden1 has quit IRC | 00:44 | |
| *** lyz_ has quit IRC | 00:44 | |
| *** lyz has joined #openstack-infra | 00:45 | |
| *** sarob has quit IRC | 00:47 | |
| fifieldt | nice post jeblair | 00:52 |
| fifieldt | beers for #openstack-infra whenever we next meet | 00:52 |
| *** sdake_ has joined #openstack-infra | 00:52 | |
| *** sdake_ has quit IRC | 00:52 | |
| *** sdake_ has joined #openstack-infra | 00:52 | |
| jeblair | fifieldt: thanks! | 00:53 |
| *** svarnau has quit IRC | 00:57 | |
| anteaya | i had no idea until now that nodepool was the change that has us ddosing our gerrit git repos | 00:58 |
| anteaya | love the retrospective emails | 00:58 |
| clarkb | I agree with fifieldt. | 00:59 |
| clarkb | modem is still dead :( I guess tonight will be a relaxing evening | 00:59 |
| anteaya | enjoy your evening clarkb | 01:00 |
| anteaya | and yay, nodepool for being so effective | 01:00 |
| jesusaurus | ls | 01:00 |
| anteaya | I got nothing | 01:02 |
| anteaya | s/has/had | 01:04 |
| clarkb | jeblair you shouldn't work too hard either | 01:06 |
| * clarkb finds a video game to play | 01:06 | |
| *** sdake_ has quit IRC | 01:08 | |
| anteaya | I'm going to sleep | 01:08 |
| anteaya | see y'all tomorrow | 01:08 |
| *** anteaya has quit IRC | 01:08 | |
| pleia2 | have a good night, anteaya | 01:08 |
| *** nati_ueno has joined #openstack-infra | 01:09 | |
| *** changbl has joined #openstack-infra | 01:11 | |
| fungi | jeblair: saw your notes about the d-g refactoring/simplification. i can play around with redrafting the documentation to deal with it, certainly | 01:12 |
| fungi | i'd probably go with option A for now | 01:13 |
| * fungi has to disappear again. i'm being kept to a tight sightseeing schedule, but hope to have some time to pick back up reviews and catch up on scrollback after dinner while packing for the boat | 01:13 | |
| *** sdake_ has joined #openstack-infra | 01:14 | |
| SpamapS | Any reason the gate hasn't picked up https://review.openstack.org/#/c/42995/ ? | 01:14 |
| clarkb | SpamapS: it was probably approved during one of the zuul restarts. go ahead and reverify | 01:18 |
| SpamapS | clarkb: ty | 01:19 |
| * SpamapS does not want to burden the queue any more than it needs to be burdened :) | 01:19 | |
| *** CaptTofu has quit IRC | 01:19 | |
| SpamapS | clarkb: note that there should be a cron job that looks for approved reviews that are not in any zuul queue | 01:20 |
| clarkb | its kinda hard to get it right | 01:20 |
| clarkb | gerrits search mechanism doesnt quite cover all of the bases you need to check | 01:20 |
| clarkb | the queue should be mostly unburdened at this point too. it is long but not derpy anymore | 01:21 |
| *** gyee has quit IRC | 01:25 | |
| *** nati_ueno has quit IRC | 01:34 | |
| *** sdake_ has quit IRC | 01:37 | |
| *** dina_belova has joined #openstack-infra | 01:37 | |
| *** dina_belova has quit IRC | 01:42 | |
| mriedem | jeblair: just read the note on the multi-master jenkins, great job (to the entire team)! mrodden and i were just talking last week about how nice it would be to have a multi-master jenkins. | 01:43 |
| mriedem | clarkb: ^ thumbs up | 01:44 |
| mriedem | no one ever appreciates build/CI until it doesn't work | 01:44 |
| erfanian | totally lurking here, but multi-master sounds like high availability ... Did you guys create that functionality in Jenkins now? | 01:45 |
| clarkb | its semi HA | 01:46 |
| clarkb | we can do rolling upgrades and the like | 01:46 |
| clarkb | but we lose slave capacity if one master goes down | 01:46 |
| clarkb | and zuul is still a spof | 01:47 |
| erfanian | Anyone have a handy link to the (I'm assuming) mailing list note? | 01:47 |
| clarkb | yeah I can dig one up | 01:47 |
| *** sarob has joined #openstack-infra | 01:47 | |
| erfanian | clarkb: Thanks. Taking out all the spofs takes time, this is a really neat step though :) | 01:48 |
| erfanian | At least from how it has been described. | 01:48 |
| clarkb | http://lists.openstack.org/pipermail/openstack-dev/2013-August/014148.html | 01:49 |
| *** psedlak has quit IRC | 01:49 | |
| *** psedlak_ has joined #openstack-infra | 01:49 | |
| clarkb | erfanian: basically we have a gearman plugin for jenkins allowing multiple jenkins to connect to that bus and run tests | 01:49 |
| clarkb | gearman is already horizontally scalable | 01:50 |
| erfanian | What else are you getting from gearman besides HA? | 01:51 |
| clarkb | simplicity and built in job semantics | 01:51 |
| *** sarob has quit IRC | 01:52 | |
| *** sarob has joined #openstack-infra | 01:52 | |
| clarkb | percentage complete, success/fail/exception, and compatibilty with arbitrary clients and workers | 01:52 |
| *** CaptTofu has joined #openstack-infra | 01:52 | |
| clarkb | you don't need to run jenkins or zuul. you can slap other things on either end of gearman | 01:53 |
| HenryG | Hi, what does "Hit by Torpedo" mean? | 01:53 |
| clarkb | thats a smokestack failure message iirc. you will have to look at the logs | 01:53 |
| clarkb | erfanian: we really like gearman | 01:53 |
| clarkb | so much simpler than amqp or celery or building something on 0mq | 01:54 |
| *** sarob has quit IRC | 01:54 | |
| *** sarob_ has joined #openstack-infra | 01:54 | |
| *** UtahDave has quit IRC | 01:56 | |
| erfanian | https://github.com/openstack-infra/gear the right spot for gerrit -> gearman integration? | 01:56 |
| *** prad has joined #openstack-infra | 01:57 | |
| mgagne | HenryG: https://github.com/dprince/smokestack/blob/master/app/templates/puppet_runner.sh.erb It means torpedo tests failed. | 01:57 |
| *** prad has quit IRC | 01:57 | |
| mgagne | HenryG: Torpedo itself: https://github.com/dprince/torpedo | 01:57 |
| clarkb | openstack-infra/zuul does the gerrit to gearman for us | 01:57 |
| erfanian | thanks | 01:57 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add some log lines https://review.openstack.org/43221 | 01:58 |
| clarkb | gear is a gearman lib for python that jim wrote because the existing alternatives were not great | 01:58 |
| clarkb | zuul uses gear | 01:58 |
| *** wenlock_ has joined #openstack-infra | 02:00 | |
| erfanian | What else does zuul/gearman dispatch jobs to right now besides Jenkins? | 02:01 |
| clarkb | just jenkins today | 02:01 |
| erfanian | So how does zuul/gearman keep merge deps in check when the tests are batched out across the nodes? | 02:06 |
| erfanian | Well I suppose if it all comes back to zuul, zuul would know.... | 02:07 |
| mgagne | erfanian: You can read about how it's done here: http://ci.openstack.org/zuul/gating.html | 02:07 |
| *** yaguang has joined #openstack-infra | 02:08 | |
| clarkb | yes zuul creates refs with the proper content and passes ref info to the workers which then fetch the refs from git | 02:08 |
| erfanian | This is very exciting stuff :) | 02:12 |
| *** ftcjeff has quit IRC | 02:13 | |
| *** jamiem has quit IRC | 02:13 | |
| *** mriedem has quit IRC | 02:21 | |
| *** yaguang has quit IRC | 02:23 | |
| jgriffith | jeblair: have a second to look at something a bit concerning? | 02:31 |
| jgriffith | https://review.openstack.org/#/c/43202/ | 02:31 |
| jgriffith | jeblair: runs clean pep8, fails neutron, recheck, no patch update etc, then fails pep8 | 02:31 |
| *** emagana has quit IRC | 02:37 | |
| *** erfanian has quit IRC | 02:38 | |
| *** dina_belova has joined #openstack-infra | 02:38 | |
| *** melwitt has quit IRC | 02:42 | |
| *** dina_belova has quit IRC | 02:43 | |
| *** gyee has joined #openstack-infra | 02:49 | |
| *** CaptTofu has quit IRC | 02:51 | |
| *** rcleere has joined #openstack-infra | 02:54 | |
| *** xchu has joined #openstack-infra | 02:58 | |
| *** amotoki has joined #openstack-infra | 03:05 | |
| *** dkliban has quit IRC | 03:06 | |
| *** gyee has quit IRC | 03:08 | |
| *** Ryan_Lane has joined #openstack-infra | 03:10 | |
| *** DennyZhang has joined #openstack-infra | 03:11 | |
| dkehn | does the "recheck no bug" still work? | 03:23 |
| jesusaurus | yeah | 03:24 |
| *** michchap_ has joined #openstack-infra | 03:27 | |
| *** michchap has quit IRC | 03:29 | |
| dkehn | jesusaurus, yepper, just requires a little more patiences | 03:29 |
| *** dina_belova has joined #openstack-infra | 03:39 | |
| *** dina_belova has quit IRC | 03:44 | |
| *** yaguang has joined #openstack-infra | 03:44 | |
| *** psedlak_ has quit IRC | 03:52 | |
| *** vogxn has joined #openstack-infra | 04:00 | |
| *** reed has quit IRC | 04:16 | |
| *** DennyZhang has quit IRC | 04:19 | |
| clarkb | jgriffith: did hacking rule changes mergein between those two test runs? we do rebase atop master or $branch when running check tests | 04:23 |
| *** psedlak has joined #openstack-infra | 04:24 | |
| *** michchap_ has quit IRC | 04:35 | |
| *** michchap has joined #openstack-infra | 04:36 | |
| *** rahmu has quit IRC | 04:38 | |
| *** dina_belova has joined #openstack-infra | 04:39 | |
| *** rahmu has joined #openstack-infra | 04:40 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Existing plugins improvements https://review.openstack.org/41375 | 04:41 |
| *** boris-42 has joined #openstack-infra | 04:42 | |
| *** dina_belova has quit IRC | 04:44 | |
| *** afazekas is now known as __afazekas | 04:48 | |
| *** Dr01d has joined #openstack-infra | 04:49 | |
| *** CaptTofu has joined #openstack-infra | 04:51 | |
| *** rcleere has quit IRC | 04:54 | |
| *** CaptTofu has quit IRC | 04:56 | |
| *** sarob_ has quit IRC | 05:06 | |
| *** sarob has joined #openstack-infra | 05:07 | |
| koolhead17 | clarkb: i will be in portland on 6th sep. lets meet for beer | 05:09 |
| *** yonglihe_ has quit IRC | 05:09 | |
| clarkb | koolhead17 I am actually in seattle. will you be making it to seattle? | 05:11 |
| koolhead17 | clarkb: i will be in seattle on 7th | 05:11 |
| *** sarob has quit IRC | 05:11 | |
| koolhead17 | i remember last time we met you were based in portland | 05:12 |
| koolhead17 | :D | 05:12 |
| clarkb | maybe I did live there just over a year ago | 05:13 |
| clarkb | and I do spend a lot of time there | 05:13 |
| clarkb | seattleites hate hearing this but portland is pretty cool | 05:13 |
| fungi | fewer beards and skinny jeans? | 05:14 |
| *** AJaeger has joined #openstack-infra | 05:14 | |
| *** AJaeger has joined #openstack-infra | 05:14 | |
| clarkb | probably more of that in portland | 05:15 |
| *** nicedice_ has quit IRC | 05:16 | |
| *** pcrews has quit IRC | 05:16 | |
| clarkb | seattle has more suits and coffee. portland has more birkenstocks and beer | 05:17 |
| fungi | goats | 05:17 |
| * fungi wears birkenstocks and drinks beer... thinks clarkb is talking about him | 05:18 | |
| fungi | portland is quite a nifty place, agreed | 05:18 |
| clarkb | both places are and for some reason there is a weird rivalry between them | 05:19 |
| clarkb | being awesome together is less hip I suppose | 05:19 |
| koolhead17 | fungi: are you in seattle as well? | 05:20 |
| koolhead17 | mordred: around? | 05:20 |
| fungi | koolhead17: i've been visiting seattle this week, but am leaving tomorroq | 05:21 |
| fungi | tomorrow | 05:21 |
| clarkb | koolhead17 mordred is out on the nevada playa | 05:21 |
| koolhead17 | ooh ok. | 05:21 |
| clarkb | without internets for a couple weeks | 05:21 |
| koolhead17 | clarkb: w00t really. mordred without internet | 05:21 |
| pleia2 | does it every year :) | 05:22 |
| pleia2 | koolhead17: hope you didn't go *inside* the hotel california | 05:22 |
| fungi | some year soon, he'll build a new internet there and connect all the tents together in harmony | 05:22 |
| *** afazekas has joined #openstack-infra | 05:23 | |
| koolhead17 | pleia2: NOPES. i went to the awesome chinese food joint opposite to it | 05:23 |
| koolhead17 | :D | 05:23 |
| pleia2 | :) | 05:23 |
| clarkb | fungi now those californians like pleia2 and jeblair the NW should have a rivalry with them :) | 05:23 |
| koolhead17 | am addicted to that shop | 05:23 |
| fungi | clarkb: they have enough problems in california to bother adding unnecessary additional rivalries | 05:24 |
| pleia2 | haha | 05:24 |
| *** AJaeger has quit IRC | 05:26 | |
| koolhead17 | pleia2: lets meetup for lunch on sunday if you are free | 05:26 |
| pleia2 | koolhead17: weekends are bad for me (family time), perhaps dinner some time next week? | 05:26 |
| *** vogxn has quit IRC | 05:26 | |
| koolhead17 | ok. let me ping you about it | 05:27 |
| pleia2 | sounds good :) | 05:27 |
| clarkb | driving to california is weird. there is some point south of shasta where you feel like you have entered another universe | 05:28 |
| clarkb | suddenly the hills and trees turn into brown shrubbery and strip malls | 05:28 |
| clarkb | oh and you have to pump your own gas again | 05:29 |
| koolhead17 | clarkb: you don`t like CA too much do you | 05:33 |
| koolhead17 | i like this weather though :D | 05:33 |
| clarkb | It is nice to visit. and san francisco is nice. But there is a big stretch of road I dont need to see again | 05:34 |
| clarkb | we drove 101 on the way back despite not really having time. the redwoods are much nicer than I5 | 05:35 |
| *** DennyZhang has joined #openstack-infra | 05:36 | |
| koolhead17 | well traffic inside city is mess i agree and u have to spend more time in finding place 4 parking rather eting | 05:37 |
| koolhead17 | :D | 05:37 |
| clarkb | ha | 05:37 |
| clarkb | but I was trying to comment at how drastically everything changes. southern oregon and much of far northern CA are very similar. eventually you reach a point where you know you are not in kansas^H^H^H^H^H^Horegon anymore | 05:39 |
| *** sridevi has joined #openstack-infra | 05:40 | |
| *** vogxn has joined #openstack-infra | 05:40 | |
| *** dina_belova has joined #openstack-infra | 05:40 | |
| clarkb | they even pumped my gas for me in crescent city. I dont think they trusted me with my oregon plates to figure it out :) | 05:42 |
| pleia2 | haha | 05:43 |
| koolhead17 | lol | 05:43 |
| *** dina_belova has quit IRC | 05:44 | |
| *** thomasbiege has joined #openstack-infra | 05:49 | |
| *** DennyZhang has quit IRC | 05:51 | |
| psedlak | anyone knows what can be behind 'ERROR:root:Could not find any typelib for GnomeKeyring' and how that can be fixed on stable/grizzly check,gate? | 05:52 |
| *** thomasbiege has quit IRC | 05:54 | |
| *** basha has joined #openstack-infra | 05:57 | |
| basha | Hi anyone around? | 05:57 |
| *** xchu has quit IRC | 05:57 | |
| clarkb | psedlak: I think dkranz and morganfainberg were looking at it | 05:58 |
| clarkb | basha: yes, though I am on a phone and it is late | 05:58 |
| basha | Sorry I'll be quick | 05:58 |
| *** nayward has joined #openstack-infra | 05:58 | |
| basha | Have an issue with https://review.openstack.org/#/c/37411/ | 05:58 |
| basha | I had made changes to sample-nova.conf and got jenkins and smoke stack and everything pass | 05:59 |
| basha | Then I renamed the CONF value I'd added and now jenkins fails for a pep8 issue | 05:59 |
| morganfainberg | clarkb: not sure if that is the one that we've identified. | 05:59 |
| basha | the log I receive says its failing for "error running configure.sh" | 05:59 |
| morganfainberg | clarkb: i know we have https://review.openstack.org/#/c/43402/ | 06:00 |
| basha | It looks like a known bug. but Im not sure how to tackle this, clarkb | 06:00 |
| clarkb | basha: it tells you near the end of the log output one line above the ERROR | 06:01 |
| basha | clarkb: yes exactly, i can run it on a devstack but that would remove a lot of lines existing in the current latest version of the sample-conf file which I don't want to do | 06:02 |
| clarkb | basha: there is apparently a script to run to sync up the conf file | 06:03 |
| clarkb | basha: sounds like you need to rebase? | 06:03 |
| basha | clarkb: is it the one they mention here? https://bugs.launchpad.net/nova/+bug/1204762 | 06:03 |
| uvirtbot | Launchpad bug 1204762 in nova "Can't generate sample configuration file by generate_sample.sh (dup-of: 1204204)" [Undecided,New] | 06:03 |
| uvirtbot | Launchpad bug 1204204 in oslo "./tools/config/generate_sample.sh requires a -b parameter" [Wishlist,In progress] | 06:03 |
| basha | clarkb:no i pulled in the latest code | 06:03 |
| basha | clarkb: if you look at the recent checked in code, it has the same CONF | 06:04 |
| clarkb | did you run generate_sample.sh after rebasing on master? | 06:05 |
| clarkb | I believe that is what needs to be done to have that check pass. the check tests will merge your code against master before running | 06:06 |
| psedlak | morganfainberg: https://bugs.launchpad.net/devstack/+bug/1193164 is reason for https://bugs.launchpad.net/tempest/+bug/1213912 ... reason why the verification failed on the review you just linked to | 06:07 |
| uvirtbot | Launchpad bug 1193164 in devstack "GnomeKeyring errors when installing devstack" [Undecided,New] | 06:07 |
| morganfainberg | psedlak: i'll look at those in the morning. | 06:08 |
| * psedlak has the morning now ... :) | 06:08 | |
| basha | clarkb: I ran the script but it changed the CONF and removed a lot of lines. I wouldn't be able to remove checked in code and submit for review | 06:08 |
| psedlak | morganfainberg: simply put that GnomeKeyring line 'messes' output validation in tempest.cli.keystone tests | 06:09 |
| *** emagana has joined #openstack-infra | 06:09 | |
| morganfainberg | psedlak: joy. | 06:09 |
| morganfainberg | we have 2 or 3 issues then to cleanup in bitrot checks. | 06:09 |
| morganfainberg | it looks like | 06:10 |
| *** xchu has joined #openstack-infra | 06:10 | |
| clarkb | basha I don't know then. you may have luck asking the nova devs as this is a nova tool | 06:10 |
| *** sridevi has quit IRC | 06:10 | |
| basha | Ok Thanks clarkb :) | 06:12 |
| *** dmakogon_ has joined #openstack-infra | 06:15 | |
| *** dmakogon_ has quit IRC | 06:17 | |
| *** p5ntangle has joined #openstack-infra | 06:25 | |
| *** wenlock_ has quit IRC | 06:26 | |
| *** jhesketh has quit IRC | 06:27 | |
| *** vogxn has quit IRC | 06:29 | |
| *** mikal has quit IRC | 06:30 | |
| *** dstufft has quit IRC | 06:31 | |
| *** mikal has joined #openstack-infra | 06:32 | |
| *** yolanda has joined #openstack-infra | 06:32 | |
| *** vogxn has joined #openstack-infra | 06:33 | |
| *** romero has joined #openstack-infra | 06:34 | |
| *** morganfainberg is now known as morganfainberg|a | 06:35 | |
| *** dstufft has joined #openstack-infra | 06:37 | |
| *** dina_belova has joined #openstack-infra | 06:40 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:42 | |
| *** dina_belova has quit IRC | 06:43 | |
| *** llu has joined #openstack-infra | 06:45 | |
| *** romero has left #openstack-infra | 06:46 | |
| *** morganfainberg|a is now known as morganfainberg | 06:49 | |
| *** p5ntangle has quit IRC | 06:49 | |
| *** Dr01d has quit IRC | 06:49 | |
| *** Dr01d has joined #openstack-infra | 06:50 | |
| llu | any one know why it seems that "recheck no bug" doesn't work anymore? https://review.openstack.org/#/c/35765/ | 06:55 |
| *** jkt has quit IRC | 06:57 | |
| *** jkt has joined #openstack-infra | 06:57 | |
| *** jkt has joined #openstack-infra | 06:57 | |
| *** SergeyLukjanov has quit IRC | 06:58 | |
| *** AJaeger has joined #openstack-infra | 06:59 | |
| *** AJaeger has joined #openstack-infra | 06:59 | |
| llu | ok, it works now. | 07:00 |
| *** sgviking has quit IRC | 07:04 | |
| *** xBsd has joined #openstack-infra | 07:05 | |
| *** yongli has joined #openstack-infra | 07:07 | |
| *** BobBallAway is now known as BobBall | 07:09 | |
| *** AJaeger has quit IRC | 07:12 | |
| *** basha has quit IRC | 07:13 | |
| *** basha has joined #openstack-infra | 07:15 | |
| *** AJaeger has joined #openstack-infra | 07:23 | |
| *** arzadr has joined #openstack-infra | 07:25 | |
| *** SlickNik has quit IRC | 07:33 | |
| *** AJaeger has quit IRC | 07:33 | |
| *** SlickNik has joined #openstack-infra | 07:34 | |
| *** vogxn has quit IRC | 07:34 | |
| *** odyssey4me4 has joined #openstack-infra | 07:37 | |
| *** dina_belova has joined #openstack-infra | 07:41 | |
| *** shardy_afk is now known as shardy | 07:41 | |
| *** ruhe has joined #openstack-infra | 07:44 | |
| *** AJaeger has joined #openstack-infra | 07:45 | |
| *** AJaeger has joined #openstack-infra | 07:45 | |
| *** dina_belova has quit IRC | 07:45 | |
| *** pblaho has joined #openstack-infra | 07:48 | |
| *** emagana_ has joined #openstack-infra | 07:50 | |
| *** emagana has quit IRC | 07:53 | |
| *** vogxn has joined #openstack-infra | 07:56 | |
| *** p5ntangle has joined #openstack-infra | 07:59 | |
| *** fbo_away is now known as fbo | 08:02 | |
| *** sjing has joined #openstack-infra | 08:13 | |
| *** arzadr has quit IRC | 08:14 | |
| *** arzadr has joined #openstack-infra | 08:14 | |
| *** jpich has joined #openstack-infra | 08:16 | |
| *** psedlak has quit IRC | 08:29 | |
| *** thomasbiege has joined #openstack-infra | 08:32 | |
| *** sjing has quit IRC | 08:34 | |
| *** thomasbiege has quit IRC | 08:37 | |
| *** basha has quit IRC | 08:39 | |
| *** psedlak has joined #openstack-infra | 08:39 | |
| *** basha has joined #openstack-infra | 08:40 | |
| *** nayward has quit IRC | 08:41 | |
| *** dina_belova has joined #openstack-infra | 08:42 | |
| *** psedlak has quit IRC | 08:43 | |
| *** psedlak has joined #openstack-infra | 08:43 | |
| *** dina_belova has quit IRC | 08:46 | |
| *** AndChat|235809 has joined #openstack-infra | 08:49 | |
| *** arzadr has quit IRC | 08:49 | |
| *** AndChat-235809 has joined #openstack-infra | 08:50 | |
| *** AndChat|235809 has quit IRC | 08:50 | |
| *** ruhe has quit IRC | 08:51 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:51 | |
| *** dina_belova has joined #openstack-infra | 08:52 | |
| *** fifieldt has quit IRC | 08:55 | |
| *** p5ntangle has quit IRC | 08:58 | |
| *** ruhe has joined #openstack-infra | 09:00 | |
| *** AndChat-235809 has quit IRC | 09:05 | |
| *** p5ntangle has joined #openstack-infra | 09:09 | |
| *** xchu has quit IRC | 09:09 | |
| *** yongli has quit IRC | 09:19 | |
| *** p5ntangle has quit IRC | 09:19 | |
| *** p5ntangle has joined #openstack-infra | 09:22 | |
| *** xchu has joined #openstack-infra | 09:22 | |
| *** jfriedly has quit IRC | 09:25 | |
| *** xBsd has quit IRC | 09:29 | |
| *** ruhe has quit IRC | 09:30 | |
| *** xBsd has joined #openstack-infra | 09:31 | |
| *** p5ntangle has quit IRC | 09:32 | |
| *** johnthetubaguy has joined #openstack-infra | 09:38 | |
| *** vogxn has quit IRC | 09:38 | |
| *** thomasbiege has joined #openstack-infra | 09:40 | |
| *** thomasbiege has quit IRC | 09:42 | |
| *** p5ntangle has joined #openstack-infra | 09:42 | |
| *** matel has joined #openstack-infra | 09:47 | |
| matel | Hi guys. | 09:47 |
| *** p5ntangle has quit IRC | 09:48 | |
| *** emagana_ has quit IRC | 09:49 | |
| *** p5ntangle has joined #openstack-infra | 09:49 | |
| *** sdake_ has joined #openstack-infra | 09:51 | |
| *** AJaeger has quit IRC | 09:52 | |
| *** morganfainberg is now known as morganfainberg|a | 09:53 | |
| *** pcm_ has joined #openstack-infra | 10:03 | |
| *** pcm_ has quit IRC | 10:04 | |
| *** pcm_ has joined #openstack-infra | 10:04 | |
| *** vogxn has joined #openstack-infra | 10:05 | |
| *** arezadr has quit IRC | 10:16 | |
| *** arezadr has joined #openstack-infra | 10:20 | |
| *** xchu has quit IRC | 10:23 | |
| *** llu has left #openstack-infra | 10:34 | |
| *** basha has quit IRC | 10:37 | |
| *** sdake_ has quit IRC | 10:41 | |
| *** juice has quit IRC | 10:41 | |
| *** AJaeger has joined #openstack-infra | 10:42 | |
| *** AJaeger has joined #openstack-infra | 10:42 | |
| *** nayward has joined #openstack-infra | 10:47 | |
| *** xBsd has quit IRC | 10:49 | |
| *** xBsd has joined #openstack-infra | 10:51 | |
| *** juice has joined #openstack-infra | 10:53 | |
| *** jjmb has quit IRC | 10:55 | |
| *** emagana has joined #openstack-infra | 10:55 | |
| *** yaguang has quit IRC | 10:56 | |
| *** dina_belova has quit IRC | 10:56 | |
| *** AJaeger has quit IRC | 11:00 | |
| *** dina_belova has joined #openstack-infra | 11:02 | |
| *** SergeyLukjanov has quit IRC | 11:04 | |
| *** dina_belova has quit IRC | 11:06 | |
| *** emagana has quit IRC | 11:09 | |
| *** nayward has quit IRC | 11:12 | |
| *** nayward has joined #openstack-infra | 11:12 | |
| *** alexpilotti has quit IRC | 11:13 | |
| *** AJaeger has joined #openstack-infra | 11:14 | |
| *** michchap has quit IRC | 11:16 | |
| *** AJaeger has quit IRC | 11:20 | |
| *** basha has joined #openstack-infra | 11:27 | |
| *** AJaeger has joined #openstack-infra | 11:29 | |
| *** AJaeger has joined #openstack-infra | 11:29 | |
| *** amotoki has quit IRC | 11:30 | |
| *** CaptTofu has joined #openstack-infra | 11:30 | |
| *** basha has quit IRC | 11:33 | |
| *** AJaeger has quit IRC | 11:35 | |
| *** vogxn has quit IRC | 11:35 | |
| *** nayward has quit IRC | 11:49 | |
| *** anteaya has joined #openstack-infra | 11:49 | |
| *** thomasbiege has joined #openstack-infra | 11:51 | |
| *** thomasbiege has quit IRC | 11:52 | |
| *** markmc has joined #openstack-infra | 11:54 | |
| *** nayward has joined #openstack-infra | 11:55 | |
| anteaya | morning | 11:56 |
| anteaya | git.o.o cpus look good, merges happened all night (or appropriate time of day for you) http://graphite.openstack.org/graphlot/?width=586&height=308&_salt=1377258976.281&target=stats.gerrit.event.change-merged | 11:57 |
| *** dkliban has joined #openstack-infra | 11:58 | |
| anteaya | zuul queue length has 0 events | 11:58 |
| anteaya | gate 23, post 4, check 23 | 11:58 |
| anteaya | smooth | 11:58 |
| ttx | all hail Fridays | 12:01 |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding AnsiColor Support https://review.openstack.org/43121 | 12:01 |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding AnsiColor Support https://review.openstack.org/43121 | 12:02 |
| anteaya | oh yeah, Friday | 12:02 |
| ttx | anteaya: any hint why https://review.openstack.org/#/c/43281/ and https://review.openstack.org/#/c/43296/ did not get tested yet ? | 12:03 |
| * anteaya looks | 12:03 | |
| ttx | I don't want to APRV until the check tests pass but it's been a day now | 12:03 |
| ttx | might be some stable/* weirdness | 12:04 |
| Alex_Gaynor | ttx: I think it's more likely they were just lost in the zuul restarts, `recheck no bug` should fix | 12:04 |
| anteaya | ttx nothing I can see | 12:04 |
| *** dina_belova has joined #openstack-infra | 12:04 | |
| anteaya | I'll add the recheck to them | 12:04 |
| *** SergeyLukjanov has joined #openstack-infra | 12:05 | |
| ttx | Alex_Gaynor: that's definitely a possibility. And they would not have been restarted manually because they are not on master | 12:05 |
| ttx | Alex_Gaynor: great to have you around, btw :) | 12:05 |
| Alex_Gaynor | I'd say I'm glad to be around, but it's 5AM here, and I sort of wish I were back in bed :) | 12:06 |
| ttx | Moar PSF members | 12:06 |
| ttx | Alex_Gaynor: you need to lose that sleeping habit if you want to strive around here :P | 12:06 |
| *** CaptTofu has quit IRC | 12:06 | |
| Alex_Gaynor | :D | 12:06 |
| anteaya | zuul queue status has 2 events, I'll give them a minute | 12:06 |
| Alex_Gaynor | anteaya: I've seen it get as far as ~30 behind, but it's always caught up | 12:07 |
| *** shadower has joined #openstack-infra | 12:07 | |
| anteaya | Alex_Gaynor: great thank you, I was wondering what you had been seeing | 12:07 |
| anteaya | ttx in the check queue | 12:07 |
| *** AJaeger has joined #openstack-infra | 12:09 | |
| *** AJaeger has joined #openstack-infra | 12:09 | |
| *** apcruz has joined #openstack-infra | 12:10 | |
| *** dkliban has quit IRC | 12:13 | |
| *** CaptTofu has joined #openstack-infra | 12:16 | |
| ttx | jeblair: Benefit #18472 of time-based releases: periodically stress beyond reason the development infrastructure in order to make it better ! | 12:17 |
| markmc | ttx, if mordred was here, I'd assume you were just trolling for Friday afternoon fun | 12:20 |
| *** zul has joined #openstack-infra | 12:24 | |
| *** AJaeger has quit IRC | 12:25 | |
| *** ruhe has joined #openstack-infra | 12:30 | |
| *** sgviking has joined #openstack-infra | 12:34 | |
| anteaya | 14 almost successful patches finishing in the gate - yay! | 12:34 |
| *** dkliban has joined #openstack-infra | 12:36 | |
| markmc | and the failure of the 15th is https://bugs.launchpad.net/neutron/+bug/1211915 | 12:38 |
| uvirtbot | Launchpad bug 1211915 in neutron "Connection to neutron failed: Maximum attempts reached" [Undecided,New] | 12:38 |
| markmc | and that's the 4th of my series of 10 patches | 12:38 |
| * markmc curls up and weeps quietly | 12:39 | |
| markmc | in other news, I filed | 12:39 |
| markmc | https://bugs.launchpad.net/openstack-ci/+bug/1215758 | 12:39 |
| uvirtbot | Launchpad bug 1215758 in openstack-ci "Allow gerrit to automatically resolve conflicts" [Undecided,New] | 12:39 |
| markmc | and https://bugs.launchpad.net/openstack-ci/+bug/1215768 | 12:39 |
| uvirtbot | Launchpad bug 1215768 in openstack-ci "Merge a patch series in one fell swoop, rather than merging individual commits" [Undecided,New] | 12:39 |
| markmc | for the latter | 12:40 |
| markmc | there's really no reason why my series of 29 patches couldn't be merged in one go | 12:40 |
| markmc | rather than batches of 3 or 4 | 12:40 |
| markmc | i.e. every 4th or 5th merge failed for whatever reason and I have to reverify or rebase | 12:41 |
| *** jamiem has joined #openstack-infra | 12:41 | |
| * anteaya pats markmc consolingly | 12:42 | |
| jpich | Bad karma | 12:43 |
| *** weshay has joined #openstack-infra | 12:44 | |
| anteaya | markmc I think the conversation about how to handle difficult merges was discussed last night | 12:44 |
| anteaya | it is entirely possible that there might be room for that discussion on today's agenda | 12:45 |
| markmc | jpich, commits should only be merged if the author can say what the angle in a given equilateral polygon is, within like 10 seconds | 12:45 |
| anteaya | that and tracking down the LOST test logs bug - which doesn't seem to be biting us atm | 12:45 |
| markmc | anteaya, ok | 12:45 |
| * markmc looks back at the scrollback | 12:45 | |
| anteaya | markmc might I suggest around timestamp 2013-08-22T22:52:46 on the 22nd | 12:47 |
| anteaya | as a beginning point | 12:47 |
| markmc | anteaya, thankie | 12:47 |
| anteaya | I hope I haven't red herring'd you | 12:48 |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Update Flask dependency https://review.openstack.org/43452 | 12:50 |
| anteaya | mtreinish: when you are around here are some testr failures on a python-glanceclient patch: https://jenkins02.openstack.org/job/gate-tempest-devstack-vm-testr-full/3746/console | 12:53 |
| markmc | anteaya, yeah, relevant ... but I think the bugs I filed add something new to the discussion | 12:55 |
| anteaya | that they do | 12:55 |
| anteaya | so timing wise the conversation might be good to have today | 12:55 |
| markmc | 1) we seem to be getting merge failures when we should be able to happily resolve those "conflicts" automatically | 12:55 |
| *** CaptTofu has quit IRC | 12:56 | |
| anteaya | and clarkb and jeblair might be in the mood for a bit of a chat about it | 12:56 |
| markmc | 2) a patch series does not really need to be merged one patch at a time | 12:56 |
| markmc | (a patch series does need to be initially checked one patch at a time, though) | 12:56 |
| anteaya | what constitutes a patch "series"? | 12:56 |
| anteaya | meaning what markers can be used by zuul to identify them | 12:57 |
| markmc | where a gerrit review depends on another which depends on another which depends on another | 12:57 |
| anteaya | okay so dependencies | 12:57 |
| markmc | yes | 12:57 |
| anteaya | what happens if they are not in the queue together? | 12:57 |
| anteaya | getting them queue in the gate pipeline as a series might be a bit of a trick | 12:57 |
| anteaya | s/queue/queued | 12:57 |
| markmc | there's a number of different ways of doing it | 12:58 |
| * anteaya listens | 12:58 | |
| markmc | the one I suggested is we see if we can use gerrits ability to review merge commits | 12:58 |
| markmc | i.e. the author of the patch series would propose a merge commit | 12:58 |
| markmc | and it's that commit which would be approved, pulled into the gate, tested and merged | 12:59 |
| markmc | thereby pulling in all the commits | 12:59 |
| anteaya | hmmm, a merge commit | 12:59 |
| markmc | you still want the individual commits to be reviewed and checked | 12:59 |
| markmc | but not approved | 12:59 |
| anteaya | hmmm | 12:59 |
| *** alexpilotti has joined #openstack-infra | 12:59 | |
| markmc | only approve the merge commit that depends on them all | 12:59 |
| markmc | this is not a trivial change, I know | 13:00 |
| markmc | it's a strawman idea | 13:00 |
| anteaya | right | 13:00 |
| anteaya | and an interesting one | 13:00 |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/hacking: Update requirements https://review.openstack.org/43457 | 13:00 |
| anteaya | I would be interesting in hearing lifeless' thoughts on this | 13:00 |
| openstackgerrit | A change was merged to openstack/requirements: Add some more filters to the .gitignore https://review.openstack.org/43216 | 13:00 |
| anteaya | and how it would play a role in continuous deployment | 13:01 |
| anteaya | and small patches | 13:01 |
| anteaya | s/interesting/interested | 13:01 |
| markmc | we have recommended smaller patches for a long time now | 13:01 |
| anteaya | yes | 13:01 |
| markmc | https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes | 13:01 |
| anteaya | how would the idea of the merge commit fit with the direction of continuous deployment? | 13:02 |
| markmc | well | 13:02 |
| markmc | it means the master branch potentially gets bigger chunks of changes merged at once | 13:03 |
| markmc | but the reality is, series of commits tend to get approved together | 13:03 |
| anteaya | um humm | 13:03 |
| markmc | and land together | 13:03 |
| anteaya | okay | 13:03 |
| markmc | it's a fair point, though | 13:03 |
| anteaya | just trying to be a good debater here | 13:04 |
| anteaya | what downsides do you see of the merge commit proposal? | 13:04 |
| anteaya | would it change how we look for issues that got into master? | 13:05 |
| anteaya | the gate has reset - 5 in the gate | 13:06 |
| markmc | the way we merge commits now means you can treat it like we have a linear history | 13:07 |
| markmc | you can walk the commits in the tree as if they were applied sequentially with no merge commits | 13:07 |
| *** pabelanger has quit IRC | 13:07 | |
| markmc | in fact, I have a git alias for that | 13:07 |
| markmc | lg = log --no-merges --topo-order --abbrev-commit | 13:07 |
| markmc | "show me the commits in the order they were merged and elide the merge commits" | 13:08 |
| markmc | allowing people to propose merge commits would confuse that | 13:08 |
| anteaya | okay | 13:08 |
| anteaya | so it would have an effect on how people identify bugs using the git history | 13:09 |
| markmc | e.g. if you bisected a series of commits in this new world | 13:09 |
| *** dkliban has quit IRC | 13:09 | |
| anteaya | or at least identify commits using the git history | 13:09 |
| *** emagana has joined #openstack-infra | 13:09 | |
| markmc | you would hit commits which were never the master tree at any stage | 13:09 |
| anteaya | ah | 13:10 |
| markmc | but it would have been tested, nonetheless | 13:10 |
| markmc | which, for the purposes of git bisect, would still be helpful | 13:10 |
| anteaya | hmmm | 13:11 |
| markmc | you'd know which commit in a series caused a regression | 13:11 |
| *** emagana has quit IRC | 13:11 | |
| markmc | but for CD, it wouldn't make sense to rollback to the commit before the bad commit | 13:11 |
| anteaya | okay, and we don't back out commits anyway, we offer new patches, correct? | 13:11 |
| *** lcestari has joined #openstack-infra | 13:11 | |
| markmc | you'd want to rollback to the merge commit before the one that merged the series | 13:11 |
| anteaya | would you rollback? | 13:12 |
| anteaya | do we do that, I didn't think we did | 13:12 |
| markmc | well, I mean in CD, you'd choose not to deploy that commit | 13:12 |
| *** ruhe has quit IRC | 13:12 | |
| markmc | you'd choose to deploy the merge commit before it | 13:12 |
| openstackgerrit | Tomas Sedovic proposed a change to openstack-infra/config: Add Jenkins jobs for Tuskar https://review.openstack.org/43461 | 13:12 |
| markmc | it would just change how you choose points in the master history to test and deploy | 13:13 |
| anteaya | hmmm, okay so yes I would definitely like to hear lifeless' views on the idea of a merge commit | 13:13 |
| anteaya | so while we are hashing this out, you said there were options plural | 13:14 |
| markmc | I'd like to hear everyone's views :) | 13:14 |
| anteaya | I like that about you | 13:14 |
| anteaya | :D | 13:14 |
| markmc | if you look at 'git log origin/master' | 13:14 |
| markmc | merge commits have two parents | 13:14 |
| anteaya | yes | 13:14 |
| markmc | Merge: bec54ac aa5b658 | 13:14 |
| markmc | if you always pick the left hand one | 13:15 |
| markmc | that's a commit which was HEAD of master at some point | 13:15 |
| markmc | that would still be the rule even if gerrit was merging merge commits | 13:15 |
| anteaya | okay | 13:15 |
| anteaya | so using git bisect would take you to the commit prior to the merge commit with an issue | 13:16 |
| anteaya | so potentially you would rollback and lose a whole series if there was an issue with any part of it | 13:16 |
| markmc | you could actually just get git-bisect to test commits that were HEAD of master at some point | 13:21 |
| markmc | pretty straightforward | 13:21 |
| markmc | if you were just trying to find the latest working HEAD of master commit | 13:21 |
| markmc | "You can even skip a range of commits, instead of just one commit, using the "<commit1>..<commit2>" notation" | 13:22 |
| markmc | from the git-bisect man page | 13:22 |
| *** dina_belova has quit IRC | 13:22 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| anteaya | okay | 13:23 |
| *** dprince has joined #openstack-infra | 13:23 | |
| anteaya | what other directions do you think are worthy of consideration for the merge a series direction? | 13:24 |
| anteaya | ttx https://review.openstack.org/#/c/43296/ passed check and https://review.openstack.org/#/c/43281/ failed <- keystone admin test failures was what I saw | 13:26 |
| markmc | anteaya, https://bugs.launchpad.net/openstack-ci/+bug/902838 | 13:28 |
| uvirtbot | Launchpad bug 902838 in openstack-ci "Gerrit should support reviewing related patchsets (aka "topic reviews")" [Wishlist,Triaged] | 13:28 |
| markmc | anteaya, that gerrit's topic review support would allow us to just merge the last commit in the series | 13:29 |
| markmc | anteaya, similar idea and implications, just different implementation | 13:29 |
| * anteaya reads | 13:29 | |
| *** CaptTofu has joined #openstack-infra | 13:29 | |
| markmc | or that zuul would see multiple changes in a series all approved | 13:29 |
| markmc | and merge the last approved one, rather than each one individually | 13:30 |
| *** SergeyLukjanov has quit IRC | 13:30 | |
| markmc | ultimately, it comes down to reducing the number of merge commits created for a patch series | 13:30 |
| markmc | reducing it to one, if the entire series gets approved together | 13:30 |
| *** thomasbiege has joined #openstack-infra | 13:31 | |
| anteaya | well Brad King and Chris Harris at Kitware seem to be using this approach and feel strongly enough about it that they are advocating for it | 13:32 |
| *** thomasbiege has quit IRC | 13:32 | |
| anteaya | I would be interested to see if they have any additional material outlining how they use it | 13:33 |
| anteaya | I | 13:33 |
| anteaya | 'll go looking | 13:33 |
| markmc | anteaya, well, the gerrit topic review stuff is more about the ability to review patch series in a sane way | 13:34 |
| markmc | anteaya, less about how they are merged | 13:34 |
| anteaya | ah okay | 13:34 |
| anteaya | can't see anything by either of them on the Kitware blog | 13:36 |
| anteaya | so there is the concept of a topic-id | 13:36 |
| anteaya | probably a good way to identify members of a series | 13:37 |
| anteaya | easier than trying to re-enforce a unique topic name | 13:37 |
| *** pabelanger has joined #openstack-infra | 13:38 | |
| anteaya | s/re-enforce/reinforce | 13:38 |
| *** jjmb has joined #openstack-infra | 13:39 | |
| markmc | anteaya, ok, captured all that in https://bugs.launchpad.net/openstack-ci/+bug/1215768 | 13:39 |
| uvirtbot | Launchpad bug 1215768 in openstack-ci "Merge a patch series in one fell swoop, rather than merging individual commits" [Undecided,New] | 13:39 |
| anteaya | nicely done | 13:41 |
| *** woodspa has joined #openstack-infra | 13:43 | |
| *** jjmb1 has joined #openstack-infra | 13:44 | |
| * markmc leaves this here in case it might help anyone | 13:44 | |
| markmc | for iii in $(git log --oneline --pretty=format:%h origin/master..); do ssh review gerrit approve --code-review +2 --approved +1 -m 'trivial\ rebase' $iii; done | 13:44 |
| markmc | (well, substitute 'review' for the review.openstack.org port blah blah) | 13:44 |
| *** jjmb has quit IRC | 13:46 | |
| anteaya | nice | 13:46 |
| *** dkliban has joined #openstack-infra | 13:47 | |
| *** jjmb1 has quit IRC | 13:48 | |
| *** changbl has quit IRC | 13:48 | |
| *** jjmb has joined #openstack-infra | 13:52 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:56 | |
| *** dina_belova has joined #openstack-infra | 13:56 | |
| *** ftcjeff has joined #openstack-infra | 13:57 | |
| *** sgviking has quit IRC | 13:58 | |
| *** sgviking has joined #openstack-infra | 13:59 | |
| *** burt has joined #openstack-infra | 14:03 | |
| *** vogxn has joined #openstack-infra | 14:04 | |
| *** dina_belova has quit IRC | 14:05 | |
| *** dina_belova has joined #openstack-infra | 14:07 | |
| *** jjmb has quit IRC | 14:07 | |
| *** pabelanger has quit IRC | 14:07 | |
| *** sgviking has quit IRC | 14:08 | |
| *** sgviking has joined #openstack-infra | 14:08 | |
| *** pabelanger has joined #openstack-infra | 14:09 | |
| *** ruhe has joined #openstack-infra | 14:15 | |
| *** rnirmal has joined #openstack-infra | 14:19 | |
| *** leifmadsen has joined #openstack-infra | 14:21 | |
| *** afazekas has quit IRC | 14:22 | |
| *** _TheDodd_ has joined #openstack-infra | 14:24 | |
| *** SergeyLukjanov has quit IRC | 14:24 | |
| *** dina_belova has quit IRC | 14:24 | |
| zul | anyone mind if we bump oslo.config in global-requirements its getting a bit long in the tooth | 14:24 |
| *** SergeyLukjanov has joined #openstack-infra | 14:25 | |
| *** dina_belova has joined #openstack-infra | 14:26 | |
| *** vogxn has quit IRC | 14:26 | |
| *** datsun180b has joined #openstack-infra | 14:28 | |
| *** senk has quit IRC | 14:33 | |
| *** thomasbiege has joined #openstack-infra | 14:33 | |
| markmc | zul, ok, I'll do that now | 14:34 |
| zul | markmc: thanks | 14:34 |
| jeblair | markmc: hi there; i responded to bug 121758 | 14:34 |
| uvirtbot | Launchpad bug 121758 in ubuntu "include remove-duplicates plugin (dup-of: 437463)" [Undecided,Invalid] https://launchpad.net/bugs/121758 | 14:34 |
| uvirtbot | Launchpad bug 437463 in ubuntu "[needs packaging] evolution plugin 'delete-duplicates-plugin'" [Wishlist,New] https://launchpad.net/bugs/437463 | 14:34 |
| *** thomasbiege has quit IRC | 14:34 | |
| jeblair | markmc: hi there; i responded to bug 1215758 | 14:34 |
| uvirtbot | Launchpad bug 1215758 in openstack-ci "Allow gerrit to automatically resolve conflicts" [Undecided,Incomplete] https://launchpad.net/bugs/1215758 | 14:34 |
| *** xBsd has quit IRC | 14:34 | |
| jeblair | yeah, that's it :) | 14:34 |
| *** xBsd has joined #openstack-infra | 14:35 | |
| markmc | jeblair, ok, so - "we'll fix this by having zuul do merges" | 14:36 |
| markmc | jeblair, any idea of when? i.e. worth fixing jgit in the meantime? | 14:36 |
| *** thomasbiege has joined #openstack-infra | 14:36 | |
| jeblair | markmc: yep, because that setting is already turned on, and jgit is just different sometimes :/ | 14:36 |
| markmc | jeblair, it's not an issue of jgit being different, in this case it's just wrong AFAICT | 14:37 |
| jeblair | markmc: of the pre-reqs in that bug "* Zuul would need to add the git review notes." is i think the only thing left | 14:37 |
| markmc | jeblair, detecting multiple merge bases where there's only one | 14:37 |
| jeblair | markmc: and that task seems pretty trivial | 14:37 |
| jeblair | markmc: yeah, i guess i was being generous. :) | 14:37 |
| markmc | jeblair, ok | 14:37 |
| jeblair | markmc: i have not checked to see if there is a new jgit that is used in newer gerrit that is better in this regard | 14:38 |
| markmc | jeblair, newer gerrit doesn't use jgit's recursive merger | 14:38 |
| markmc | jeblair, if it did, that might fix it | 14:38 |
| jeblair | how weird | 14:38 |
| jeblair | i thought they were basically developed in tandem at this point | 14:39 |
| jeblair | markmc: anyway, i think we can probably get zuul doing merges this cycle | 14:39 |
| markmc | ok | 14:40 |
| *** thomasbiege has quit IRC | 14:40 | |
| markmc | zul, ok, 1.2.0a4 will show up on tarballs.o.o soon - you go ahead and propose the dev-requirements.txt change? | 14:41 |
| * markmc writes release notes | 14:41 | |
| zul | markmc: thanks | 14:41 |
| david-lyle | jeblair: django-openstack-auth (a horizon dependency) was recently brought in as an openstack managed git repo, what is the process for adding a launchpad page to track bugs/etc for it? | 14:42 |
| jeblair | markmc: as for the other bug; it's definitely worth considering -- i think the current situation is mostly there out of a desire to preserve bisectability; if we decide we want to change that, we can certainly adapt the tools. | 14:42 |
| markmc | jeblair, bisectability is important, I don't want to lose that | 14:43 |
| markmc | jeblair, all individual commits need to be tested | 14:43 |
| markmc | jeblair, but they don't need to be merged individually | 14:43 |
| jd__ | good to see Jenkins working great today thanks guys :) | 14:43 |
| jeblair | markmc: also remember though that merge commits are scary and if we allow anyone to upload them, we may well end up with people accidentally uploading hundreds of commits (though if people are using git-review it should help with that) | 14:44 |
| * markmc can't help being a little grumpy trying to get this series landed | 14:44 | |
| markmc | sorry :) | 14:44 |
| jeblair | markmc: i actually think you're being remarkably patient :) | 14:44 |
| markmc | jeblair, cheers :) | 14:44 |
| markmc | jeblair, yes, merge commits are scarey | 14:44 |
| markmc | jeblair, so maybe the idea of having zuul go "wait, there's a patch series here, let's just do one merge commit" is best | 14:45 |
| markmc | jeblair, that way people would learn to e.g. approve the first commit in the series last | 14:45 |
| anteaya | thanks jd__, we are happy Jenkins is working well today too | 14:45 |
| anteaya | :D | 14:45 |
| markmc | jeblair, hmm, zuul would also need to enforce that the individual commits had passed the check queue already | 14:45 |
| jeblair | markmc: definitely technically feasible; especially after zuul is doing the merges (it would be awkward if zuul were surprised halfway through merging a series that gerrit refused to merge it) | 14:46 |
| markmc | jeblair, which would be new | 14:46 |
| *** changbl has joined #openstack-infra | 14:46 | |
| jeblair | markmc: this is probably a topic to think about for a bit and maybe have a summit conversation about, what do you think? | 14:46 |
| markmc | jeblair, absolutely, yeah - just trying to capture thoughts in launchpad | 14:47 |
| jeblair | markmc: i'll leave some quick thoughts in that bug for now and let it stew for a while | 14:47 |
| markmc | jeblair, yeah, feel free to stick it on WishList | 14:47 |
| anteaya | david-lyle: I see your question but I don't know the answer | 14:47 |
| jeblair | david-lyle: i'll make one in just a moment | 14:48 |
| david-lyle | jeblair: thanks, I know this is low priority, just wasn't sure if there was a process to follow | 14:49 |
| *** russellb is now known as rustlebee | 14:50 | |
| anteaya | looking good, rustlebee | 14:50 |
| anteaya | Happy Friday | 14:50 |
| *** vogxn has joined #openstack-infra | 14:50 | |
| * rustlebee waves | 14:51 | |
| *** leifmadsen is now known as blitzrage | 14:51 | |
| markmc | http://logs.openstack.org/57/42757/8/gate/gate-nova-python26/e9e6e81/console.html | 14:51 |
| *** anteaya is now known as anTeaya | 14:51 | |
| markmc | 2013-08-23 14:21:11.473 | AssertionError: OS processes left [29687] | 14:52 |
| markmc | have we seen this before? | 14:52 |
| jeblair | i have not | 14:52 |
| markmc | lovely | 14:52 |
| * markmc dispatches his babies again | 14:52 | |
| markmc | mostly empty gate queue, at least | 14:53 |
| anTeaya | yay | 14:54 |
| *** bnemec is now known as RyanStiles | 14:54 | |
| RyanStiles | Heh | 14:55 |
| * RyanStiles <3's casual nick Friday | 14:55 | |
| anTeaya | looking good RyanStiles | 14:55 |
| *** anTeaya is now known as ColinMochrie | 14:56 | |
| ColinMochrie | I'll save the other one for another day | 14:56 |
| ColinMochrie | I only wore if for a few minutes, it is still good | 14:57 |
| * ColinMochrie puts anTeaya on the "can still be worn" pile | 14:57 | |
| jeblair | whose nick it is anyway? | 14:57 |
| ColinMochrie | :D | 14:57 |
| RyanStiles | lol | 14:57 |
| ColinMochrie | markmc: did the OS process left log come from: https://review.openstack.org/#/c/42736/ ? | 14:58 |
| markmc | ColinMochrie, no, different one | 14:58 |
| * ColinMochrie looks around for Wayne Brady and Greg Proops | 14:58 | |
| ColinMochrie | markmc: k | 14:58 |
| markmc | https://review.openstack.org/42757 | 14:59 |
| jeblair | david-lyle: should the horizon drivers also be the drivers for django-openstack-auth? | 14:59 |
| david-lyle | jeblair: yes | 14:59 |
| jeblair | david-lyle: should they share a bug team as well? | 15:00 |
| david-lyle | yes | 15:00 |
| ColinMochrie | markmc: thanks, let's see what happens this time | 15:00 |
| *** SergeyLukjanov has quit IRC | 15:01 | |
| jeblair | david-lyle: https://launchpad.net/django-openstack-auth | 15:01 |
| *** SergeyLukjanov has joined #openstack-infra | 15:02 | |
| david-lyle | jeblair: got it, thanks! | 15:02 |
| jeblair | david-lyle: the basics are set up, and i set horizon-drivers as the driver and maintainer, so you should be able to update anything you need | 15:02 |
| *** dina_bel_ has joined #openstack-infra | 15:03 | |
| *** dina_belova has quit IRC | 15:03 | |
| david-lyle | jeblair: will fill in the blanks shortly. Thanks again. | 15:03 |
| *** danger_fo_away is now known as danger_fo | 15:05 | |
| *** ruhe has quit IRC | 15:05 | |
| *** rfolco has quit IRC | 15:05 | |
| ColinMochrie | had to email in house about my nick change so some new folks can find me, let's see how this goes over | 15:09 |
| RyanStiles | ColinMochrie: Tell them to look for the bald Canadian guy. That should clear it right up. :-D | 15:10 |
| *** wenlock has joined #openstack-infra | 15:10 | |
| wenlock | morning | 15:11 |
| ColinMochrie | ha ha ha | 15:12 |
| ColinMochrie | yeah, I'm easy to spot | 15:12 |
| * ColinMochrie touches her/his bald spot | 15:12 | |
| ColinMochrie | feeling a bit gender confused on the pronouns | 15:12 |
| ColinMochrie | morning wenlock | 15:12 |
| *** vogxn has quit IRC | 15:12 | |
| wenlock | im checking out http://ci.openstack.org/wiki.html trying to learn it better, and followed some of the docs there. | 15:13 |
| wenlock | i have a couple questions | 15:13 |
| * ColinMochrie goes to talk to the phone guy, who just drove up | 15:13 | |
| wenlock | how do i figure out what extensions list used are and where to get them? | 15:13 |
| *** mgagne is now known as DrewCarey | 15:13 | |
| wenlock | such as strapping / openid? | 15:13 |
| * DrewCarey was feeling alone | 15:14 | |
| wenlock | and also, where can i find missing shell scirpts like /srv/mediawiki/backup.sh | 15:14 |
| RyanStiles | ColinMochrie: That sounds like the setup for a skit. | 15:14 |
| RyanStiles | DrewCarey: What style should ColinMochrie talk to the phone guy in? | 15:14 |
| DrewCarey | RyanStiles: ColinMochrie tries to answer the phone but lost his dentures | 15:17 |
| *** DrewCarey is now known as mgagne | 15:17 | |
| mgagne | ain't got time for that :O | 15:18 |
| RyanStiles | :-) | 15:18 |
| ColinMochrie | ain't nobody got time for that | 15:23 |
| ColinMochrie | phone guy is beside me in the house | 15:23 |
| RyanStiles | Now it's time to play "Only Questions"! | 15:26 |
| RyanStiles | :-) | 15:26 |
| * RyanStiles obviously has too much time on his hands | 15:26 | |
| annegentle | I've got a broken doc build that should be fixed with https://review.openstack.org/#/c/42984/, can I get a review? | 15:26 |
| ColinMochrie | how much time on your hands do you have, RyanStiles? | 15:26 |
| ColinMochrie | annegentle: jeblair is around, he should be back before long | 15:28 |
| RyanStiles | ColinMochrie: How should I know? | 15:28 |
| ColinMochrie | RyanStiles: You don't know? | 15:28 |
| annegentle | ColinMochrie: thanks | 15:28 |
| RyanStiles | ColinMochrie: Do I look like Father Time to you? | 15:28 |
| ColinMochrie | RyanStiles: Aren't you Father Time? | 15:28 |
| Shrews | Oooh. My favorite TV show playing out on IRC. Fun! | 15:28 |
| RyanStiles | ColinMochrie: Would I know if I was? | 15:29 |
| ColinMochrie | RyanStiles: don't you know who you are? | 15:29 |
| *** SergeyLukjanov has quit IRC | 15:29 | |
| RyanStiles | ColinMochrie: Do you? | 15:33 |
| clarkb | wenlock it is all a bit manual today. If you ping me in a couple hours I can find that info for you. then maybe we can start a puppet change to record more of this stuff in puppet | 15:33 |
| RyanStiles | (doing this on IRC is totally cheating. Too easy when you can think about your response) | 15:34 |
| clarkb | jeblair: markmc: remember check tests aren't quite the same as gate tests as $branch may have moved. in the case of testing merge commits this is probably mostly ok because in most cases the merge commit should fail if an individual change would | 15:35 |
| jeblair | clarkb: I'm reviewing 42608 (the groups-staging change); i'm not spinning it up now though because i want it to use trove | 15:35 |
| clarkb | there are corner cases where patch 1 adds a new hacking violation and patch 2 removes it | 15:36 |
| *** rfolco has joined #openstack-infra | 15:36 | |
| jeblair | clarkb: good point | 15:36 |
| *** michchap has joined #openstack-infra | 15:37 | |
| *** dolphm has joined #openstack-infra | 15:38 | |
| *** vogxn has joined #openstack-infra | 15:40 | |
| dolphm | i've got two committers that have independently claimed authorship over the exact same patch, in discrete Change-Id's... i'm trying to unwind who was the original author of what | 15:40 |
| dolphm | can i trust the timestamp shown in the Committer field on a patchset in gerrit? are they guaranteed to reflect the same timezone / UTC? | 15:41 |
| jeblair | dolphm: i believe they come from the local git commit, and could be anything (and set by the user) | 15:41 |
| dolphm | is that the date the patchset was uploaded to gerrit? | 15:41 |
| jeblair | dolphm: i think you can get better data with the gerrit query interface | 15:42 |
| dolphm | jeblair: i was assuming/hoping that the timestamp shown in the Author field came from the commit? | 15:42 |
| clarkb | gerrit should have a change created timestamp | 15:42 |
| jeblair | dolphm: "ssh review.openstack.org gerrit query #### --patch-sets " | 15:43 |
| *** tomhe has joined #openstack-infra | 15:43 | |
| dolphm | jeblair: thanks, was about to go look that up | 15:43 |
| jeblair | dolphm: that will give you 'createdOn' times which are gerrit | 15:43 |
| jeblair | dolphm: bouth the author and commit timestamps are git (not gerrit) | 15:44 |
| *** SergeyLukjanov has joined #openstack-infra | 15:44 | |
| dolphm | jeblair: perfect! thanks | 15:45 |
| tomhe | Question on Gerrit upgrade: I'm running Zuul from May 2 and considering upgrading Gerrit from 2.5 to 2.6.1. Anybody know if my Zuul would be compatible? | 15:47 |
| clarkb | jeblair: py26 jobs seem caught up with current zuul state and are ahead of the other jobs. we may not need more slaves | 15:48 |
| jeblair | tomhe: afaik; the wikimedia folks are using 2.7 i think. | 15:48 |
| openstackgerrit | A change was merged to openstack/requirements: Update Flask dependency https://review.openstack.org/43452 | 15:49 |
| *** ColinMochrie has quit IRC | 15:49 | |
| jeblair | clarkb: cool. i think static.o.o and logs are a problem; i think we should expand the storage pools until both logs and docs-draft are using 50% | 15:50 |
| jeblair | clarkb: and then change the cron job to run on weekends only | 15:50 |
| *** nayward has quit IRC | 15:50 | |
| jeblair | clarkb: (and maybe we should have run-tox gzip things if it isn't already) | 15:50 |
| jeblair | clarkb: also, somehow we ended up with a 1g server for static; i think we should make that an 8g because 300 scp sessions is bound to use some cpu. | 15:51 |
| fungi | i finally have a morning where my hands aren't swelled badly enough to make typing hard, and am not completely zonked out on antihistamines. already packed and have a couple hours to kill before i catch a cab to the pier... what's top priority/most broken at the moment? | 15:52 |
| tomhe | jeblair: Ok, was just hesitant, as my Zuul is rather old. | 15:52 |
| *** michchap has quit IRC | 15:52 | |
| *** ruhe has joined #openstack-infra | 15:53 | |
| *** svarnau has joined #openstack-infra | 15:53 | |
| jeblair | fungi: wow, i hope your vacation gets better | 15:53 |
| fungi | jeblair: yeah, this week was kinda a bust | 15:53 |
| fungi | next should be better | 15:53 |
| jeblair | fungi: what do you think about the logs stuff i just wrote? | 15:54 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=138&rra_id=all | 15:54 |
| fungi | i think it's a stellar idea. we need more cushion | 15:54 |
| *** dolphm has left #openstack-infra | 15:54 | |
| fungi | what was killing the cpu on it? find/compress runs? | 15:55 |
| fungi | iowait looks to be through the roof earlier | 15:55 |
| jeblair | i think it's that combined with spikes from jobs finishing | 15:55 |
| jeblair | yeah, the iowait is the find runs | 15:55 |
| *** pabelanger has quit IRC | 15:56 | |
| jeblair | ti looks like it only gets bad enough to timeout snmp during the find runs, so i suspect that when the jenkins jobs error out, it's during those times too | 15:56 |
| *** xBsd has quit IRC | 15:56 | |
| jeblair | so just running those on saturday should mostly take care of that until we can do something better | 15:56 |
| fungi | i wonder if we shouldn't use -O3 on those find commands | 15:57 |
| openstackgerrit | A change was merged to openstack/requirements: Add pyghmi https://review.openstack.org/41086 | 15:57 |
| fungi | i suspect the ordering is sub-optimal as they stand now | 15:57 |
| jeblair | fungi: probably :) | 15:57 |
| jeblair | fungi: know of a way to slow it down? | 15:57 |
| fungi | mmm, not really, no | 15:58 |
| fungi | other than batch it up by second-level directories and cram a sleep in each iteration or something | 15:58 |
| fungi | but that wouldn't be slow so much as bursty | 15:58 |
| openstackgerrit | A change was merged to openstack/requirements: Bump python-ceilometerclient to 1.0.3 https://review.openstack.org/43217 | 15:59 |
| fungi | or exec a sleep 1 on the find maybe? can sleep take subsecond intervals i wonder... | 15:59 |
| jeblair | fungi: you want to write a change to move the find command to very early (around midnight) saturday us-ish time? and maybe add that O3? | 16:00 |
| fungi | manpage says sleep takes a float | 16:00 |
| fungi | on itr | 16:00 |
| *** xBsd has joined #openstack-infra | 16:00 | |
| *** dkliban has quit IRC | 16:00 | |
| zaro | jeblair: i was trying to use gear's status protocol but i'm noticing that the info doesn't appear consistently. have you used the protocol lately? | 16:02 |
| *** cthulhup has joined #openstack-infra | 16:02 | |
| zaro | jeblair: i'm running the latest on master. | 16:02 |
| fungi | jeblair: clarkb: oh, also worth keeping an eye on... df -i /srv/static/docs-draft says we're at 90% inodes capacity on that filesystem | 16:03 |
| fungi | many, many small files i guess | 16:03 |
| jeblair | wow | 16:03 |
| *** dkliban has joined #openstack-infra | 16:04 | |
| zaro | jeblair: I also noticed an "UnknownJobError" when the gear client sends jobs too quickly to the server. http://paste.openstack.org/show/45040/ | 16:04 |
| jeblair | zaro: i have not noticed a problem with the status command | 16:05 |
| jeblair | zaro: you have https://review.openstack.org/#/c/40462/ this change in your tree, right? | 16:06 |
| zaro | jeblair: yes, it's there. | 16:07 |
| pleia2 | good morning | 16:08 |
| zaro | jeblair: to repo i just submit about 10 jobs in a loop. | 16:08 |
| jeblair | zaro: i don't have any quick ideas about that (and don't have time to dig into it now, sorry) | 16:08 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: Conditionally run the slow heat tests https://review.openstack.org/43500 | 16:08 |
| zaro | jeblair: ok. i'll dig a little deeper. | 16:09 |
| *** sarob has joined #openstack-infra | 16:10 | |
| *** dina_bel_ has quit IRC | 16:12 | |
| openstackgerrit | Monsyne Dragon proposed a change to openstack/requirements: Add jsonpath-rw https://review.openstack.org/43503 | 16:13 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Reduce log compression frequency and optimize https://review.openstack.org/43505 | 16:16 |
| *** mikal has quit IRC | 16:17 | |
| *** mikal has joined #openstack-infra | 16:19 | |
| fungi | zaro: we did the computer museum yesterday morning on very short notice, so i didn't get a chance to give you a heads up, sorry :/ (but i highly recommend it as long as you can get past some of the microsoft propaganda... they even have a working pdp-7 on display!) | 16:19 |
| *** dina_belova has joined #openstack-infra | 16:20 | |
| fungi | i hacked at a pdp-11 a little (figured out how to fire up trek on it and played a few rounds) | 16:21 |
| *** reed has joined #openstack-infra | 16:22 | |
| *** dina_belova has quit IRC | 16:22 | |
| *** dina_belova has joined #openstack-infra | 16:22 | |
| clarkb | jeblair everything you said about static sounds great to me | 16:22 |
| *** david-lyle has quit IRC | 16:22 | |
| jeblair | fungi: so i think i'll make a new filesystem for docs-draft and do a bytes-per-inode of 8192 (about half of the current ratio); sound good? | 16:22 |
| clarkb | I stepped away. still trying to sort out dsl situation before the weekend. about to head to the office though | 16:23 |
| *** lcestari has quit IRC | 16:23 | |
| jeblair | clarkb: cool, np | 16:23 |
| fungi | jeblair: yeah, that seems saner. just growing that volume could put us in a bad way if it fills up too much more | 16:23 |
| fungi | a opposed to /srv/static/logs which is only around 10% inode utilization, so i think it's safe to just extend in place | 16:24 |
| zaro | fungi: np. good to hear that you got to see it. | 16:24 |
| clarkb | morning pleia2 | 16:25 |
| jeblair | clarkb, fungi: one of you want to aprv this to fix a docs build problem? https://review.openstack.org/#/c/42984/ | 16:25 |
| fungi | looking over it now | 16:25 |
| *** mrodden has quit IRC | 16:26 | |
| *** david-lyle has joined #openstack-infra | 16:26 | |
| *** ruhe has quit IRC | 16:26 | |
| fungi | lgtm, approved | 16:27 |
| jeblair | annegentle: ^ | 16:27 |
| *** ^d has joined #openstack-infra | 16:28 | |
| *** jpich has quit IRC | 16:28 | |
| *** dkliban has quit IRC | 16:28 | |
| openstackgerrit | A change was merged to openstack-infra/config: Reduce log compression frequency and optimize https://review.openstack.org/43505 | 16:29 |
| *** matel has quit IRC | 16:29 | |
| openstackgerrit | A change was merged to openstack-infra/config: Ensure that the release.path.name is set for the Block Storage https://review.openstack.org/42984 | 16:29 |
| *** svarnau has quit IRC | 16:29 | |
| *** cthulhup has quit IRC | 16:32 | |
| *** dkliban has joined #openstack-infra | 16:32 | |
| *** pblaho has quit IRC | 16:33 | |
| fungi | jeblair: also, if you're going to be growing those lvms on static.o.o, be aware that cutting-and-pasting from the html-rendered notes on ci.o.o isn't so great. https://review.openstack.org/42483 aims to fix it | 16:33 |
| fungi | though i just noticed i introduced trailing whitespace on one line there. fixing | 16:34 |
| *** ColinMochrie has joined #openstack-infra | 16:34 | |
| jeblair | fungi: thx | 16:34 |
| *** weshay is now known as weshay_brb | 16:35 | |
| *** ruhe has joined #openstack-infra | 16:36 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Clean up Static Web Hosting document https://review.openstack.org/42483 | 16:37 |
| fungi | better ^ | 16:37 |
| *** rfolco has quit IRC | 16:37 | |
| *** dina_belova has quit IRC | 16:39 | |
| openstackgerrit | A change was merged to openstack-infra/config: Clean up Static Web Hosting document https://review.openstack.org/42483 | 16:40 |
| reed | hello folks | 16:40 |
| fungi | hello reed | 16:40 |
| jeblair | reed: hello; i have reviewed mrmartin's patch; left some suggestions, and took a quick look at yours (i think we may want to do the same as the suggestion i left on groups) | 16:41 |
| reed | jeblair, regarding the mysql thingie? | 16:41 |
| jeblair | reed: yep | 16:41 |
| *** SergeyLukjanov has quit IRC | 16:41 | |
| reed | uhm | 16:42 |
| reed | what do I need to do then? | 16:42 |
| jeblair | reed: i left detailed notes on mrmartin's patch; you can probably translate that to yours | 16:43 |
| reed | ok | 16:43 |
| *** sarob has quit IRC | 16:44 | |
| *** Ryan_Lane has quit IRC | 16:44 | |
| *** sarob has joined #openstack-infra | 16:44 | |
| *** ruhe has quit IRC | 16:47 | |
| *** fbo is now known as fbo_away | 16:47 | |
| *** sarob_ has joined #openstack-infra | 16:48 | |
| *** sarob has quit IRC | 16:48 | |
| *** dkehn has quit IRC | 16:48 | |
| *** rcleere has joined #openstack-infra | 16:48 | |
| *** dansmith is now known as Steely_Dan | 16:49 | |
| *** dkehn has joined #openstack-infra | 16:49 | |
| *** rfolco has joined #openstack-infra | 16:50 | |
| ColinMochrie | reed: are there any atc summit passes yet to be issued? | 16:51 |
| reed | colinMochrie: yes, as soon as the next milestone is released we'll send another batch | 16:52 |
| ColinMochrie | ah okay thanks | 16:52 |
| *** nicedice_ has joined #openstack-infra | 16:52 | |
| reed | if you know somebody that has committed code or is going to do it by next milestone have them book hotel and flights | 16:52 |
| *** nicedice_ has joined #openstack-infra | 16:52 | |
| ColinMochrie | okay thanks, NobodyCam was asking about a friend of his | 16:52 |
| jeblair | fungi, clarkb: the cron change has taken effect, so we shouldn't see another run of the find until tomorrow. | 16:56 |
| pleia2 | jeblair: do we want to let this one through, or wait until monday? https://review.openstack.org/#/c/38177/ | 16:56 |
| *** weshay_brb has quit IRC | 16:56 | |
| jeblair | fungi, clarkb: and i'm rsyncing docs-draft to a new fs with twice the inode ratio | 16:56 |
| fungi | jeblair: if we still see impact to snmp et cetera when it does run, i wonder if nicing it will help? | 16:57 |
| jeblair | pleia2: sooner the better i think | 16:57 |
| *** weshay_brb has joined #openstack-infra | 16:57 | |
| * pleia2 nods | 16:57 | |
| jeblair | fungi: i tried that last night with the run at around midnight utc and did not notice a difference on the graphs | 16:57 |
| jeblair | fungi: that's pretty far from conclusive though | 16:57 |
| fungi | ahh, okay. nice is really only effective for reducing %user on cpu anyway. system and iowait tend to be outside its context | 16:58 |
| jeblair | fungi: but my _guess_ is that the niceness doesn't end up affecting the io time | 16:58 |
| fungi | yup | 16:58 |
| jeblair | was worth a shot though :) | 16:58 |
| jeblair | pleia2: aprvd | 16:59 |
| fungi | so, yeah, about the only other throttle i can come up with would be to inline a sleep 0.1 or something on every -exec parameter | 16:59 |
| pleia2 | thanks | 17:00 |
| jeblair | fungi: yeah, but my guess is we actually don't exec very often | 17:00 |
| fungi | agreed. traversing the linklist is probably most of the i/o | 17:00 |
| fungi | which we can't really do anything about other than possibly make it more bursty | 17:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Use cgit server instead of github for everything https://review.openstack.org/38177 | 17:01 |
| *** dkehn has quit IRC | 17:02 | |
| *** dkehn has joined #openstack-infra | 17:02 | |
| clarkb | ok in the office now with an internet connection. ISP assures me it is the modems power supply (if only I had a multimeter) and will drop a new one off today | 17:02 |
| clarkb | I should buy a multimeter this weekend | 17:03 |
| jeblair | clarkb: i got a free one with my oscilloscope; stop by and i'll give it to you. | 17:03 |
| clarkb | road trip! | 17:04 |
| ColinMochrie | mine is so old it is still analog | 17:04 |
| ColinMochrie | oscilloscope sounds like a fun toy | 17:05 |
| clarkb | jeblair: do you have one of the big tektronix bench type oscilloscopes or one of those neat small handheld things? | 17:05 |
| *** Dr01d has quit IRC | 17:06 | |
| jeblair | clarkb: somewhere in between; it's a new short digital benchtop; bout the size of a toolbox. | 17:07 |
| fungi | clarkb: you really don't have a dmm? how does that happen? i think i have four, mostly by accident | 17:07 |
| clarkb | fungi: moving a lot and always having access to tools either at school or at work | 17:07 |
| jeblair | clarkb: if i had the room, i'd get an analog tektronix, those are the best (and still can measure things you can't really measure with digital ones) | 17:07 |
| fungi | i've been eyeballing some of the handheld 'scopes but the timeslice resolution on those digital models usually isn't great | 17:07 |
| *** jfriedly has joined #openstack-infra | 17:08 | |
| fungi | so i will probably spring for an analog model when i finally decide i need one badly enough | 17:08 |
| jeblair | clarkb: but i replaced the firmware on it so it's at least upgraded to a 100mhz resolution | 17:08 |
| clarkb | this HP office is the first one I have worked in without a proper tool bench ( EntropyWorks' office doesn't count ) | 17:08 |
| ColinMochrie | clarkb: HP offices come with toys/tools in them? | 17:10 |
| clarkb | ColinMochrie: not this one | 17:10 |
| ColinMochrie | but other ones do? | 17:10 |
| clarkb | I dunno. THis is the only one I have been in | 17:10 |
| ColinMochrie | ah okay, I mis-understood | 17:10 |
| wenlock | not much different here | 17:10 |
| clarkb | I probably neglected to use punctuation properly | 17:11 |
| ColinMochrie | I was looking forward to tools/toys | 17:11 |
| ColinMochrie | I do that all the time too | 17:11 |
| ColinMochrie | wenlock: no tool bench for you either? | 17:11 |
| wenlock | they do come with nasty coffee | 17:11 |
| ColinMochrie | ha ha ha | 17:11 |
| reed | nasty coffee is everywhere in US | 17:14 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Fix zuul status hours display. https://review.openstack.org/43375 | 17:14 |
| reed | at least in offices it's usually $0... | 17:14 |
| *** weshay_brb is now known as weshay | 17:14 | |
| *** dina_belova has joined #openstack-infra | 17:14 | |
| reed | it's unbelievable to me that people pay over $3 for horrible, nasty coffee in shops | 17:15 |
| clarkb | reed: I solve that problem by not drinking coffee | 17:15 |
| reed | and do we want to talk about $5 lattes? Seriously folks, that's messed up | 17:15 |
| *** sdake_ has joined #openstack-infra | 17:16 | |
| reed | clarkb, yeah, boycott Peet's, starbucks, blue bottle and all those nasty people that go there :) | 17:16 |
| jeblair | clarkb, fungi: should we go ahead and spin the resize roulette wheel on static.o.o? | 17:16 |
| reed | spin it! | 17:16 |
| pleia2 | clarkb: I have git.o.o announcement prepped, would like to have docs updated when I post though, so when you get a chance if you could review :) | 17:16 |
| *** nati_ueno has joined #openstack-infra | 17:16 | |
| ColinMochrie | clarkb: + | 17:16 |
| reed | $20 on red | 17:16 |
| clarkb | pleia2: yup that is first up | 17:16 |
| clarkb | jeblair: the gate queue is small I say go for it | 17:17 |
| fungi | jeblair: should be safe enough | 17:18 |
| fungi | i've done online pv/vg/lv/fs resize operations under heavy i/o load in the past with no ill effects | 17:18 |
| jeblair | i meant the server resize :) | 17:19 |
| jeblair | to give it more cpus | 17:19 |
| fungi | oh, that | 17:19 |
| jeblair | i mean, it should just be a momentary outage for the reboot; assuming all the volumes move over ok | 17:19 |
| fungi | it's a 1m flavor now... what size were you wanting to take it up to? | 17:19 |
| jeblair | 8g | 17:19 |
| jeblair | 4vcpus | 17:20 |
| fungi | that shouldn't take too terribly long. my guess is you'll be waiting for about an hour and then see a 10 minute outage for the vm as it does its final sync and reboot | 17:20 |
| jeblair | yeah | 17:20 |
| *** morganfainberg|a is now known as morganfainberg | 17:21 | |
| clarkb | and it is a bit of roulette because we don't know exactly when that will happen, but it is friday and the devs have been patient with us so far this week :) | 17:21 |
| jeblair | we should also scale git.o.o _down to an 8g | 17:21 |
| clarkb | I say go for it | 17:21 |
| clarkb | pleia2: reviewed. One small thing inline | 17:21 |
| jeblair | fungi: did you see our shiny git server setup? | 17:21 |
| jeblair | fungi: 5 load balanced hosts | 17:21 |
| *** BobBall is now known as BobBallAway | 17:22 | |
| fungi | jeblair: i did not, unfortunately. been more out of touch the past few days than i'd hoped | 17:22 |
| *** sarob_ has quit IRC | 17:22 | |
| fungi | but that sounds awesome | 17:22 |
| clarkb | jeblair: I agree. git.o.o is very oversized (as is the pool according to cacti but I am willing to wait for H3 to happen before worrying about it) | 17:22 |
| jeblair | fungi: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2 | 17:22 |
| ColinMochrie | jgriffith: I am looking at https://review.openstack.org/#/c/43202/ and I see that it is failing on a pep8 test - Python 3.x incompatible use of print operator | 17:23 |
| fungi | jeblair: oh, neat! that's performing waaaay better | 17:24 |
| * fungi needs to power down his netbook and go hail a cab to the pier | 17:24 | |
| clarkb | ColinMochrie: I had a theory about that last night that I couldn't follow because doing so on a phone was meh. But I think it is possible that the allowed hacking rules changes between two runs of that test | 17:24 |
| jeblair | clarkb: what do you think about downsizing git.o.o sooner? | 17:24 |
| jeblair | fungi: have a nice trip! | 17:24 |
| clarkb | jeblair: I think we can downsize git.o.o sooner | 17:24 |
| ColinMochrie | clarkb: ah okay, so do a recheck? | 17:24 |
| fungi | i may be back on from the boat later, depending on how the day in dock shapes up | 17:24 |
| clarkb | jeblair: or (and this might take a bit of dancing) make git01.o.o the new git.o.o | 17:24 |
| clarkb | jeblair: and kill git.o.o | 17:24 |
| jeblair | clarkb: interesting idea... | 17:25 |
| ColinMochrie | clarkb: or jgriffith might have to edit the patch to please the new hacking guidelines | 17:25 |
| *** gyee has joined #openstack-infra | 17:25 | |
| clarkb | ColinMochrie: no recheck, I think those are legit failures | 17:25 |
| ColinMochrie | clarkb: very good | 17:26 |
| jeblair | clarkb: we can take git01 out of the rotation on git.o; lower dns ttls, configure git01 to be git.o.o, change dns, wait for propogation, delete old git.o | 17:26 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Update git documentation with HAProxy details https://review.openstack.org/43385 | 17:26 |
| jeblair | clarkb: should be zero downtime, and they'll both be git.o.o for about 5 minutes | 17:26 |
| clarkb | jeblair: I was thinking of making the puppet manifest smart enough to put haproxy on git01 but not 02-04, make git.o.o a CNAME to git01 then killing git.o.o later | 17:27 |
| clarkb | jeblair: and let git.o.o be a thing in name only. but either way works | 17:27 |
| ColinMochrie | clarkb: I appended that point in a comment on the jgriffith patch | 17:27 |
| clarkb | jeblair: we will also need to update the gerrit replication config | 17:28 |
| *** sdake_ has quit IRC | 17:28 | |
| clarkb | ColinMochrie: thanks | 17:28 |
| ColinMochrie | np | 17:28 |
| jeblair | clarkb: oh, i think we should have git.o.o continue to be the load balancer and it should be different than the workers | 17:29 |
| jeblair | clarkb: i also think that we should probably stop having it serve locally, and just let it load balance | 17:30 |
| clarkb | jeblair: different than the workers = not actually serving git? | 17:30 |
| clarkb | jeblair: ok | 17:30 |
| *** Ryan_Lane has joined #openstack-infra | 17:30 | |
| jeblair | clarkb: yeah, i'll get really confused if i have to remember that git01 is also the load balancer. :) | 17:30 |
| clarkb | in that case a new git.o.o is probably the easiest route. I can work on updating the puppet manifests to split those things apart | 17:30 |
| jeblair | clarkb: just putzing inth rax lbaas config, i don't see 'source' as a config option | 17:31 |
| clarkb | jeblair: the lbaas offerings are very rudimentary | 17:32 |
| clarkb | jeblair: there is no queueing support either | 17:32 |
| jeblair | shame | 17:32 |
| jeblair | clarkb: ok, so you want to just do a new replacement git server, 8g, no local git serving, and have it talk to the existing 4 workers? | 17:32 |
| *** vogxn has left #openstack-infra | 17:33 | |
| clarkb | jeblair: I think so. Adding a new server that isn't replicated too seems simpler than taking an old server and cleaning it up | 17:33 |
| jeblair | i guess that will decrease our pool size by 1 (by removing the local serving from the lb), which should be fine | 17:33 |
| clarkb | jeblair: and doing what you describe will need a puppet update anyways | 17:33 |
| clarkb | ya Ithink we have plenty of capacity | 17:33 |
| jeblair | ok, sounds good. | 17:33 |
| jeblair | clarkb: i'll go ahead and lower the ttl on git.o.o | 17:33 |
| jeblair | static is rebooting now | 17:33 |
| clarkb | that was fast | 17:34 |
| jeblair | ya | 17:34 |
| clarkb | jeblair: I'd like to get some code review in today as I have neglected that. Can updating puppet for git.o.o wait for next week? | 17:34 |
| jeblair | clarkb: sure | 17:34 |
| jeblair | i'll leave the ttl's as they are then | 17:34 |
| jeblair | clarkb: i'm going to continue focusing on static.o. and the zuul re-run-job thing | 17:35 |
| jeblair | as those are still operational issues | 17:35 |
| jeblair | clarkb: let me know if you spot something i should review | 17:35 |
| clarkb | yup, one of the things I really want to review is your nodepool testnode change | 17:36 |
| clarkb | which is at least semi related to the zuul re run job thng | 17:36 |
| jeblair | clarkb: yeah, i still don't want to use it though :) | 17:36 |
| clarkb | ok | 17:36 |
| *** sdake_ has joined #openstack-infra | 17:36 | |
| *** johnthetubaguy has quit IRC | 17:36 | |
| jeblair | i'm thinking of it as more of a backup plan if the re-run jobs thing doesn't work out | 17:36 |
| jeblair | because MOST of the time nodes are fine, so having zuul re-run a job for the few that aren't is much more efficient than having jenkins do lots of extra work for the many thare are | 17:37 |
| jeblair | that are | 17:37 |
| dkranz | clarkb: I just wanted to verify that a shell variable exported in devstack-gate.yaml will be available to devstack when it runs. Is that correct? | 17:38 |
| *** cppcabrera has joined #openstack-infra | 17:40 | |
| *** cppcabrera has left #openstack-infra | 17:40 | |
| clarkb | dkranz: yes | 17:40 |
| dkranz | clarkb: Thanks. | 17:40 |
| jeblair | dkranz: (though most things like that end up in localrc, right?) | 17:41 |
| *** lcestari has joined #openstack-infra | 17:41 | |
| clarkb | ya typically those exports are for devstack-gate and not for devstack itself | 17:41 |
| jeblair | dkranz: putting it in localrc makes it more self-documenting; you can download the localrc from the result log server | 17:41 |
| dkranz | jeblair: OK, I can do that instead. | 17:42 |
| jeblair | static is back up | 17:48 |
| clarkb | jeblair: did you want to read pleia2's cgit doc changes before I approve? | 17:50 |
| *** lbragstad has joined #openstack-infra | 17:51 | |
| *** UtahDave has joined #openstack-infra | 17:51 | |
| *** pabelanger has joined #openstack-infra | 17:51 | |
| jeblair | clarkb: aprvd | 17:52 |
| *** mrodden has joined #openstack-infra | 17:53 | |
| jeblair | clarkb: i'm going to do one more disruptive thing on static; i want to swap out the /srv partition for a new lvm volume | 17:53 |
| jeblair | #status alert recent UNSTABLE jobs were due to maintenance to expand capacity which is complete; recheck or reverify as needed | 17:53 |
| openstackstatus | NOTICE: recent UNSTABLE jobs were due to maintenance to expand capacity which is complete; recheck or reverify as needed | 17:53 |
| *** apcruz has quit IRC | 17:53 | |
| *** ChanServ changes topic to "recent UNSTABLE jobs were due to maintenance to expand capacity which is complete; recheck or reverify as needed" | 17:53 | |
| jeblair | (hopefully it shouldn't be that disruptive, i think i can swap things out pretty quickly | 17:53 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: Conditionally run the slow heat tests https://review.openstack.org/43500 | 17:54 |
| clarkb | jeblair: I won't approve any nodepool changes becaues I am not sure which ones you have in production and so on | 17:56 |
| jeblair | clarkb: ok, i'll take care of it (puppet still isn't running there though) | 17:56 |
| *** zul has quit IRC | 17:58 | |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: Conditionally run the slow heat tests https://review.openstack.org/43500 | 18:00 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/config: Add slow heat job https://review.openstack.org/43511 | 18:02 |
| openstackgerrit | A change was merged to openstack-infra/config: Add socat package to cgit servers https://review.openstack.org/43354 | 18:06 |
| clarkb | jeblair: ^ that confirms that static.o.o is happy again | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/config: Update git documentation with HAProxy details https://review.openstack.org/43385 | 18:07 |
| *** SergeyLukjanov has joined #openstack-infra | 18:07 | |
| jeblair | i need to break it again though :( | 18:08 |
| jeblair | just for a reboot, i hope | 18:08 |
| clarkb | is that for the inodes? | 18:08 |
| jeblair | no; it's very hard to swap out the /srv mountpoint | 18:08 |
| clarkb | so you are just going to change fstab and reboot? | 18:09 |
| jeblair | well, i'm mostly rebooting to fix up the strange situation i got it into | 18:09 |
| clarkb | ok let me know if you need an extra set of hands or eyes | 18:10 |
| jeblair | will do | 18:10 |
| *** sgviking has quit IRC | 18:10 | |
| jeblair | what an interesting server -- we can't start or finish jobs without it | 18:11 |
| *** apcruz has joined #openstack-infra | 18:11 | |
| jeblair | (pypi and logs) | 18:11 |
| *** sgviking has joined #openstack-infra | 18:11 | |
| clarkb | that is a fun spof | 18:12 |
| jeblair | clarkb: ok, should be done; i notice that my shell is kind of laggy; can you log into it and let me know how it seems to you? | 18:12 |
| clarkb | ssh connection to it took a while to ask for my key | 18:13 |
| clarkb | and yes laggy | 18:13 |
| jeblair | clarkb: i have almost 1000ms ping | 18:13 |
| jeblair | not sure i like our rebuilt server | 18:13 |
| *** xBsd has quit IRC | 18:13 | |
| clarkb | mtr shows packet loss | 18:13 |
| clarkb | that last router in the route seems to be laggy | 18:14 |
| jeblair | clarkb: weird, it's not in the route to zuul. | 18:15 |
| jeblair | clarkb: nor is there an equivalent | 18:15 |
| nati_ueno | Wow! log is colerd! cool | 18:15 |
| *** dina_belova has quit IRC | 18:15 | |
| clarkb | nati_ueno: sdague did that for us | 18:15 |
| clarkb | nati_ueno: should make it a little easier to read and the timestamps are anchor tags so you can link into them | 18:16 |
| nati_ueno | sdague: Thanks! | 18:16 |
| nati_ueno | wooooot it has link!! | 18:16 |
| clarkb | jeblair: perhaps it is only a problem coming from the outside world? | 18:16 |
| ColinMochrie | yay for linking to timestamps | 18:16 |
| nati_ueno | yay! | 18:16 |
| jeblair | clarkb: oh, no i meant i was comparing a traceroute from here to static, and from here to zuul | 18:16 |
| clarkb | oh | 18:17 |
| nati_ueno | This makes bug reports really easy | 18:17 |
| jeblair | clarkb: but there is an equivalent hop in there for the git servers (though rdns isn't working); it's fast | 18:17 |
| clarkb | jeblair: it is laggy between zuul and static | 18:17 |
| jeblair | clarkb: i'll open a tt and mention the router | 18:17 |
| clarkb | jeblair: ok | 18:17 |
| clarkb | nati_ueno: it doesn't work for all log files but making it easier to work with logs was the intention | 18:18 |
| nati_ueno | clarkb: yeah this is really cool | 18:18 |
| ColinMochrie | nati_ueno: what logs are you looking at? I can't seem to find logs with linking timestamps | 18:18 |
| nati_ueno | ColinMochrie: http://logs.openstack.org/58/43358/2/check/gate-tempest-devstack-vm-neutron/48dcf6a/logs/screen-q-l3.txt.gz#_2013-08-22_22_55_07_841 | 18:19 |
| * ColinMochrie looks | 18:19 | |
| ColinMochrie | awesome, i hadn't seen that before | 18:20 |
| clarkb | ColinMochrie: that was a thing that sdague brainstormed at the bootcamp | 18:20 |
| ColinMochrie | it is great | 18:20 |
| ColinMochrie | makes finding the useful bits in the log so much faster | 18:21 |
| ColinMochrie | just in tempest logs? | 18:21 |
| ColinMochrie | so far? | 18:21 |
| jeblair | clarkb: oh, i think ipv6 may not work (we don't have a AAAA record defined for it) | 18:21 |
| jeblair | clarkb: <sigh> maybe we should spin up a replacement server? | 18:23 |
| ColinMochrie | look at this log awesomeness for testr results: http://logs.openstack.org/34/43034/2/check/gate-tempest-devstack-vm-neutron/6dd68b0/testr_results.html.gz | 18:24 |
| jeblair | clarkb: i'm going to get that started | 18:25 |
| clarkb | jeblair: I think you are correct about ipv6, and since we try to have ipv6 everywhere a new node to fix the problem isn't a bad idea | 18:25 |
| clarkb | jeblair: with the volumes there is no expensive copying right? | 18:26 |
| jeblair | clarkb: right | 18:26 |
| jeblair | lowering the ttl on static | 18:26 |
| ColinMochrie | the timestamp links seem to be for the devstack/screen logs | 18:27 |
| jeblair | ah, it's already 300 | 18:28 |
| clarkb | ColinMochrie: yes and I think they must be in oslo format | 18:28 |
| *** mikal has quit IRC | 18:28 | |
| ColinMochrie | clarkb: cool, I will see if I can spot the oslo ones too | 18:29 |
| *** mikal has joined #openstack-infra | 18:30 | |
| *** basha has joined #openstack-infra | 18:31 | |
| clarkb | ColinMochrie: nova, glance, neutron, cinder all use oslo logging iirc | 18:31 |
| clarkb | ColinMochrie: oh and heat and ceilometer | 18:31 |
| ColinMochrie | looking at olso logs, so far I can find it, I will continue to look, thanks for project names | 18:31 |
| *** senk has joined #openstack-infra | 18:31 | |
| jeblair | clarkb: rtt got better | 18:32 |
| clarkb | much better | 18:33 |
| *** zul has joined #openstack-infra | 18:34 | |
| jeblair | still have a replacement spinning up (to deal with v6), but no rush on that | 18:34 |
| ColinMochrie | so far I can't seem to find the oslo format logs of which you speak: https://review.openstack.org/#/c/43489/ https://review.openstack.org/#/c/41685/ https://review.openstack.org/#/c/41177/ | 18:35 |
| ColinMochrie | bbiab | 18:35 |
| jeblair | i've restarted the docs-draft rsync; so when that's done, i can move the docs-draft fs, and then expand the logs fs | 18:35 |
| jeblair | i've already added the devices | 18:35 |
| clarkb | ColinMochrie: nova logs are in oslo format | 18:36 |
| clarkb | ColinMochrie: there isn't an oslo.log file instead nova uses oslo.log to write its log files. By default they come out in a standard format | 18:36 |
| *** moted has quit IRC | 18:39 | |
| *** moted has joined #openstack-infra | 18:39 | |
| *** dina_belova has joined #openstack-infra | 18:40 | |
| ColinMochrie | clarkb: I looked at these three logs for gate-nova-python26: http://logs.openstack.org/77/41177/4/check/gate-nova-python26/145ac37/ | 18:41 |
| ColinMochrie | I don't see the same kind of linked timestamps as in the devstack/screen logs | 18:41 |
| ColinMochrie | the testr logs are doing their own lovely thing | 18:41 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix releastestatus dependency https://review.openstack.org/43516 | 18:42 |
| jeblair | clarkb: ok, new static server exists for whenever we want to move it | 18:43 |
| ColinMochrie | jeblair: just commented on the commit message for 43516 | 18:44 |
| *** dmakogon_ has joined #openstack-infra | 18:44 | |
| *** xBsd has joined #openstack-infra | 18:46 | |
| *** dkehn has quit IRC | 18:47 | |
| *** dkehn has joined #openstack-infra | 18:47 | |
| clarkb | ColinMochrie: oh yeah it is only for the screen logs that are in the oslo format | 18:50 |
| clarkb | jeblair: ok | 18:51 |
| ColinMochrie | clarkb: that is okay, I was hopeful | 18:52 |
| *** pabelanger has quit IRC | 18:54 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add ProviderManager https://review.openstack.org/42973 | 18:59 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add JenkinsManager https://review.openstack.org/43014 | 18:59 |
| *** danger_fo is now known as danger_fo_away | 19:00 | |
| *** w_ has joined #openstack-infra | 19:00 | |
| *** basha has quit IRC | 19:01 | |
| *** pcm_ has quit IRC | 19:02 | |
| *** olaph has quit IRC | 19:02 | |
| *** datsun180b has quit IRC | 19:03 | |
| *** markmc has quit IRC | 19:03 | |
| *** alexpilotti has quit IRC | 19:06 | |
| ttx | jeblair, clarkb: could you kill https://review.openstack.org/#/c/43424/ before it merges ? I don't really want us to disable tests without a larger discussion | 19:10 |
| *** datsun180b has joined #openstack-infra | 19:10 | |
| jeblair | dkranz, mtreinish: can you weigh in on that ^ ? | 19:12 |
| mtreinish | ttx: I already -2'd it | 19:12 |
| *** p5ntangle has quit IRC | 19:12 | |
| ttx | mtreinish: not sure that will stop it :) | 19:13 |
| mtreinish | ttx: it'll stop it from merging | 19:13 |
| jeblair | it will | 19:13 |
| *** mrmartin has joined #openstack-infra | 19:13 | |
| ttx | mtreinish: cool | 19:13 |
| jeblair | it will stop it from merging | 19:13 |
| ttx | jeblair: ignore me :) | 19:13 |
| *** olaph has joined #openstack-infra | 19:13 | |
| ttx | mtreinish: you on stable-maint ML ? | 19:14 |
| dkranz | ttx: I approved this because stable/grizzly has been dead for 6 days | 19:14 |
| ttx | dkranz: I think we are closing in on a solution though | 19:15 |
| *** ladquin_afk has quit IRC | 19:15 | |
| dkranz | ttx: And it is just two cli tests. | 19:15 |
| dkranz | ttx: Excellent | 19:15 |
| mtreinish | ttx: no I don't think so. I'll subscribe right now | 19:15 |
| ttx | like, we have all the bits in place, afaict | 19:15 |
| dkranz | ttx: Then we can revert | 19:15 |
| ttx | pushing the discussion to stable-maint ML | 19:15 |
| dkranz | ttx: I am not on stable maint but saw a lot of frustration there. | 19:15 |
| dkranz | ttx: Did not know a solution was imminent | 19:16 |
| *** w_ has quit IRC | 19:16 | |
| ttx | dkranz: it's harder to tell we have the solution though... unless we make a whole copy of the tests and make it non-votng | 19:16 |
| *** _TheDodd_ is now known as JamesBond | 19:16 | |
| dkranz | ttx: I'm happy to turn it back on. | 19:16 |
| *** JamesBond is now known as WillSmith | 19:16 | |
| ttx | dkranz: my understanding is that the combination of the keystoneclient 0.3.2 and the addition of netaddr to the test-requires should fix it | 19:16 |
| *** WillSmith is now known as Jesus | 19:17 | |
| dkranz | ttx: It was just a strange coincidence that after 6 days of failure a solution is ready at exactly the moment we gave up :) | 19:17 |
| *** Jesus is now known as Guest69671 | 19:17 | |
| ttx | dkranz: the problem is more that the discussion happens on a variety of media | 19:17 |
| ttx | dkranz: te review, openstack-dev for the announcement of 0.3.2 and stable-maint for the discussion of the fix | 19:18 |
| *** Guest69671 is now known as _TheDodd_ | 19:18 | |
| *** _TheDodd_ has quit IRC | 19:18 | |
| annegentle | Any launchpad wizards know how to find out how many bugs have been set to Fix Released in the last week? | 19:18 |
| *** changbl has quit IRC | 19:18 | |
| annegentle | Oh I think I'll just do an adv. search and sort by Date last updated | 19:19 |
| annegentle | nevermind | 19:20 |
| *** odyssey4me4 has quit IRC | 19:20 | |
| ttx | dkranz: cc-ed you on the stable-maint discussion | 19:20 |
| ttx | mtreinish: same for you | 19:20 |
| *** psedlak has quit IRC | 19:20 | |
| mtreinish | ttx: ok thanks, I'm subscribed now too. :) | 19:21 |
| ColinMochrie | why are we getting hit so hard in check right now? we have 70 in the queue and 87 events? | 19:21 |
| ColinMochrie | been less than 50 for most of the day in check, where did we get the extra 100 all of a sudden? | 19:22 |
| ColinMochrie | or is this just Friday afternoon rush? | 19:22 |
| *** psedlak has joined #openstack-infra | 19:22 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add py33 jobs for WSME https://review.openstack.org/43112 | 19:22 |
| *** datsun180b has quit IRC | 19:23 | |
| *** datsun180b has joined #openstack-infra | 19:23 | |
| *** _Jesus_ has joined #openstack-infra | 19:23 | |
| ttx | We have Neutron and Heat feature proposal freezes today, maybe that helps | 19:23 |
| *** _Jesus_ is now known as Guest65518 | 19:23 | |
| *** Guest65518 has quit IRC | 19:24 | |
| ColinMochrie | ttx ah the end of day feature proposal freeze rush | 19:25 |
| *** __God__ has joined #openstack-infra | 19:25 | |
| ColinMochrie | maybe that's it | 19:25 |
| *** __God__ has quit IRC | 19:25 | |
| *** thedodd has joined #openstack-infra | 19:25 | |
| ColinMochrie | thedodd: you when from James Bond to Will Smith to Jesus to God - I wanted to see what you had next | 19:26 |
| ColinMochrie | s/when/went | 19:26 |
| *** __afazekas is now known as afazekas | 19:28 | |
| rustlebee | <3 https://twitter.com/russellbryant/status/370990448715649024 | 19:28 |
| ColinMochrie | one neutron patch in the check queue so far, no heat | 19:29 |
| ColinMochrie | but a boatload of nova patches | 19:29 |
| ColinMochrie | thanks rustlebee | 19:29 |
| ColinMochrie | I think rustlebee decided it is beer o'clock soon and cleared the desk | 19:30 |
| jeblair | rustlebee: thank you :) | 19:30 |
| ttx | Ti'Punch o'clock all day here | 19:30 |
| * jeblair wipes a solitary tear from his eye and gets back to hacking on zuul | 19:31 | |
| *** datsun180b_ has joined #openstack-infra | 19:31 | |
| ColinMochrie | ttx well according to wikipedia you are in the Caribbean on a French speaking island | 19:32 |
| ttx | ColinMochrie: almost | 19:32 |
| ColinMochrie | you are in the water | 19:33 |
| ttx | Caipirinha o'clock is alost an option around here. | 19:33 |
| ColinMochrie | you are in a plane about to land there | 19:33 |
| ttx | also* | 19:33 |
| ColinMochrie | you are in Brazil | 19:33 |
| ttx | ColinMochrie: Between Brazil and the Caribbean islands. | 19:34 |
| ColinMochrie | do you see any of the fruits mordred was talking about the other week, the ones that clarkb's grandparents grow in Hawaii? | 19:34 |
| * ColinMochrie consults a map | 19:34 | |
| ColinMochrie | are you on a boat? | 19:35 |
| ttx | ColinMochrie: no. | 19:35 |
| ColinMochrie | hmmm | 19:35 |
| *** datsun180b has quit IRC | 19:35 | |
| *** datsun180b_ is now known as datsun180b | 19:35 | |
| *** prad_ has joined #openstack-infra | 19:36 | |
| ColinMochrie | Venezuela? | 19:36 |
| ttx | ColinMochrie: French Guiana. | 19:36 |
| ColinMochrie | that was on the list | 19:36 |
| ColinMochrie | nice | 19:36 |
| ColinMochrie | been there before? | 19:36 |
| ttx | ColinMochrie: no, visiting family | 19:36 |
| ColinMochrie | great | 19:37 |
| ColinMochrie | how much longer are you there? | 19:37 |
| ttx | ColinMochrie: a few more days. | 19:38 |
| ColinMochrie | nice | 19:38 |
| ColinMochrie | I know very little about French Guiana - how is the architecture? | 19:38 |
| * ColinMochrie loves architecture | 19:39 | |
| ttx | ColinMochrie: pretty poor. | 19:39 |
| ColinMochrie | :( | 19:39 |
| ColinMochrie | poor as in disrepair? | 19:39 |
| ttx | you could say that. It's like the shores of the Amazonian forest. | 19:40 |
| ColinMochrie | wow | 19:40 |
| ColinMochrie | okay so potentially dramatic and beautiful for natural reasons | 19:40 |
| ColinMochrie | potentially | 19:40 |
| *** dmakogon_ has quit IRC | 19:41 | |
| jeblair | clarkb: static rtt just got a lot worse | 19:43 |
| dkranz | ttx: I will subscribe to stable-maint as well. | 19:44 |
| clarkb | jeblair ok | 19:45 |
| clarkb | maybe they are debugging it? | 19:45 |
| jeblair | clarkb: i checked because i saw a bunch of unstable results, but i think those are actually aborts, so it may not be having an impact; i think i'll leave it for now | 19:46 |
| jeblair | btw, i'm writing the 'ignore results after aborting' patch, which is actually a prereq for 're-run jobs' | 19:46 |
| jeblair | (re-running an aborted job would be, unfortunate) | 19:46 |
| openstackgerrit | Monsyne Dragon proposed a change to openstack/requirements: Add jsonpath-rw https://review.openstack.org/43503 | 19:50 |
| ColinMochrie | 63 in the check queue and 135 events | 19:51 |
| *** prad_ has quit IRC | 19:52 | |
| *** vipul is now known as vipul-away | 19:52 | |
| *** zul has quit IRC | 19:53 | |
| *** yolanda has quit IRC | 19:55 | |
| *** prad_ has joined #openstack-infra | 19:56 | |
| *** nayward has joined #openstack-infra | 20:05 | |
| *** prad_ has quit IRC | 20:08 | |
| *** dkliban has joined #openstack-infra | 20:08 | |
| *** comstud is now known as bearhands | 20:08 | |
| *** vipul-away is now known as vipul | 20:09 | |
| *** lcestari has quit IRC | 20:09 | |
| *** dkliban has quit IRC | 20:12 | |
| *** arezadr has quit IRC | 20:13 | |
| *** arezadr has joined #openstack-infra | 20:13 | |
| *** RyanStiles is now known as bnemec | 20:13 | |
| *** pcrews has joined #openstack-infra | 20:15 | |
| *** woodspa has quit IRC | 20:15 | |
| *** hemna has joined #openstack-infra | 20:15 | |
| *** prad_ has joined #openstack-infra | 20:16 | |
| *** gyee has quit IRC | 20:16 | |
| clarkb | there is a folsom change in the gate | 20:17 |
| jeblair | i wonder how that's going to work out | 20:18 |
| clarkb | jeblair: it is currently building docs. everything else passed | 20:18 |
| jeblair | neato | 20:19 |
| hemna | do we have reviewers for the requirements project in here? | 20:19 |
| clarkb | hemna: some of them | 20:21 |
| hemna | can we get someone to take a look at this please? https://review.openstack.org/#/c/42991/ | 20:21 |
| clarkb | though I think those of us that got core as a result of being infra core try to be hands off | 20:21 |
| hemna | we are being held up in cinder waiting for this guy to land | 20:21 |
| * clarkb looks | 20:21 | |
| *** p5ntangle has joined #openstack-infra | 20:23 | |
| *** prad_ has quit IRC | 20:23 | |
| clarkb | hemna: so I understand, version 1.0.0 is not going to be good enough for havana cinder? and 2.0.0 will be required? | 20:24 |
| hemna | correct | 20:24 |
| clarkb | because as is you should be able to use either | 20:24 |
| clarkb | ok | 20:24 |
| hemna | 1.0.0 won't work for havana, due to changes in the 3par firmware | 20:24 |
| *** pcm_ has joined #openstack-infra | 20:24 | |
| hemna | this package is only for supporting the 3par drivers in cinder | 20:24 |
| hemna | so it doesn't affect anything else | 20:24 |
| hemna | 2.0 is a major change in the api and the firmware | 20:25 |
| hemna | which isn't backward compatible. | 20:25 |
| clarkb | I just wanted to make sure that a more relaxed requirement like is currenlty listed wouldn't be more appropriate | 20:26 |
| clarkb | the old requirement allows both versions | 20:26 |
| *** p5ntangle has quit IRC | 20:27 | |
| hemna | ok thank you | 20:27 |
| hemna | yah we actually don't want the old version for havana | 20:28 |
| *** mrodden1 has joined #openstack-infra | 20:28 | |
| *** mrodden has quit IRC | 20:31 | |
| *** mrmartin has quit IRC | 20:33 | |
| *** changbl has joined #openstack-infra | 20:35 | |
| lifeless | Alex_Gaynor: sorry about the delay; I'm now actually working on the pyp subunit bug. | 20:35 |
| lifeless | Alex_Gaynor: thanks in part to plane delays | 20:35 |
| Alex_Gaynor | lifeless: cool, let me know if I can help | 20:35 |
| lifeless | Alex_Gaynor: nope, all copacetic | 20:36 |
| lifeless | Alex_Gaynor: did you file a bug ? | 20:36 |
| Alex_Gaynor | lifeless: ah no, I didn't, would you like me to? | 20:36 |
| lifeless | nah, I'l do it | 20:36 |
| lifeless | I just didn't see one... | 20:36 |
| lifeless | Alex_Gaynor: the test was to import builtins right? | 20:37 |
| lifeless | (or __builtin__ ) | 20:38 |
| lifeless | Alex_Gaynor: https://bugs.launchpad.net/subunit/+bug/1216101 | 20:38 |
| uvirtbot | Launchpad bug 1216101 in subunit "broken on pypy" [Critical,Triaged] | 20:39 |
| Alex_Gaynor | lifeless: yes, add `try: import __builtin__ as builtins; except ImportError: import builtins` and then replace __builtins__ with that | 20:39 |
| *** kmartin has joined #openstack-infra | 20:41 | |
| lifeless | Alex_Gaynor: hah, no :) | 20:42 |
| Alex_Gaynor | lifeless: ? | 20:42 |
| lifeless | builtins = try_imports(['__builtin__', 'builtins']) | 20:42 |
| Alex_Gaynor | lifeless: Ah yes, that :) | 20:42 |
| lifeless | done | 20:42 |
| *** pabelanger has joined #openstack-infra | 20:43 | |
| lifeless | first off I need to fix an issue in 3.3 so I can test that I don't break 3.x | 20:43 |
| lifeless | then I get to do this | 20:43 |
| lifeless | then some cleanup on the plane | 20:43 |
| lifeless | *then* a release. | 20:43 |
| Alex_Gaynor | cool | 20:44 |
| ColinMochrie | lifeless: where are you flying to/from? | 20:45 |
| *** jjmb has joined #openstack-infra | 20:45 | |
| *** sarob has joined #openstack-infra | 20:45 | |
| *** dkliban has joined #openstack-infra | 20:45 | |
| lifeless | ColinMochrie: CHC->SYD | 20:46 |
| ColinMochrie | how long are you in sydney? | 20:47 |
| lifeless | ~30 hours | 20:47 |
| ColinMochrie | hope it is worth it | 20:47 |
| lifeless | well, it was more worth it before this 4 hour delay | 20:47 |
| ColinMochrie | nice | 20:47 |
| *** gyee has joined #openstack-infra | 20:47 | |
| lifeless | it's only a 3 hour flight | 20:48 |
| lifeless | -> delay is more than the entire trip | 20:48 |
| *** prad_ has joined #openstack-infra | 20:50 | |
| *** prad_ has quit IRC | 20:52 | |
| *** lbragstad has quit IRC | 20:53 | |
| *** SergeyLukjanov has quit IRC | 20:54 | |
| ColinMochrie | :( | 20:54 |
| *** jjmb1 has joined #openstack-infra | 20:55 | |
| *** jjmb has quit IRC | 20:57 | |
| *** psedlak is now known as __psedlak | 20:57 | |
| *** __psedlak is now known as psedlakZzz | 20:58 | |
| *** dims has quit IRC | 20:58 | |
| *** emagana has joined #openstack-infra | 20:59 | |
| *** dims has joined #openstack-infra | 21:00 | |
| *** svarnau has joined #openstack-infra | 21:03 | |
| *** dina_belova has quit IRC | 21:04 | |
| *** mriedem has quit IRC | 21:05 | |
| *** rfolco has quit IRC | 21:05 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Ignore builds once they have been canceled https://review.openstack.org/43537 | 21:05 |
| lifeless | oh yay, another hour delay | 21:07 |
| ColinMochrie | lifeless: are you not even on the plane yet/ | 21:08 |
| lifeless | I may get subunit pushed before I succumb to an alcoholic haze in the lounge | 21:08 |
| ColinMochrie | ? | 21:08 |
| lifeless | ColinMochrie: the plane isn't here yet | 21:08 |
| ColinMochrie | lifeless: maybe you can hire a boat | 21:08 |
| ColinMochrie | lifeless: sounds like Canada | 21:08 |
| *** mrodden1 has quit IRC | 21:08 | |
| lifeless | ColinMochrie: Don't you mean a boot ? :P | 21:08 |
| ColinMochrie | only if you can go oot in it | 21:08 |
| lifeless | ColinMochrie: what happened here is that there was fog last night | 21:09 |
| lifeless | so the inbound planes all ended up 2.5 hours away in AKL | 21:09 |
| ColinMochrie | jeblair: is the code in 43537 already running on zuul now? | 21:09 |
| ColinMochrie | lifeless: ah okay | 21:09 |
| ColinMochrie | fun times for the airlines | 21:09 |
| jeblair | ColinMochrie: nope, zuul is running master | 21:09 |
| ColinMochrie | jeblair: okay thanks | 21:09 |
| lifeless | now they are all trying to get down here, constrained by airport hours in AKL; by morning fog here; by crews etc etc etc | 21:10 |
| lifeless | nightmare time | 21:10 |
| jeblair | ColinMochrie: we're almost getting back to the usual order of things where we review code _before_ putting into production. | 21:10 |
| ColinMochrie | ha ha ha | 21:11 |
| ColinMochrie | oh good | 21:11 |
| ColinMochrie | lifeless: have another drink at the bar | 21:11 |
| ColinMochrie | I'm sure the subunit patch will be...colourful | 21:11 |
| lifeless | I will be colourful | 21:12 |
| lifeless | it will be what it is | 21:12 |
| *** apcruz has quit IRC | 21:14 | |
| clarkb | jeblair: I am trying to catch up :) | 21:16 |
| ColinMochrie | lifeless: ha ha ha | 21:16 |
| jeblair | clarkb: that's still a fix for current operational problems | 21:16 |
| ColinMochrie | jeblair: so 43537 hopefully will reduce the number of LOST logs? | 21:16 |
| jeblair | clarkb: it's just not crazy | 21:16 |
| clarkb | jeblair: yup it is on the list to review | 21:17 |
| jeblair | clarkb: rather, it's a prereq for such a fix | 21:17 |
| lifeless | garh, per-dict and per-set random sort order is sooooo annoying | 21:17 |
| clarkb | lifeless: call sort and burn cpu cycles? | 21:17 |
| jeblair | ColinMochrie: nope, it will mask unpredictable result values from jenkins so only CANCELED is displayed | 21:17 |
| lifeless | clarkb: not in the production code | 21:18 |
| lifeless | clarkb: thankyouverymuch | 21:18 |
| ColinMochrie | jeblair: ah okay so LOST becomes CANCELED but the reason for either is still outstanding | 21:18 |
| *** apcruz has joined #openstack-infra | 21:18 | |
| ColinMochrie | I'm seeing a lot of nova patch failures in the check queue all of a sudden: https://jenkins02.openstack.org/job/gate-tempest-devstack-vm-full/4299/console | 21:19 |
| jeblair | ColinMochrie: i believe that's because people are writing bad code | 21:20 |
| ColinMochrie | okay great | 21:20 |
| ColinMochrie | just so long as it isn't us | 21:20 |
| ColinMochrie | the Friday patches | 21:20 |
| *** sarob has quit IRC | 21:21 | |
| clarkb | jeblair: https://review.openstack.org/#/c/43537/1/tests/test_scheduler.py line 1347 where do the two aborted jobs comes from? | 21:21 |
| jeblair | clarkb: change C test1 and test2 | 21:21 |
| clarkb | jeblair: I would've expected 3 aborted jobs for B test2 and C test1 and test2 | 21:21 |
| clarkb | oh thats right | 21:21 |
| clarkb | B will finish its test | 21:21 |
| jeblair | clarkb: B is a potential future queue head, so it gets to finish all jobs | 21:22 |
| clarkb | ya | 21:22 |
| jeblair | clarkb: (i, uh, had to chase that one down myself too. :) | 21:22 |
| *** xBsd has quit IRC | 21:24 | |
| *** mrodden has joined #openstack-infra | 21:25 | |
| *** melwitt has joined #openstack-infra | 21:26 | |
| clarkb | jeblair: so what does that set the state to when a build is aborted/cancelled? | 21:26 |
| jeblair | clarkb: CANCELED; it's done in the scheduler (that's current behavior, it's just overwritten when actual results arrive) | 21:27 |
| clarkb | I see which is why we only get ABORTED back when a job was running but queued switches to CANCELLED | 21:28 |
| jeblair | clarkb: this causes the overwriting not to happen | 21:28 |
| jeblair | yep | 21:28 |
| *** dprince has quit IRC | 21:28 | |
| jeblair | i mean, it might be nice to have CANCELED and ABORTED, but very few of us actually care about that, so... | 21:28 |
| clarkb | hmm mikal's response to the git.o.o thread may need a diplomatic response | 21:28 |
| clarkb | jeblair: yeah and they confuse most users | 21:29 |
| clarkb | jeblair: https://review.openstack.org/#/c/42709/ may interest you because it potentially affects our puppet changes | 21:30 |
| jeblair | i can respond | 21:30 |
| clarkb | I plan on approving that change today though. It is simple | 21:30 |
| jeblair | clarkb: thx | 21:33 |
| *** dolphm has joined #openstack-infra | 21:34 | |
| lifeless | and huh | 21:36 |
| lifeless | it looks like with a simple 8-way possible sort only 7 sort orders appear regularly. | 21:36 |
| lifeless | 10m of looping and I can't get the 8th to appear. | 21:36 |
| lifeless | I wonder if this has implications for the randomness and thus security of these dicts | 21:36 |
| lifeless | Alex_Gaynor: ^ may interest you | 21:36 |
| clarkb | jeblair: also https://review.openstack.org/#/c/39846/ | 21:38 |
| ColinMochrie | any stats on whether the other 7 sort orders show up in a more or less balanced frequency? | 21:38 |
| lifeless | ColinMochrie: not worth writing home about | 21:39 |
| ColinMochrie | ok | 21:39 |
| lifeless | ColinMochrie: would require more sober thinking than I am capable of right now ;P | 21:39 |
| ColinMochrie | enjoy the beverage | 21:39 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix ACL for Murano projects https://review.openstack.org/41650 | 21:40 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Add more deterministic port selection for Gerrit. https://review.openstack.org/42581 | 21:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Redirect stderr to stdout for puppet-lint https://review.openstack.org/42709 | 21:41 |
| Alex_Gaynor | lifeless: yeah, dicts are not super well randomized (actually on 2.x randomization isn't even on by default), when it's used there are only 256 possible states for it to be in, instead of the theoretical LONG_MAX amount | 21:42 |
| ColinMochrie | I just went to look at the summit presentations to consider voting and I can only look at one presentation write up at a time. Pretty frustrating trying to compare all the talks in a category and select one or two favourites. | 21:42 |
| lifeless | ColinMochrie: you're not meant to do that :P | 21:42 |
| ColinMochrie | am I not? | 21:42 |
| ColinMochrie | what I am supposed to do? | 21:42 |
| ColinMochrie | cause I closed the page | 21:43 |
| lifeless | assess each talk on it's merits | 21:43 |
| lifeless | vote for as many as you have time to review | 21:43 |
| lifeless | everyone gets random talks to review, - it's crowdsourced | 21:43 |
| ColinMochrie | yes, so I have to read the write ups to assess them | 21:43 |
| lifeless | right | 21:43 |
| ColinMochrie | I have to read all the ones in a given category to get a feel for what the category title it trying to convey | 21:43 |
| ColinMochrie | then I read them again for merit | 21:44 |
| ColinMochrie | the category titles are meaningless to me | 21:44 |
| ColinMochrie | it is only through context that I understand them | 21:44 |
| jeblair | ColinMochrie: it's possible they are meaningless to everyone | 21:44 |
| ColinMochrie | yeah | 21:44 |
| lifeless | some of them are truely shite :) | 21:44 |
| ColinMochrie | so I have no basis to evaluating a talk | 21:44 |
| ColinMochrie | sure | 21:44 |
| ColinMochrie | so with nothing to compare them to, I ask myself - is listening to this better than the hallway track | 21:45 |
| jeblair | ColinMochrie: you could go with "is this interesting?" "would i attend it?" "do i understand what the proposal is even trying to say?", all of whech are more absolute than relative. | 21:45 |
| jeblair | ColinMochrie: that is an excellent metric. | 21:45 |
| ColinMochrie | I'd end up just voting for all or none of them | 21:46 |
| ColinMochrie | when I shop I like to assess the field before I make a decision | 21:46 |
| jeblair | ColinMochrie: (very similar to Siskel's "Is the movie that I am watching as interesting as a documentary of the same actors having lunch together?") | 21:46 |
| ColinMochrie | ha ha ha | 21:46 |
| ColinMochrie | I guess I can give that a try | 21:47 |
| lifeless | Alex_Gaynor: ok, just pushed up a working pypy patch. pypy seems kinda slow though :( | 21:47 |
| lifeless | Alex_Gaynor: 1s vs 0.3 to run the tests | 21:48 |
| clarkb | vishy did some pypy + nova stuff and found there wasn't much difference in speed | 21:49 |
| lifeless | clarkb: I have a knack of finding idioms pypy is slower than CPython with | 21:49 |
| lifeless | clarkb: lmirror for instance triggers a gc pathological case when mirroring an Ubuntu archive. | 21:50 |
| lifeless | clarkb: there's a bug for that one somewhere | 21:50 |
| ColinMochrie | the css seems to be broken on this page for me: http://www.openstack.org/rate/Presentation/learn-openstack-from-trystack other pages seem okay | 21:50 |
| ColinMochrie | I have just opened a bunch of tabs in my browser | 21:50 |
| clarkb | broken for me too | 21:51 |
| mgagne | broken too here | 21:52 |
| ColinMochrie | who is in charge of css for the website? | 21:52 |
| dolphm | are global requirements automatically installed, or are they just made available in the pypi mirror? | 21:52 |
| ColinMochrie | does it have a repo? | 21:52 |
| ColinMochrie | found a winner: http://www.openstack.org/rate/Presentation/jenkins-failed-my-patch-what-do-i-do | 21:53 |
| ColinMochrie | hope to see a few of you there | 21:53 |
| clarkb | dolphm: they are made available to the pypi mirror and substituted in for requirements lists in the tempest tests | 21:55 |
| clarkb | dolphm: they are not substituted in in the unittests | 21:55 |
| clarkb | dolphm: that tests that we can install all of openstack with one set of requirements which is part of the goal behind the global requirements | 21:55 |
| dolphm | clarkb: thank you | 21:56 |
| Alex_Gaynor | lifeless: tests are not a benchmark: http://alexgaynor.net/2013/jul/15/your-tests-are-not-benchmark/ | 21:56 |
| Alex_Gaynor | lifeless: also, awesome! | 21:56 |
| lifeless | Alex_Gaynor: they aren't! But I care that they are fast ;) | 21:57 |
| Alex_Gaynor | lifeless: that's fair :) | 21:57 |
| ColinMochrie | not sure about this one, how much fun is it watching you eat, morganfainberg? http://www.openstack.org/rate/Presentation/articulating-the-cloud-value-proposition-how-to-communicate-effectively-about-iaas-paas-saas-and-the-rest-of-it | 21:57 |
| morganfainberg | ColinMochrie: watching me eat? | 21:58 |
| dstufft | Alex_Gaynor: lifeless wtb siphash backed dictionaries for Python :[ | 21:58 |
| lifeless | Alex_Gaynor: 0.0.14 on pypi | 21:58 |
| ColinMochrie | <ColinMochrie> so with nothing to compare them to, I ask myself - is listening to this better than the hallway track | 21:58 |
| ColinMochrie | <jeblair> ColinMochrie: (very similar to Siskel's "Is the movie that I am watching as interesting as a documentary of the same actors having lunch together?") | 21:58 |
| lifeless | Alex_Gaynor: you have < 15 to tell me how I broke it before I hope on a plane. I hope. | 21:58 |
| lifeless | m, that is | 21:59 |
| jeblair | no way that was 4 hours | 21:59 |
| morganfainberg | ColinMochrie: ahh i see now | 21:59 |
| ColinMochrie | :D | 21:59 |
| lifeless | jeblair: it's now 1000 | 22:00 |
| ColinMochrie | so are you entertaining when you eat? | 22:00 |
| lifeless | jeblair: I arrived at the airport at 0445 | 22:00 |
| jeblair | i guess i'm having fun :) | 22:00 |
| lifeless | jeblair: :) | 22:00 |
| morganfainberg | ColinMochrie: not so much when i am chewing, but can be entertaining over a lunch table | 22:00 |
| lifeless | jeblair: you also didn't observe the entire delay, only from when I hit the lounge and started whinging | 22:00 |
| morganfainberg | ColinMochrie: :P | 22:00 |
| ColinMochrie | morganfainberg: okay thanks for the metric | 22:00 |
| ColinMochrie | :D | 22:00 |
| ColinMochrie | here is my metric, is the description of the talk compelling enough for me to play the password reset rodeo and sign in to vote | 22:03 |
| *** sdake_ has quit IRC | 22:03 | |
| lifeless | Alex_Gaynor: how's it looking? | 22:04 |
| *** burt has quit IRC | 22:04 | |
| Alex_Gaynor | lifeless: oh, I haven't tested, I should do that :D | 22:04 |
| pleia2 | jeblair: unless you're on it already, I can get the latest question on -dev (gerrit remote - no, it's still review.o.o) | 22:05 |
| jeblair | pleia2: your turn :) | 22:06 |
| *** dkliban has quit IRC | 22:07 | |
| Alex_Gaynor | lifeless: ok it fails with `residue: '0\r\n'`, but progress at least :) | 22:07 |
| Alex_Gaynor | lifeless: the memoryview.decode issue at elast is fixed, will investigate this a bit later | 22:08 |
| *** datsun180b has quit IRC | 22:09 | |
| lifeless | Alex_Gaynor: something funny in chunk decoding then | 22:10 |
| lifeless | Alex_Gaynor: get me a SSCCE / bug / both... | 22:10 |
| Alex_Gaynor | lifeless: yup, will do | 22:10 |
| *** thedodd has quit IRC | 22:17 | |
| ColinMochrie | I'm off uptown for dinner | 22:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Re-launch a job if the worker fails to run it https://review.openstack.org/43547 | 22:21 |
| *** mgagne has quit IRC | 22:21 | |
| jeblair | clarkb: ^ that's an 11 line change. | 22:22 |
| jeblair | i like this new scheduler. :) | 22:22 |
| *** dolphm has quit IRC | 22:23 | |
| *** rcleere has quit IRC | 22:25 | |
| clarkb | looking | 22:29 |
| *** nayward has quit IRC | 22:30 | |
| *** changbl has quit IRC | 22:33 | |
| clarkb | jeblair: does it retry the job automagically because you have removed the old build from the build set? | 22:34 |
| *** ftcjeff has quit IRC | 22:37 | |
| clarkb | jeblair: if not I feel like we are missing a change to scheduler.py | 22:38 |
| *** boris-42 has quit IRC | 22:41 | |
| *** rnirmal has quit IRC | 22:41 | |
| clarkb | jeblair: I think I see it sched.onBuildCompleted puts the event on the result_event_queue. Then in process_event_queue pipeline.onBuildCompleted is called which will call pipeline.processQueue | 22:42 |
| *** reed has quit IRC | 22:52 | |
| pabelanger | So, been look at nodepool breifly, if I understand right, it will actually launch an instance in the cloud, installed jenkins-slave and hook it back in the master? | 22:56 |
| clarkb | pabelanger: correct | 22:57 |
| clarkb | pabelanger: it has taken the node management that was in devstack gate and made it a daemon so that we don't have to coordinate state across jenkins jobs | 22:57 |
| pabelanger | clarkb, okay, so the part I haven't figured out, is the nodes lifetime. Once launched, it is persistent? Or does it get destroyed at some point in time? | 22:57 |
| pabelanger | Sorry, don't know much about devstack | 22:58 |
| clarkb | pabelanger: it gets destroyed a minute after running a job on that node | 22:58 |
| pabelanger | that is pretty cool | 22:58 |
| pabelanger | infact hotness | 22:58 |
| clarkb | pabelanger: though that will probably end up being more configurable so that we can use nodepool nodes for unittests and other things | 22:58 |
| clarkb | also very possible that this is configurable today | 22:59 |
| clarkb | and I missed that feature | 22:59 |
| jeblair | not configurable yet | 23:00 |
| jeblair | but if you look at the zmq listener, that gets the result information from jenkins and is how it decides what to do when a build on a node it manages is complete | 23:01 |
| jeblair | so, pretty easy to make it do whatever there | 23:01 |
| jeblair | clarkb: you are correct about zuul | 23:01 |
| pabelanger | jeblair, clarkb: Nice... so, nodepool lists to zuul? | 23:03 |
| pabelanger | Oh, jenkins | 23:04 |
| *** pcrews has quit IRC | 23:04 | |
| *** sdake_ has joined #openstack-infra | 23:04 | |
| clarkb | yup it is on the zmq event bus. I think jeblair considered more tightly coupling it to zuul | 23:04 |
| pabelanger | still cool | 23:07 |
| pabelanger | will have to find some time to play with it to see how it works | 23:08 |
| *** gyee has quit IRC | 23:11 | |
| *** fifieldt has joined #openstack-infra | 23:14 | |
| fifieldt | hi infra, I know you're crazily busy, and this is a low priority patch, but if anyone felt like ticking approve for one that already has 2 +2s and 3 +1s, there's one here: https://review.openstack.org/#/c/39846/ :) | 23:15 |
| clarkb | fifieldt: I was hoping jeblair would take a look at it | 23:16 |
| clarkb | fifieldt: but I can approve it | 23:16 |
| clarkb | jeblair: ^ | 23:16 |
| fifieldt | oh, that's fine | 23:16 |
| jeblair | oh, can do | 23:16 |
| * fifieldt is not a jenkins-job-builder expert | 23:17 | |
| fifieldt | so it's important it doesn't bring things crashing down | 23:17 |
| fifieldt | :D | 23:17 |
| clarkb | fifieldt: we actually have a reasonable amount of testing around that so I am not worried about that happening :) | 23:17 |
| fifieldt | :D | 23:17 |
| fifieldt | you guys are awesome | 23:17 |
| fifieldt | I will be in SFO from 4-8 sept if you want to claim free beers | 23:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Add build job for Japanese Operations Guide https://review.openstack.org/39846 | 23:19 |
| fifieldt | oh, cool! | 23:20 |
| fifieldt | thank you so much | 23:20 |
| fifieldt | now to see if we can get funky kanji happening | 23:20 |
| *** senk has quit IRC | 23:21 | |
| *** changbl has joined #openstack-infra | 23:37 | |
| *** pcrews has joined #openstack-infra | 23:38 | |
| *** emagana has quit IRC | 23:51 | |
| *** Ryan_Lane has quit IRC | 23:54 | |
| *** Ryan_Lane1 has joined #openstack-infra | 23:54 | |
| jeblair | docs-draft finished 1st rsync | 23:55 |
| jeblair | starting a new one | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!