| jeblair | fungi: you may also be interested in 130355 | 00:00 |
|---|---|---|
| *** armax has joined #openstack-infra | 00:00 | |
| *** ddieterly has quit IRC | 00:00 | |
| clarkb | so what do people think about my trying to replace nodepool tomorrow? is that a reasonable time to do it? | 00:01 |
| jeblair | clarkb: ooh, what are we replacing it with? :) | 00:01 |
| clarkb | jeblair: a trusty server | 00:01 |
| clarkb | sorry the nodepool service remains but it needs a new home | 00:01 |
| fungi | jeblair: another nodepool! ba-bum-tsh! | 00:01 |
| clarkb | I think my plan is start spinning up new node. stop nodepool service on old server when new server is puppeting, update dns, then hopefully everything just works | 00:02 |
| clarkb | at that point clean up aliens | 00:02 |
| jogo | suds-jurko' is not a global requirement but it should be,something went wrong | 00:02 |
| jeblair | clarkb: wfm | 00:02 |
| fungi | clarkb: sounds good to me. should be minimally disruptive if the gods smile upon us | 00:02 |
| jogo | oslo.vmware | 00:02 |
| clarkb | mordred: where are we on booting nodes with volumes preattached an all that? | 00:02 |
| mtreinish | jogo: the fix is already in progress | 00:02 |
| clarkb | jogo: icehouse? | 00:02 |
| *** wenlock has quit IRC | 00:02 | |
| jogo | mtreinish: ahh cool is there a bug for it | 00:03 |
| jogo | mtreinish: getting our classification rate up | 00:03 |
| mtreinish | jogo: don't think so | 00:03 |
| mtreinish | dims: ^^^ ? | 00:03 |
| *** emagana has quit IRC | 00:03 | |
| *** emagana has joined #openstack-infra | 00:04 | |
| jogo | mtreinish: do you know when that was fixed patch wise? | 00:04 |
| mtreinish | like within the last 2 hours: https://review.openstack.org/#/q/If91a87ede96a82a2a0d181ee0d4fdc102dc0299d,n,z | 00:04 |
| *** bhuvan_ has joined #openstack-infra | 00:05 | |
| mtreinish | the icehouse change landed like 2 min ago | 00:05 |
| dims | icehouse just merged, need to reverify the juno one | 00:05 |
| mtreinish | dims: the juno one shouldn't be blocking anyone in the gate it was just the icehosue causing the failures | 00:05 |
| *** bhuvan has quit IRC | 00:06 | |
| *** baoli has quit IRC | 00:06 | |
| anteaya | mikal: this patch adds a voting ironic job on nova, please review when you have a chance: https://review.openstack.org/#/c/126627/ | 00:07 |
| dims | mtreinish: tristan was saying something else broke, so he backported to juno as well | 00:07 |
| mtreinish | dims: yah, it's good to do the backport because it'll break when you push the next release | 00:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-httpd: Add .gitreview https://review.openstack.org/135781 | 00:07 |
| fungi | mtreinish: dims: not just tristanC but also jungleboyj | 00:07 |
| mtreinish | but, people were getting confused because juno grenade was failing | 00:08 |
| mtreinish | but that was still caused by icehouse | 00:08 |
| *** toolzark has joined #openstack-infra | 00:08 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/elastic-recheck: Add query for bug 1394408 https://review.openstack.org/135789 | 00:08 |
| uvirtbot | Launchpad bug 1394408 in oslo.vmware "suds-jurko' is not a global requirement " [Undecided,Fix committed] https://launchpad.net/bugs/1394408 | 00:08 |
| *** emagana has quit IRC | 00:08 | |
| *** Sukhdev has quit IRC | 00:08 | |
| jogo | mtreinish: ^ | 00:08 |
| fungi | mtreinish: oh, right, the one jungleboyj linked was indeed a grenade job | 00:08 |
| dims | mtreinish: i did mention i wanted the icehouse one to merge :) | 00:08 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Fix grenade direction for grenade-dsvm-ironic https://review.openstack.org/135790 | 00:09 |
| *** oomichi_ has joined #openstack-infra | 00:09 | |
| mtreinish | jogo: +Ad | 00:10 |
| dims | mtreinish: want to block it? https://review.openstack.org/#/c/135721/ | 00:11 |
| jogo | mtreinish: thanks looks like that caused a lot of failures | 00:11 |
| wendar | Is there any good reason I should be getting hostname resolution errors on review.openstack.org when I try to run 'git review -s' in a fresh checkout? | 00:11 |
| mtreinish | dims: no it's fine, it's a premptive fix | 00:11 |
| wendar | (Discovered on gertty, but seems to be on every fresh checkout.) | 00:12 |
| mtreinish | if we don't do this then on the next oslo.vmware release things would explode | 00:12 |
| dims | mtreinish: cool | 00:12 |
| clarkb | wendar: maybe local name resolution problems? | 00:12 |
| openstackgerrit | Merged stackforge/gertty: Allow specifying the path to CA certificate bundle https://review.openstack.org/130355 | 00:12 |
| fungi | wendar: hostname resolution errors? can you provide the exact error message? | 00:12 |
| clarkb | wendar: I have not seen those issues | 00:12 |
| wendar | Exception: Could not connect to gerrit at ssh://wendar@review.openstack.org:29418/stackforge/gertty.git | 00:12 |
| wendar | thrown on line 344 of /usr/bin/git-review | 00:13 |
| fungi | hrm, that doesn't explicitly say it had trouble with name resolution though | 00:13 |
| *** ryanpetrello has joined #openstack-infra | 00:13 | |
| wendar | fungi: right, I'm getting that by expanding out the ssh command it's running and trying it in verbose | 00:14 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for bug 1394408 https://review.openstack.org/135789 | 00:14 |
| uvirtbot | Launchpad bug 1394408 in oslo.vmware "suds-jurko' is not a global requirement " [Undecided,Fix committed] https://launchpad.net/bugs/1394408 | 00:14 |
| fungi | wendar: can you try to 'ssh -p 29418 wendar@review.openstack.org gerrit ls-projects' and see what specific error you get? | 00:14 |
| *** adalbas has quit IRC | 00:14 | |
| wendar | got it, host key changed | 00:15 |
| *** dims has quit IRC | 00:15 | |
| fungi | wendar: ayup, probably the first time you've ssh'd to it from there since we regenerated all our keys for heartbleed fun and excitement | 00:15 |
| *** dims has joined #openstack-infra | 00:15 | |
| *** yamamoto has joined #openstack-infra | 00:16 | |
| mtreinish | nibalizer: I updated https://review.openstack.org/122311 per your comments, but I'm sure I still screwed things up... | 00:16 |
| wendar | fungi: yeah, thanks! | 00:16 |
| nibalizer | mtreinish: ill have to look tommorrow | 00:18 |
| nibalizer | im headed for bed | 00:18 |
| fungi | i too am knocking off. another day, another pile of things i probably won't get around to fixing | 00:19 |
| clarkb | fungi: hey you got to the lp stuff | 00:19 |
| krotscheck | Ok, done with all the things | 00:19 |
| krotscheck | jeblair, fungi: Any last comments? If so I’ll just send this email out verbatim. | 00:19 |
| clarkb | we are storyboarding now. thats something | 00:19 |
| mtreinish | nibalizer: ok, thanks. | 00:19 |
| fungi | clarkb: eventually, yes ;) | 00:19 |
| *** abhishekk_ has joined #openstack-infra | 00:19 | |
| fungi | krotscheck: fire when ready | 00:19 |
| clarkb | krotscheck: I read it in an earlier form and it looked good | 00:19 |
| clarkb | I am sure that fungi and jeblair suggested edits that made it even better | 00:20 |
| fungi | i won't make any guarantees that my edits made things in any way better | 00:20 |
| *** wuhg has joined #openstack-infra | 00:20 | |
| *** abhishekk__ has joined #openstack-infra | 00:20 | |
| *** yamamoto has quit IRC | 00:20 | |
| abhishekk__ | hi, can any one help me with this error | 00:20 |
| abhishekk__ | https://review.openstack.org/#/c/133858/ | 00:20 |
| *** AJaeger has joined #openstack-infra | 00:21 | |
| krotscheck | Here goes nothing. | 00:21 |
| wuhg | i have a new patch for 135273 | 00:21 |
| clarkb | abhishekk__: I think that is the oslo.vmware thing. its beeing worked | 00:21 |
| fungi | abhishekk__: grenade was failing on stable/juno because of a dependency change in icehouse. it's being worked on | 00:21 |
| wuhg | where can i get the testing log result for the new patch | 00:22 |
| clarkb | krotscheck: such excite | 00:22 |
| clarkb | wuhg: "jenkins" leaves comments on the changes with links to all of the job logs | 00:22 |
| fungi | abhishekk__: looks like https://review.openstack.org/135692 (fixing it) merged about 20 minutes ago, so should be safe to recheck | 00:22 |
| *** abhishekk___ has joined #openstack-infra | 00:23 | |
| *** abhishekk_ has quit IRC | 00:23 | |
| wuhg | clarkb:yes , but there is no for gate-horizon-dsvm-integration | 00:23 |
| *** esker has quit IRC | 00:24 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add mysql-proxy to enable read-only access to a db https://review.openstack.org/122311 | 00:24 |
| clarkb | wuhg: that test apperas to be "experimental" you have to explicitly request that those tests run by leaving a comment of "check experimental" | 00:24 |
| abhishekk___ | hey all, I am getting this error in gate job, "'suds-jurko' is not a global requirement but it should be, something went wrong" | 00:24 |
| abhishekk___ | can any one please help me how to get rid of this error | 00:25 |
| *** abhishekk__ has quit IRC | 00:25 | |
| fungi | abhishekk___: grenade was failing on stable/juno because of a dependency change in icehouse. https://review.openstack.org/135692 (fixing it) merged about 20 minutes ago, so should be safe to recheck | 00:25 |
| clarkb | fungi: too slow | 00:25 |
| *** amitgandhinz has joined #openstack-infra | 00:25 | |
| abhishekk___ | fungi: so should I reverify now? | 00:26 |
| fungi | abhishekk___: or recheck. they do exactly the same thing these days | 00:26 |
| *** adalbas has joined #openstack-infra | 00:26 | |
| abhishekk___ | fungi: thank you | 00:27 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Update module issues_url & include storyboard https://review.openstack.org/135452 | 00:28 |
| wuhg | clarkb:Jenkins will leave comments with link to experimental job log ? | 00:28 |
| AJaeger | propose-requirements fails on Icehouse now - what has been changed? https://jenkins.openstack.org/job/propose-requirements-updates/292/console | 00:28 |
| *** amitgandhinz has quit IRC | 00:30 | |
| * AJaeger backports fungi's patch for renaming... | 00:31 | |
| clarkb | wuhg: yes | 00:31 |
| AJaeger | requiremnts backport for icehouse https://review.openstack.org/135796 - still I wonder why we didn't notice that before... | 00:32 |
| wuhg | clarkb:http://logs.openstack.org/73/135273 , 73=horizon? | 00:34 |
| clarkb | wuhg: I don't understand the question. There will be a comment from jenkins with the link to the specific job when it is done running | 00:34 |
| jeblair | wuhg: no, 73 == last 2 digits of change number for directory hashing purposes | 00:34 |
| jeblair | wuhg: logs.o.o/YZ/ABCDYZ | 00:35 |
| wuhg | ok ,thanks jeblair | 00:35 |
| *** shashankhegde has quit IRC | 00:37 | |
| *** shashankhegde has joined #openstack-infra | 00:37 | |
| wuhg | clarkb:many thanks | 00:38 |
| *** ryanpetrello has quit IRC | 00:38 | |
| wuhg | clarkb:jenkins comments only link to the lastest job ,right? | 00:40 |
| *** yamamoto has joined #openstack-infra | 00:41 | |
| clarkb | wuhg: yes, but you get the history in the comments | 00:41 |
| wuhg | for example , i have 4 patch, jenis doesn't link to the first job or senond job | 00:41 |
| clarkb | wuhg: it does if you click the toggle CI button at the bottom of the page and look at the comment history | 00:41 |
| clarkb | but the table at the top of the page is latest patchset only | 00:41 |
| *** AJaeger has quit IRC | 00:42 | |
| *** liam_ has joined #openstack-infra | 00:43 | |
| *** liam_ is now known as Guest89217 | 00:44 | |
| *** zul has quit IRC | 00:44 | |
| *** Guest89217 has quit IRC | 00:44 | |
| *** otherwiseguy has joined #openstack-infra | 00:46 | |
| *** homeless1 has joined #openstack-infra | 00:46 | |
| *** homeless1 has left #openstack-infra | 00:48 | |
| wuhg | clarkb:toggle CI works, thanks again | 00:48 |
| *** homeless has quit IRC | 00:48 | |
| *** rmcall has quit IRC | 00:50 | |
| *** alex7376 has joined #openstack-infra | 00:50 | |
| *** alex7376 is now known as asettle | 00:50 | |
| *** toolzark has quit IRC | 00:50 | |
| openstackgerrit | Allison Randal proposed stackforge/gertty: Sample color palette for inline review comments https://review.openstack.org/135799 | 00:51 |
| *** bhuvan_ has quit IRC | 00:51 | |
| *** bhuvan has joined #openstack-infra | 00:52 | |
| *** abhishekk___ has quit IRC | 00:54 | |
| *** unicell has joined #openstack-infra | 00:57 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: Dirty hack to enable non shared storage live migration https://review.openstack.org/135802 | 00:58 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: Dirty hack to enable non shared storage live migration https://review.openstack.org/135802 | 01:00 |
| *** cpowell has joined #openstack-infra | 01:02 | |
| *** zz_dimtruck is now known as dimtruck | 01:02 | |
| *** reed has quit IRC | 01:05 | |
| clarkb | jeblair: zuul is running experimental jobs for two patchsets on the same change | 01:05 |
| clarkb | jogo: ^ did you do anything funny with that chnage? | 01:05 |
| fungi | clarkb: do we lack dequeue on new patchset for that pipeline maybe? | 01:07 |
| *** cpowell has quit IRC | 01:07 | |
| clarkb | maybe? I thought we dequeued properly though. Something to investigate when it isn't the end of a day | 01:07 |
| *** yamamoto has quit IRC | 01:08 | |
| *** yamamoto has joined #openstack-infra | 01:09 | |
| fungi | yeah, that's default and i don't see us overriding it to false in layout.yaml | 01:10 |
| fungi | agreed, for another day | 01:10 |
| fungi | night all | 01:10 |
| clarkb | good night | 01:10 |
| *** yamamoto has quit IRC | 01:10 | |
| *** JayJ has quit IRC | 01:14 | |
| *** unicell has quit IRC | 01:15 | |
| *** AlexF has joined #openstack-infra | 01:16 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:17 | |
| *** gyee has quit IRC | 01:18 | |
| *** unicell has joined #openstack-infra | 01:20 | |
| *** mmaglana has quit IRC | 01:21 | |
| *** otherwiseguy has quit IRC | 01:23 | |
| *** dims has quit IRC | 01:24 | |
| *** dims has joined #openstack-infra | 01:24 | |
| *** achanda has quit IRC | 01:25 | |
| *** xchu has joined #openstack-infra | 01:26 | |
| wuhg | hi , all i have question on let (Code Review) [review@openstack.org] send out mail like the folowing | 01:27 |
| *** xchu has quit IRC | 01:27 | |
| wuhg | Hello Jenkins, Vlad Okhrimenko, Jason E. Rist, Tomáš Nová?ik, Daniel Korn, Ravikumar Venkatesan, | 01:27 |
| wuhg | I'd like you to reexamine a change. Please visit | 01:27 |
| wuhg | https://review.openstack.org/99329 | 01:27 |
| *** melwitt has quit IRC | 01:28 | |
| wuhg | this mail is send out automatically when a patch is commit for review | 01:29 |
| jogo | clarkb: two different versions of the patch | 01:30 |
| *** ivar-lazzaro has quit IRC | 01:31 | |
| *** sarob has quit IRC | 01:32 | |
| *** sarob_ has joined #openstack-infra | 01:32 | |
| clarkb | jogo ya but only one should be testes | 01:32 |
| clarkb | wow that tyop | 01:33 |
| clarkb | tested | 01:33 |
| * clarkb blames being on a bus | 01:33 | |
| jogo | clarkb: yeah I agree | 01:33 |
| devananda | afaict, gertty does not make a review visible again when someone comments on it (eg, when the author replies to my comment) -- is there a way to look for that? | 01:37 |
| *** VijayTripathi has joined #openstack-infra | 01:44 | |
| *** mmaglana has joined #openstack-infra | 01:48 | |
| *** packet has joined #openstack-infra | 01:51 | |
| *** JayJ has joined #openstack-infra | 01:51 | |
| *** yaguang has joined #openstack-infra | 01:53 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Enable live block migration https://review.openstack.org/135768 | 01:56 |
| *** radez_g0n3 is now known as radez | 01:56 | |
| *** esker has joined #openstack-infra | 01:56 | |
| *** esker has quit IRC | 01:56 | |
| *** yjiang5 is now known as yjiang5_away | 01:57 | |
| *** SumitNaiksatam has quit IRC | 01:57 | |
| *** AlexF has quit IRC | 01:58 | |
| *** alexpilotti has quit IRC | 01:58 | |
| *** nosnos has joined #openstack-infra | 01:59 | |
| *** alexpilotti has joined #openstack-infra | 02:00 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 02:00 | |
| *** alexpilotti has quit IRC | 02:01 | |
| *** alexpilotti has joined #openstack-infra | 02:02 | |
| *** iain-king has joined #openstack-infra | 02:02 | |
| *** alexpilotti has quit IRC | 02:03 | |
| *** alexpilotti has joined #openstack-infra | 02:04 | |
| *** alexpilotti has quit IRC | 02:06 | |
| *** mmaglana has quit IRC | 02:06 | |
| *** viglesias has joined #openstack-infra | 02:08 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Omit ironic agent_ssh jobs from 139099 https://review.openstack.org/135770 | 02:08 |
| *** mudassirlatif has quit IRC | 02:11 | |
| *** dmsimard_away is now known as dmsimard | 02:18 | |
| *** dkranz has joined #openstack-infra | 02:19 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Import puppet-openstack-cloud to Stackforge https://review.openstack.org/135729 | 02:23 |
| jogo | oh fun a nova patch that references nova-ci | 02:25 |
| jogo | openstack-infra * | 02:25 |
| *** otherwiseguy has joined #openstack-infra | 02:27 | |
| clarkb | jogo ? | 02:27 |
| jogo | https://review.openstack.org/#/c/126603/6 | 02:27 |
| *** ivar-lazzaro has joined #openstack-infra | 02:28 | |
| clarkb | nice | 02:28 |
| *** dmsimard is now known as dmsimard_away | 02:29 | |
| jogo | yeah | 02:30 |
| *** packet has quit IRC | 02:30 | |
| *** bhuvan has quit IRC | 02:31 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch oslo.vmware from py33 to py34 https://review.openstack.org/135822 | 02:32 |
| *** Longgeek has joined #openstack-infra | 02:33 | |
| dims | hehe, circle of life :) | 02:33 |
| jogo | dims: yup! | 02:35 |
| *** Longgeek has quit IRC | 02:40 | |
| *** oomichi_ has quit IRC | 02:41 | |
| *** viglesias has quit IRC | 02:44 | |
| *** viglesias has joined #openstack-infra | 02:44 | |
| *** dimtruck is now known as zz_dimtruck | 02:47 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Revert "Exclude testtools 1.4.0" https://review.openstack.org/135828 | 02:49 |
| *** doug-fish1 has left #openstack-infra | 02:49 | |
| *** shashankhegde has quit IRC | 02:51 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 02:54 | |
| *** sarob_ has quit IRC | 02:55 | |
| *** dkranz has quit IRC | 02:55 | |
| jkt | ugh, am I super-stupid, or is the zuul's embedded gearman really limited to IPv4? | 02:55 |
| *** viglesias has quit IRC | 02:55 | |
| *** viglesias has joined #openstack-infra | 02:56 | |
| clarkb | jkt it might be? do you have any more info? its certainly a bug if it is | 02:56 |
| *** KanagarajM has joined #openstack-infra | 02:56 | |
| jkt | clarkb: looking at my ~/.local/lib/python2.7/site-packages/gear/__init__.py , the initialization sequence seems rather wonky -- it appears to bind to a first available socket | 02:56 |
| jkt | clarkb: but it's almost 4am here, so I might be missing something very obvious | 02:57 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add python34 check for taskflow https://review.openstack.org/135829 | 02:59 |
| *** Ryan_Lane has quit IRC | 03:00 | |
| *** atan8 has quit IRC | 03:03 | |
| clarkb | oh I see ya maybe? /me does some quick testing | 03:03 |
| *** rmcall has joined #openstack-infra | 03:04 | |
| jkt | >>> socket.getaddrinfo(None, 12345, socket.AF_UNSPEC, socket.SOCK_STREAM, 0) | 03:05 |
| jkt | [(10, 1, 6, '', ('::1', 12345, 0, 0)), (2, 1, 6, '', ('127.0.0.1', 12345))] | 03:05 |
| jkt | >>> socket.getaddrinfo(None, 12345, socket.AF_UNSPEC, socket.SOCK_STREAM, 0, socket.AI_PASSIVE) | 03:05 |
| jkt | [(2, 1, 6, '', ('0.0.0.0', 12345)), (10, 1, 6, '', ('::', 12345, 0, 0))] | 03:05 |
| jkt | w-t-f | 03:05 |
| *** yaguang has quit IRC | 03:06 | |
| *** yaguang has joined #openstack-infra | 03:06 | |
| jkt | clarkb: https://sourceware.org/bugzilla/show_bug.cgi?id=9981 | 03:07 |
| uvirtbot | jkt: Error: Could not parse XML returned by sourceware.org: HTTP Error 404: Not Found | 03:07 |
| *** dims has quit IRC | 03:08 | |
| *** dims has joined #openstack-infra | 03:08 | |
| *** adalbas has quit IRC | 03:08 | |
| clarkb | oh so its a glibc thing? | 03:09 |
| clarkb | fun | 03:09 |
| clarkb | jkt: but ya definitely a bug that should likely be fixed | 03:09 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 03:11 |
| *** krtaylor has quit IRC | 03:12 | |
| *** viglesias has quit IRC | 03:12 | |
| *** HenryG has quit IRC | 03:15 | |
| jkt | I should probably switch to a proper gear server anyway -- "For testing, not for real use under load." | 03:17 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add python34 check for taskflow https://review.openstack.org/135829 | 03:17 |
| *** mmaglana has joined #openstack-infra | 03:17 | |
| clarkb | jkt: that should probably be updated | 03:18 |
| clarkb | zuul won't work properly without gear iirc | 03:18 |
| clarkb | and it works just fine | 03:18 |
| *** HenryG has joined #openstack-infra | 03:18 | |
| *** zz_dimtruck is now known as dimtruck | 03:19 | |
| jkt | clarkb: would you accept a patch which passes the hostname of the zuul server to gear so that it uses a correct AF? | 03:19 |
| jkt | clarkb: my system, centos7, apparently returns AF_INET as a first entry, and therefore I cannot get it to work over ipv6 without patching this | 03:20 |
| clarkb | ya I think passing the host/ip to bind to is a reasonable thing to do | 03:20 |
| jkt | (it's an ipv6-only system, more or less) | 03:20 |
| clarkb | jkt: if its v6 only you should lose the v4 stack :P | 03:21 |
| jkt | seems like I have a homework for tomorrow | 03:21 |
| clarkb | (that also solves the problem) | 03:21 |
| *** mmaglana has quit IRC | 03:22 | |
| clarkb | but ya definitely a bug and a way to be specific seems like a good way to fix it | 03:22 |
| *** nosnos has quit IRC | 03:24 | |
| *** krtaylor has joined #openstack-infra | 03:25 | |
| *** dimtruck is now known as zz_dimtruck | 03:25 | |
| *** adalbas has joined #openstack-infra | 03:25 | |
| *** yamahata has joined #openstack-infra | 03:27 | |
| *** rushiagr_away is now known as rushiagr | 03:27 | |
| *** ivar-laz_ has joined #openstack-infra | 03:31 | |
| *** yaguang has quit IRC | 03:33 | |
| *** ivar-lazzaro has quit IRC | 03:35 | |
| *** ivar-laz_ has quit IRC | 03:36 | |
| *** yaguang has joined #openstack-infra | 03:36 | |
| *** Longgeek has joined #openstack-infra | 03:36 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/gear: Support binding to a specified host name or address https://review.openstack.org/135832 | 03:36 |
| *** Ryan_Lane has joined #openstack-infra | 03:38 | |
| *** Longgeek has quit IRC | 03:41 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Flush out the db api documenation https://review.openstack.org/129405 | 03:42 |
| *** jamesmcarthur has joined #openstack-infra | 03:42 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Reorganize the python api documentation https://review.openstack.org/129406 | 03:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add section on the basic data model https://review.openstack.org/129407 | 03:42 |
| jamesmcarthur | Hi all - a heads up that openstack.org is currently having an outage. It seems our db server failed and hasn’t come back online. We have tickets open with RAX and I’ll update this when we’re back online. | 03:43 |
| clarkb | jamesmcarthur: thanks for the heads up | 03:44 |
| jamesmcarthur | NP. | 03:45 |
| *** Ryan_Lane has quit IRC | 03:45 | |
| *** nosnos has joined #openstack-infra | 03:49 | |
| jamesmcarthur | ok - back online | 03:50 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/zuul: Gear: bind to a specified address https://review.openstack.org/135837 | 03:53 |
| *** yamamoto has joined #openstack-infra | 03:53 | |
| *** rushiagr is now known as rushiagr_away | 03:53 | |
| *** jgrimm is now known as zz_jgrimm | 03:54 | |
| *** sarob has joined #openstack-infra | 03:56 | |
| *** Ryan_Lane has joined #openstack-infra | 03:58 | |
| *** sarob has quit IRC | 04:00 | |
| *** Ryan_Lane has quit IRC | 04:03 | |
| *** patrickeast has quit IRC | 04:05 | |
| *** banix has joined #openstack-infra | 04:08 | |
| *** radez is now known as radez_g0n3 | 04:12 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** pcrews has quit IRC | 04:16 | |
| *** bhuvan has joined #openstack-infra | 04:17 | |
| *** sandywalsh has quit IRC | 04:22 | |
| *** sandywalsh has joined #openstack-infra | 04:23 | |
| *** jamesmcarthur has joined #openstack-infra | 04:24 | |
| * jkt just got a first noop test done :] | 04:24 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:24 | |
| *** zz_dimtruck is now known as dimtruck | 04:27 | |
| *** stevemar has joined #openstack-infra | 04:29 | |
| *** rushiagr_away is now known as rushiagr | 04:33 | |
| *** dimtruck is now known as zz_dimtruck | 04:37 | |
| *** Longgeek has joined #openstack-infra | 04:37 | |
| *** yamamoto has quit IRC | 04:39 | |
| *** jamesmcarthur has quit IRC | 04:41 | |
| *** Longgeek has quit IRC | 04:42 | |
| *** bhunter71 has quit IRC | 04:42 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add plugins_info to module registry object. https://review.openstack.org/132927 | 04:43 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Reorganize tests/cmd https://review.openstack.org/134055 | 04:43 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8 https://review.openstack.org/135843 | 04:43 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add plugins_info to module registry object. https://review.openstack.org/132927 | 04:46 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Reorganize tests/cmd https://review.openstack.org/134055 | 04:46 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8 https://review.openstack.org/135843 | 04:46 |
| *** _buzz has quit IRC | 04:48 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 04:49 | |
| *** sarob has joined #openstack-infra | 04:58 | |
| *** Nakato_ is now known as Nakato | 04:58 | |
| *** sarob has quit IRC | 05:02 | |
| *** yamamoto has joined #openstack-infra | 05:06 | |
| *** viglesias has joined #openstack-infra | 05:08 | |
| *** garyh has quit IRC | 05:10 | |
| *** shashankhegde has joined #openstack-infra | 05:13 | |
| *** viglesias has quit IRC | 05:13 | |
| *** garyh has joined #openstack-infra | 05:13 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 05:13 | |
| *** SumitNaiksatam has quit IRC | 05:16 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 05:17 | |
| *** viglesias has joined #openstack-infra | 05:18 | |
| *** viglesias has quit IRC | 05:22 | |
| *** yfried has quit IRC | 05:27 | |
| *** VijayTripathi has quit IRC | 05:30 | |
| *** ryanpetrello has joined #openstack-infra | 05:31 | |
| *** viglesias has joined #openstack-infra | 05:32 | |
| *** Hefeweizen has quit IRC | 05:33 | |
| *** Hefeweizen has joined #openstack-infra | 05:34 | |
| *** viglesias has quit IRC | 05:34 | |
| *** VijayTripathi has joined #openstack-infra | 05:35 | |
| *** achanda has joined #openstack-infra | 05:38 | |
| *** nelsnelson has quit IRC | 05:38 | |
| *** viglesias has joined #openstack-infra | 05:40 | |
| *** viglesias has quit IRC | 05:41 | |
| *** nelsnelson has joined #openstack-infra | 05:46 | |
| *** viglesias has joined #openstack-infra | 05:47 | |
| *** viglesias has quit IRC | 05:49 | |
| *** dannywilson has quit IRC | 05:49 | |
| *** chandankumar has joined #openstack-infra | 05:52 | |
| *** viglesias has joined #openstack-infra | 05:53 | |
| *** Longgeek has joined #openstack-infra | 06:01 | |
| *** banix has quit IRC | 06:01 | |
| *** Longgeek has quit IRC | 06:02 | |
| *** Longgeek has joined #openstack-infra | 06:03 | |
| *** unicell has quit IRC | 06:05 | |
| *** talluri has joined #openstack-infra | 06:07 | |
| *** viglesias has quit IRC | 06:10 | |
| *** viglesias has joined #openstack-infra | 06:11 | |
| openstackgerrit | daisy-ycguo proposed openstack-infra/project-config: Add a new project in stackforge named as i18n-test to maintain the test codes for i18n and l10n functions. https://review.openstack.org/135857 | 06:12 |
| *** Vek has quit IRC | 06:12 | |
| *** shashankhegde has quit IRC | 06:13 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 06:15 | |
| *** yfried has joined #openstack-infra | 06:17 | |
| *** SumitNaiksatam has quit IRC | 06:18 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 06:18 | |
| *** oomichi has joined #openstack-infra | 06:22 | |
| *** amcrn has quit IRC | 06:24 | |
| *** ildikov has quit IRC | 06:25 | |
| *** Daisy has joined #openstack-infra | 06:26 | |
| *** viglesias has quit IRC | 06:31 | |
| *** harlowja is now known as harlowja_away | 06:31 | |
| *** viglesias has joined #openstack-infra | 06:31 | |
| *** k4n0 has joined #openstack-infra | 06:33 | |
| *** iain-king has quit IRC | 06:35 | |
| *** ujuc has joined #openstack-infra | 06:37 | |
| *** VijayTripathi has quit IRC | 06:39 | |
| *** yfried is now known as yfried|afk | 06:40 | |
| *** ryanpetrello has quit IRC | 06:40 | |
| *** talluri has quit IRC | 06:44 | |
| *** viglesias has quit IRC | 06:44 | |
| *** otherwiseguy has quit IRC | 06:44 | |
| *** BharatK has joined #openstack-infra | 06:45 | |
| *** viglesias has joined #openstack-infra | 06:45 | |
| *** achanda has quit IRC | 06:45 | |
| *** sputnik13 has joined #openstack-infra | 06:46 | |
| *** sputnik13 has quit IRC | 06:47 | |
| *** sputnik13 has joined #openstack-infra | 06:48 | |
| *** yfried|afk is now known as yfried | 06:48 | |
| *** sputnik13 has quit IRC | 06:49 | |
| *** sputnik13 has joined #openstack-infra | 06:49 | |
| *** sputnik13 has quit IRC | 06:51 | |
| *** sputnik13 has joined #openstack-infra | 06:51 | |
| *** afazekas has quit IRC | 06:55 | |
| *** pcrews has joined #openstack-infra | 06:55 | |
| *** vigneshvar has joined #openstack-infra | 06:55 | |
| *** ashaeron has joined #openstack-infra | 06:56 | |
| *** achanda has joined #openstack-infra | 06:58 | |
| *** achanda has quit IRC | 06:58 | |
| *** achanda has joined #openstack-infra | 06:59 | |
| *** oomichi__ has joined #openstack-infra | 07:01 | |
| *** fandi has quit IRC | 07:01 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 07:02 | |
| *** oomichi has quit IRC | 07:02 | |
| *** teran has quit IRC | 07:03 | |
| *** rmcall has quit IRC | 07:05 | |
| *** SumitNaiksatam has quit IRC | 07:05 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 07:05 | |
| *** sressot has quit IRC | 07:07 | |
| *** talluri has joined #openstack-infra | 07:10 | |
| *** armax has quit IRC | 07:12 | |
| *** viglesias has quit IRC | 07:13 | |
| *** viglesias has joined #openstack-infra | 07:14 | |
| *** sressot has joined #openstack-infra | 07:15 | |
| *** hdd has joined #openstack-infra | 07:16 | |
| *** pcrews has quit IRC | 07:16 | |
| openstackgerrit | Aaron Rosen proposed openstack-infra/project-config: Create congressmiddleware repo https://review.openstack.org/133567 | 07:17 |
| *** jgallard has joined #openstack-infra | 07:20 | |
| *** mpaolino has joined #openstack-infra | 07:22 | |
| *** achanda has quit IRC | 07:25 | |
| *** achanda has joined #openstack-infra | 07:25 | |
| *** achanda has quit IRC | 07:25 | |
| *** ildikov has joined #openstack-infra | 07:27 | |
| *** AlexF has joined #openstack-infra | 07:27 | |
| *** viglesias has quit IRC | 07:29 | |
| *** e0ne has joined #openstack-infra | 07:30 | |
| *** viglesias has joined #openstack-infra | 07:30 | |
| *** jcoufal has joined #openstack-infra | 07:35 | |
| *** otter768 has quit IRC | 07:36 | |
| *** viglesias has quit IRC | 07:36 | |
| *** viglesias has joined #openstack-infra | 07:37 | |
| *** talluri has quit IRC | 07:38 | |
| *** che-arne has quit IRC | 07:39 | |
| *** mrmartin has joined #openstack-infra | 07:43 | |
| *** afazekas has joined #openstack-infra | 07:45 | |
| *** teran has joined #openstack-infra | 07:47 | |
| *** stevemar has quit IRC | 07:47 | |
| *** viglesias has quit IRC | 07:50 | |
| *** redrobot has quit IRC | 07:52 | |
| *** viglesias has joined #openstack-infra | 07:52 | |
| *** redrobot has joined #openstack-infra | 07:55 | |
| *** redrobot is now known as Guest27879 | 07:55 | |
| *** gilliard_ has joined #openstack-infra | 07:56 | |
| *** ZZelle has quit IRC | 07:56 | |
| *** viglesias has quit IRC | 07:57 | |
| *** jpich has joined #openstack-infra | 07:58 | |
| *** talluri has joined #openstack-infra | 07:58 | |
| *** AlexF has quit IRC | 07:59 | |
| *** talluri has quit IRC | 07:59 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Remove unnecessary test-requirements install https://review.openstack.org/135876 | 08:00 |
| *** SumitNaiksatam has quit IRC | 08:00 | |
| *** ZZelle has joined #openstack-infra | 08:02 | |
| *** HeOS_ has quit IRC | 08:02 | |
| *** viglesias has joined #openstack-infra | 08:03 | |
| *** HeOS has quit IRC | 08:03 | |
| *** HeOS_ has joined #openstack-infra | 08:04 | |
| *** amuller has joined #openstack-infra | 08:05 | |
| *** sergsh_away is now known as sergsh | 08:08 | |
| *** matrohon has joined #openstack-infra | 08:09 | |
| *** belmoreira has joined #openstack-infra | 08:10 | |
| *** links has joined #openstack-infra | 08:11 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add plugins_info to module registry object. https://review.openstack.org/132927 | 08:11 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Reorganize tests/cmd https://review.openstack.org/134055 | 08:11 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8 https://review.openstack.org/135843 | 08:11 |
| *** oomichi_ has joined #openstack-infra | 08:12 | |
| *** jcoufal has quit IRC | 08:12 | |
| *** oomichi__ has quit IRC | 08:12 | |
| *** Vek has joined #openstack-infra | 08:12 | |
| *** gilliard_ has quit IRC | 08:12 | |
| *** gilliard_ has joined #openstack-infra | 08:12 | |
| *** jcoufal has joined #openstack-infra | 08:12 | |
| *** gilliard_ has quit IRC | 08:13 | |
| *** dtantsur|afk is now known as dtantsur | 08:14 | |
| *** oomichi_ has quit IRC | 08:14 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add plugins_info to module registry object. https://review.openstack.org/132927 | 08:15 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8 https://review.openstack.org/135843 | 08:15 |
| *** viglesias has quit IRC | 08:18 | |
| *** viglesias has joined #openstack-infra | 08:19 | |
| *** koolhead17 has joined #openstack-infra | 08:28 | |
| *** yamamoto has quit IRC | 08:29 | |
| *** arxcruz has joined #openstack-infra | 08:31 | |
| *** talluri has joined #openstack-infra | 08:34 | |
| *** sarob has joined #openstack-infra | 08:36 | |
| *** luqas has joined #openstack-infra | 08:38 | |
| *** sarob has quit IRC | 08:41 | |
| *** Hal has joined #openstack-infra | 08:42 | |
| *** MaxV has joined #openstack-infra | 08:42 | |
| *** talluri has quit IRC | 08:43 | |
| *** Hal is now known as Guest63270 | 08:43 | |
| *** talluri has joined #openstack-infra | 08:49 | |
| *** dmellado has joined #openstack-infra | 08:50 | |
| *** yolanda has quit IRC | 08:52 | |
| *** dmsimard_away is now known as dmsimard | 08:57 | |
| *** reed has joined #openstack-infra | 08:58 | |
| *** yamamoto has joined #openstack-infra | 08:58 | |
| *** uvirtbot has quit IRC | 09:00 | |
| *** teran has quit IRC | 09:01 | |
| *** viglesias has quit IRC | 09:01 | |
| *** teran has joined #openstack-infra | 09:02 | |
| *** viglesias has joined #openstack-infra | 09:02 | |
| *** yolanda has joined #openstack-infra | 09:06 | |
| *** sputnik13 has quit IRC | 09:06 | |
| *** teran has quit IRC | 09:07 | |
| *** jedimike has joined #openstack-infra | 09:10 | |
| *** derekh has joined #openstack-infra | 09:11 | |
| *** ujuc has quit IRC | 09:11 | |
| *** yolanda has quit IRC | 09:11 | |
| *** yolanda has joined #openstack-infra | 09:13 | |
| *** yfried is now known as yfried|afk | 09:14 | |
| *** yfried|afk is now known as yfried | 09:15 | |
| *** uvirtbot has joined #openstack-infra | 09:16 | |
| *** HeOS has joined #openstack-infra | 09:17 | |
| *** e0ne has quit IRC | 09:17 | |
| *** viglesias has quit IRC | 09:17 | |
| *** viglesias has joined #openstack-infra | 09:17 | |
| *** Daisy has quit IRC | 09:18 | |
| *** BobBall_Away is now known as BobBall | 09:19 | |
| *** dmsimard is now known as dmsimard_away | 09:20 | |
| *** oomichi has joined #openstack-infra | 09:27 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 09:29 |
| *** mwagner_lap has quit IRC | 09:30 | |
| *** Longgeek has quit IRC | 09:32 | |
| *** andreykurilin_ has joined #openstack-infra | 09:34 | |
| *** viglesias has quit IRC | 09:34 | |
| *** vponomaryov has quit IRC | 09:34 | |
| *** viglesias has joined #openstack-infra | 09:35 | |
| *** vponomaryov has joined #openstack-infra | 09:36 | |
| *** otter768 has joined #openstack-infra | 09:37 | |
| *** talluri has quit IRC | 09:38 | |
| *** yamamoto has quit IRC | 09:39 | |
| *** Longgeek has joined #openstack-infra | 09:41 | |
| *** BobBall has quit IRC | 09:41 | |
| *** otter768 has quit IRC | 09:41 | |
| *** unicell has joined #openstack-infra | 09:44 | |
| *** yfried is now known as yfried|afk | 09:45 | |
| *** che-arn4 has joined #openstack-infra | 09:45 | |
| *** che-arne has joined #openstack-infra | 09:45 | |
| *** che-arne has quit IRC | 09:45 | |
| *** che-arn4 is now known as che-arne | 09:46 | |
| *** BobBall has joined #openstack-infra | 09:48 | |
| *** BobBall is now known as Guest35780 | 09:48 | |
| *** dizquierdo has joined #openstack-infra | 09:50 | |
| *** JayJ has quit IRC | 09:50 | |
| *** JayJ has joined #openstack-infra | 09:51 | |
| *** pblaho has joined #openstack-infra | 09:51 | |
| *** mpaolino has quit IRC | 09:51 | |
| *** mpaolino has joined #openstack-infra | 09:52 | |
| *** yfried|afk is now known as yfried | 09:53 | |
| *** uvirtbot has quit IRC | 09:53 | |
| *** Guest35780 is now known as Bobba | 09:54 | |
| *** uvirtbot has joined #openstack-infra | 09:55 | |
| *** uvirtbot has quit IRC | 09:55 | |
| *** JayJ has quit IRC | 09:57 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:57 | |
| *** JayJ has joined #openstack-infra | 09:58 | |
| *** pblaho has quit IRC | 10:02 | |
| *** JayJ has quit IRC | 10:02 | |
| *** JayJ has joined #openstack-infra | 10:03 | |
| *** uvirtbot has joined #openstack-infra | 10:04 | |
| *** e0ne has joined #openstack-infra | 10:06 | |
| *** teran has joined #openstack-infra | 10:06 | |
| *** JayJ has quit IRC | 10:07 | |
| *** andreykurilin_ has quit IRC | 10:08 | |
| *** JayJ has joined #openstack-infra | 10:08 | |
| *** andreykurilin_ has joined #openstack-infra | 10:09 | |
| *** jlibosva has joined #openstack-infra | 10:11 | |
| *** mpaolino has quit IRC | 10:12 | |
| *** JayJ has quit IRC | 10:12 | |
| *** JayJ has joined #openstack-infra | 10:13 | |
| *** talluri has joined #openstack-infra | 10:15 | |
| *** mpaolino has joined #openstack-infra | 10:17 | |
| *** salv-orlando_ has joined #openstack-infra | 10:17 | |
| *** JayJ has quit IRC | 10:18 | |
| *** JayJ has joined #openstack-infra | 10:18 | |
| *** dmsimard_away is now known as dmsimard | 10:19 | |
| *** salv-orlando has quit IRC | 10:19 | |
| *** salv-orlando_ is now known as salv-orlando | 10:19 | |
| *** jgjl has joined #openstack-infra | 10:20 | |
| *** viglesias has quit IRC | 10:22 | |
| *** viglesias has joined #openstack-infra | 10:22 | |
| *** JayJ has quit IRC | 10:23 | |
| *** JayJ has joined #openstack-infra | 10:23 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/storyboard: Added tags controllers https://review.openstack.org/114217 | 10:26 |
| *** sarob has joined #openstack-infra | 10:26 | |
| *** JayJ has quit IRC | 10:28 | |
| *** talluri has quit IRC | 10:28 | |
| *** talluri has joined #openstack-infra | 10:28 | |
| *** JayJ has joined #openstack-infra | 10:28 | |
| *** uvirtbot has quit IRC | 10:31 | |
| *** sarob has quit IRC | 10:31 | |
| *** dtantsur is now known as dtantsur|bbl | 10:33 | |
| *** JayJ has quit IRC | 10:33 | |
| *** rlandy has joined #openstack-infra | 10:33 | |
| *** JayJ has joined #openstack-infra | 10:33 | |
| *** pelix has joined #openstack-infra | 10:34 | |
| *** vrutkovs has joined #openstack-infra | 10:35 | |
| *** viglesias has quit IRC | 10:37 | |
| *** uvirtbot has joined #openstack-infra | 10:37 | |
| *** viglesias has joined #openstack-infra | 10:38 | |
| *** JayJ has quit IRC | 10:38 | |
| *** JayJ has joined #openstack-infra | 10:38 | |
| *** yamamoto has joined #openstack-infra | 10:39 | |
| *** JayJ has quit IRC | 10:43 | |
| *** chandankumar has quit IRC | 10:43 | |
| *** JayJ has joined #openstack-infra | 10:44 | |
| *** ldnunes has joined #openstack-infra | 10:44 | |
| *** yamamoto has quit IRC | 10:44 | |
| *** nelsnelson has quit IRC | 10:45 | |
| *** JayJ has quit IRC | 10:48 | |
| *** Guest63270 has quit IRC | 10:48 | |
| *** JayJ has joined #openstack-infra | 10:49 | |
| *** chandankumar has joined #openstack-infra | 10:50 | |
| *** mase_x200 has joined #openstack-infra | 10:50 | |
| *** hdd has quit IRC | 10:52 | |
| *** mase_x200 has quit IRC | 10:52 | |
| *** uvirtbot has quit IRC | 10:52 | |
| *** yfried is now known as yfried|afk | 10:52 | |
| *** JayJ has quit IRC | 10:53 | |
| *** JayJ has joined #openstack-infra | 10:54 | |
| *** matrohon has quit IRC | 10:58 | |
| *** JayJ has quit IRC | 10:58 | |
| *** uvirtbot has joined #openstack-infra | 10:58 | |
| *** JayJ has joined #openstack-infra | 10:59 | |
| *** uvirtbot has quit IRC | 10:59 | |
| *** dims has quit IRC | 11:01 | |
| *** dims has joined #openstack-infra | 11:01 | |
| *** uvirtbot has joined #openstack-infra | 11:02 | |
| *** chandankumar has quit IRC | 11:03 | |
| *** uvirtbot has quit IRC | 11:03 | |
| *** JayJ has quit IRC | 11:03 | |
| *** JayJ has joined #openstack-infra | 11:04 | |
| *** uvirtbot has joined #openstack-infra | 11:05 | |
| *** chandankumar has joined #openstack-infra | 11:05 | |
| *** uvirtbot has quit IRC | 11:05 | |
| *** mase_x200 has joined #openstack-infra | 11:06 | |
| *** zz_jgrimm is now known as jgrimm | 11:06 | |
| *** uvirtbot has joined #openstack-infra | 11:06 | |
| *** uvirtbot has quit IRC | 11:07 | |
| *** jgallard has quit IRC | 11:07 | |
| *** JayJ has quit IRC | 11:08 | |
| *** JayJ has joined #openstack-infra | 11:09 | |
| *** rfolco has joined #openstack-infra | 11:11 | |
| *** jcoufal has quit IRC | 11:11 | |
| *** jcoufal has joined #openstack-infra | 11:12 | |
| *** yfried|afk is now known as yfried | 11:13 | |
| *** JayJ has quit IRC | 11:14 | |
| *** JayJ has joined #openstack-infra | 11:14 | |
| *** unicell has quit IRC | 11:15 | |
| *** asselin has quit IRC | 11:16 | |
| *** JayJ has quit IRC | 11:19 | |
| *** mase_x200 has quit IRC | 11:19 | |
| *** JayJ has joined #openstack-infra | 11:19 | |
| *** yfried is now known as yfried|afk | 11:23 | |
| *** JayJ has quit IRC | 11:24 | |
| *** JayJ has joined #openstack-infra | 11:24 | |
| *** yamahata has quit IRC | 11:26 | |
| *** jcoufal has quit IRC | 11:27 | |
| *** rfolco has quit IRC | 11:27 | |
| *** yfried|afk is now known as yfried | 11:27 | |
| *** jcoufal has joined #openstack-infra | 11:27 | |
| *** andreykurilin_ has quit IRC | 11:27 | |
| *** rfolco has joined #openstack-infra | 11:28 | |
| *** andreykurilin_ has joined #openstack-infra | 11:28 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add API call to return task statuses https://review.openstack.org/135221 | 11:28 |
| *** adalbas has quit IRC | 11:28 | |
| *** yamahata has joined #openstack-infra | 11:29 | |
| *** viglesias has quit IRC | 11:29 | |
| *** JayJ has quit IRC | 11:29 | |
| *** JayJ has joined #openstack-infra | 11:29 | |
| *** viglesias has joined #openstack-infra | 11:29 | |
| openstackgerrit | Merged openstack-dev/hacking: Don't change log level as import side effect https://review.openstack.org/134374 | 11:33 |
| openstackgerrit | Merged openstack-dev/hacking: Add H237 to HACKING.rst https://review.openstack.org/134377 | 11:33 |
| *** JayJ has quit IRC | 11:34 | |
| *** teran has quit IRC | 11:34 | |
| *** JayJ has joined #openstack-infra | 11:34 | |
| *** teran has joined #openstack-infra | 11:35 | |
| *** bhuvan has quit IRC | 11:36 | |
| *** talluri has quit IRC | 11:37 | |
| *** otter768 has joined #openstack-infra | 11:37 | |
| *** lttrl has quit IRC | 11:38 | |
| *** Longgeek has quit IRC | 11:39 | |
| *** JayJ has quit IRC | 11:39 | |
| *** fandi has joined #openstack-infra | 11:39 | |
| *** JayJ has joined #openstack-infra | 11:40 | |
| *** viglesias has quit IRC | 11:40 | |
| *** rediskin has joined #openstack-infra | 11:40 | |
| *** viglesias has joined #openstack-infra | 11:41 | |
| *** otter768 has quit IRC | 11:42 | |
| openstackgerrit | Mike Heald proposed openstack-infra/zuul: Pin GitPython to 0.3.2.RC1 https://review.openstack.org/135921 | 11:44 |
| *** JayJ has quit IRC | 11:44 | |
| *** JayJ has joined #openstack-infra | 11:45 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Adding readme https://review.openstack.org/135376 | 11:46 |
| *** habib has joined #openstack-infra | 11:49 | |
| *** JayJ has quit IRC | 11:49 | |
| *** JayJ has joined #openstack-infra | 11:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add API call to return task statuses https://review.openstack.org/135221 | 11:52 |
| *** talluri has joined #openstack-infra | 11:53 | |
| *** nfedotov has joined #openstack-infra | 11:54 | |
| *** JayJ has quit IRC | 11:54 | |
| *** pc_m has joined #openstack-infra | 11:55 | |
| *** JayJ has joined #openstack-infra | 11:55 | |
| *** radez_g0n3 is now known as radez | 11:59 | |
| *** uvirtbot has joined #openstack-infra | 12:03 | |
| *** mpaolino has quit IRC | 12:05 | |
| *** Guest63270 has joined #openstack-infra | 12:05 | |
| *** teran has quit IRC | 12:06 | |
| *** mkolesni__ has joined #openstack-infra | 12:08 | |
| *** hdd has joined #openstack-infra | 12:09 | |
| *** talluri has quit IRC | 12:10 | |
| *** viglesias has quit IRC | 12:11 | |
| *** viglesias has joined #openstack-infra | 12:12 | |
| *** rushiagr is now known as rushiagr_away | 12:15 | |
| openstackgerrit | Salvatore Orlando proposed openstack-infra/devstack-gate: Add support for enabling/disabling service extensions https://review.openstack.org/116129 | 12:15 |
| *** andreykurilin_ has quit IRC | 12:15 | |
| *** hdd has quit IRC | 12:16 | |
| reed | what's going to happen now to https://bugs.launchpad.net/openstack-ci ? | 12:16 |
| *** KanagarajM has quit IRC | 12:18 | |
| *** sputnik13 has joined #openstack-infra | 12:18 | |
| *** Longgeek has joined #openstack-infra | 12:19 | |
| *** uvirtbot has quit IRC | 12:22 | |
| *** uvirtbot has joined #openstack-infra | 12:25 | |
| *** adalbas has joined #openstack-infra | 12:25 | |
| *** uvirtbot has quit IRC | 12:26 | |
| *** hashar has joined #openstack-infra | 12:27 | |
| *** jp_at_hp has joined #openstack-infra | 12:27 | |
| *** hdd has joined #openstack-infra | 12:27 | |
| *** baoli has joined #openstack-infra | 12:27 | |
| *** habib has quit IRC | 12:30 | |
| *** uvirtbot has joined #openstack-infra | 12:30 | |
| *** oomichi has quit IRC | 12:31 | |
| *** baoli has quit IRC | 12:31 | |
| *** uvirtbot has quit IRC | 12:31 | |
| *** baoli has joined #openstack-infra | 12:31 | |
| *** sputnik13 has quit IRC | 12:32 | |
| *** talluri has joined #openstack-infra | 12:32 | |
| *** nadya_ has joined #openstack-infra | 12:34 | |
| *** links has quit IRC | 12:35 | |
| *** chandankumar has quit IRC | 12:37 | |
| *** teran has joined #openstack-infra | 12:38 | |
| *** sergsh is now known as sergsh_away | 12:38 | |
| *** nadya_ has quit IRC | 12:38 | |
| *** matrohon has joined #openstack-infra | 12:40 | |
| *** yamamoto has joined #openstack-infra | 12:41 | |
| *** jcoufal has quit IRC | 12:42 | |
| rcarrillocruz | reed: there were talks in the summit to create a script to import/export those onto storyboard | 12:42 |
| *** jcoufal has joined #openstack-infra | 12:43 | |
| *** rushiagr_away is now known as rushiagr | 12:46 | |
| *** yamamoto has quit IRC | 12:47 | |
| *** jcoufal has quit IRC | 12:51 | |
| *** doug-fish has joined #openstack-infra | 12:51 | |
| *** links has joined #openstack-infra | 12:54 | |
| *** omrim has joined #openstack-infra | 12:56 | |
| *** matrohon has quit IRC | 12:56 | |
| omrim | Hello, which user should create the public ssh key for the CI? | 12:57 |
| *** lttrl has joined #openstack-infra | 12:58 | |
| *** nuritv has joined #openstack-infra | 12:58 | |
| *** Shrews_ has joined #openstack-infra | 12:58 | |
| *** Shrews_ has quit IRC | 12:58 | |
| *** talluri has quit IRC | 12:59 | |
| *** dtantsur|bbl is now known as dtantsur | 12:59 | |
| *** mbacchi has joined #openstack-infra | 13:00 | |
| *** jcoufal has joined #openstack-infra | 13:01 | |
| *** viglesias has quit IRC | 13:05 | |
| *** viglesias has joined #openstack-infra | 13:05 | |
| *** nosnos has quit IRC | 13:05 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-kibana: Add standard puppet module files and .gitreview https://review.openstack.org/131771 | 13:05 |
| clarkb | reed ya we have synced bugs but left the lp stuff around for elastic recheck | 13:06 |
| clarkb | omrim is this for third party ci? | 13:06 |
| *** nosnos has joined #openstack-infra | 13:06 | |
| omrim | clarkb: Yes | 13:07 |
| *** mkolesni__ has quit IRC | 13:07 | |
| clarkb | omrim the user that needs the key is the one running zuul (or whatever tool you use to talk togerrit) | 13:08 |
| omrim | clarkb: I have two users zuul and jenkins so I just wonder which user need to produce the ssh key | 13:10 |
| clarkb | zuul should do it (realy any user can generate the key you just need to give it to zuul for use) | 13:10 |
| *** hdd has quit IRC | 13:11 | |
| clarkb | its also likely you will need a second key for jenkins to log into slaves | 13:11 |
| *** afazekas has quit IRC | 13:13 | |
| *** jgallard has joined #openstack-infra | 13:14 | |
| *** hdd has joined #openstack-infra | 13:14 | |
| *** Murad has joined #openstack-infra | 13:15 | |
| Murad | Hi | 13:16 |
| Murad | Is there any body? | 13:17 |
| clarkb | yup | 13:17 |
| Murad | I have a question regarding requesting a service aaccount | 13:17 |
| clarkb | ask away | 13:18 |
| Murad | do I need to create the rsa key from root or jenkins or what? | 13:18 |
| clarkb | any user can generate they key | 13:18 |
| Murad | I know but does matter which user? or its the same? | 13:19 |
| clarkb | you just have to give it to the tools that need it. gerrit needs the public key which is why you provide ot in the request. then zuul(or whatever talks to gerrit) needs the private key | 13:20 |
| *** vigneshvar has quit IRC | 13:21 | |
| *** zz_dimtruck is now known as dimtruck | 13:21 | |
| Murad | I see, but the username is at the end of the public | 13:22 |
| clarkb | thats just a comment. you can edit it to whatever you like | 13:23 |
| Murad | ok, that you very much :) | 13:23 |
| Murad | your info was very helpfull | 13:23 |
| clarkb | usually used to indicate the primary user of the key so that you can keep track of what public key goes where | 13:23 |
| Murad | I see | 13:24 |
| Murad | Thanks Clark | 13:24 |
| *** viglesias has quit IRC | 13:27 | |
| jedimike | hi everyone, the latest GitPython releases break zuul by requiring gitdb 0.6 or higher, and our tests fail with that version. Branch to pin us to RC1 is here: https://review.openstack.org/#/c/135921/ | 13:28 |
| *** viglesias has joined #openstack-infra | 13:28 | |
| *** matrohon has joined #openstack-infra | 13:29 | |
| *** afazekas has joined #openstack-infra | 13:29 | |
| *** unicell has joined #openstack-infra | 13:29 | |
| *** ddieterly has joined #openstack-infra | 13:29 | |
| clarkb | ya dims has a patch up for that too | 13:30 |
| *** andreykurilin_ has joined #openstack-infra | 13:30 | |
| clarkb | https://review.openstack.org/#/c/135782/ | 13:30 |
| dims | clarkb: thanks :) | 13:31 |
| *** dkranz has joined #openstack-infra | 13:33 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: fix glance_store testing https://review.openstack.org/135949 | 13:33 |
| clarkb | sdague is your part of NY also under all the snow? | 13:33 |
| sdague | clarkb: nope | 13:33 |
| sdague | it's below freezing, barely, but no snow this far east | 13:34 |
| sdague | clarkb: so, I realized why no one noticed glance store install was broken in devstack... turns out we weren't testing it https://review.openstack.org/135949 | 13:35 |
| dims | sdague: it was missing requirements/projects.txt too - https://review.openstack.org/#/c/135603/ | 13:35 |
| anteaya | snow here, not as much as nyc though | 13:35 |
| sdague | dims: yeh, in moving to a more targetted approach we actually need some sanity check engine on these things | 13:36 |
| sdague | because human eyes are not catching all of these | 13:36 |
| dims | yep | 13:36 |
| sdague | I wrote some hacktastic tool for devstack right now about this | 13:37 |
| *** k4n0 has quit IRC | 13:38 | |
| *** BharatK has left #openstack-infra | 13:38 | |
| *** otter768 has joined #openstack-infra | 13:38 | |
| *** afazekas has quit IRC | 13:39 | |
| dims | sdague: ya, was looking at that, one suggestion, the ALL_LIBS variable, maybe avoid chopping off the library names at the end of line? will help when folks grep for stuff - https://review.openstack.org/#/c/135711/3/tests/test_libs_from_pypi.sh,unified | 13:39 |
| *** andreykurilin__ has joined #openstack-infra | 13:40 | |
| *** andreykurilin_ has quit IRC | 13:40 | |
| sdague | dims: ?? | 13:40 |
| sdague | it's just wrapping | 13:40 |
| sdague | look at the line numbers | 13:41 |
| dims | ah duh, sorry :) | 13:41 |
| sdague | clarkb: is project-config git tree available on the unit test nodes? | 13:42 |
| *** ddieterly has quit IRC | 13:42 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: fix glance_store testing https://review.openstack.org/135949 | 13:43 |
| *** otter768 has quit IRC | 13:43 | |
| sdague | clarkb: forgot to add the templates, should pass jenkins now | 13:43 |
| *** nuritv has quit IRC | 13:44 | |
| clarkb | ya we cache all repos on all single use slaves | 13:44 |
| clarkb | you have to update it though | 13:45 |
| sdague | clarkb: it won't be up to date? | 13:46 |
| clarkb | not necessarily. on unittest slaves we only update the repo you are testing with gerrit git prep | 13:46 |
| sdague | ok, gotcha | 13:46 |
| sdague | so thinking out loud here .... | 13:46 |
| *** miqui has joined #openstack-infra | 13:46 | |
| clarkb | I think the way I would fix that is to use zuul cloner | 13:46 |
| *** dprince has joined #openstack-infra | 13:47 | |
| clarkb | the puppet apply jobs do thisif you want an example (instead of ggp) | 13:47 |
| sdague | I'm trying to figure out the right way to handle the mismatch of devstack pypi support and our job definitions | 13:47 |
| sdague | and trying to figure out if project-config should check for that, or devstack should check for that in test automation | 13:47 |
| *** salv-orlando has quit IRC | 13:47 | |
| sdague | have thoughts | 13:48 |
| sdague | ? | 13:48 |
| *** rfolco has quit IRC | 13:48 | |
| sdague | also, need more coffee, back in a minute | 13:48 |
| clarkb | hrm. I think it depends on who you expect to fix problems with it | 13:48 |
| clarkb | if its somethibg infra should stay on top of then test there. if devstack should do it test in devstack | 13:48 |
| clarkb | that way the correct eyeballs see the results | 13:49 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-kibana: Add standard puppet module files and .gitreview https://review.openstack.org/131771 | 13:49 |
| *** miqui_ has joined #openstack-infra | 13:50 | |
| *** miqui__ has joined #openstack-infra | 13:52 | |
| *** andreykurilin__ has quit IRC | 13:53 | |
| *** miqui___ has joined #openstack-infra | 13:54 | |
| *** achuprin_ has joined #openstack-infra | 13:54 | |
| *** miqui____ has joined #openstack-infra | 13:55 | |
| *** jamesmcarthur has joined #openstack-infra | 13:55 | |
| *** dkliban_afk is now known as dkliban | 13:55 | |
| *** miqui_____ has joined #openstack-infra | 13:55 | |
| *** miqui___ has quit IRC | 13:56 | |
| *** yfried has quit IRC | 13:56 | |
| *** kgiusti has joined #openstack-infra | 13:56 | |
| *** mpaolino has joined #openstack-infra | 13:57 | |
| *** koolhead17 has quit IRC | 13:57 | |
| *** miqui___ has joined #openstack-infra | 13:57 | |
| *** koolhead17 has joined #openstack-infra | 13:58 | |
| sdague | maybe it's about sequencing... | 13:58 |
| *** weshay has joined #openstack-infra | 13:58 | |
| sdague | project-config should probably land first | 13:58 |
| *** unicell has quit IRC | 13:58 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Add more details on event dashboard https://review.openstack.org/135954 | 13:58 |
| *** miqui__ has quit IRC | 13:58 | |
| *** miqui has quit IRC | 13:58 | |
| *** miqui____ has quit IRC | 13:58 | |
| *** miqui_ has quit IRC | 13:58 | |
| *** miqui___ has quit IRC | 13:58 | |
| *** viglesias has quit IRC | 13:58 | |
| sdague | which means the check would need to be in devstack | 13:58 |
| sdague | saying that something was mismatched between the 2 | 13:59 |
| *** rushiagr is now known as rushiagr_away | 13:59 | |
| *** jgjl has quit IRC | 13:59 | |
| *** miqui has joined #openstack-infra | 13:59 | |
| *** viglesias has joined #openstack-infra | 13:59 | |
| sdague | unrelated pondering | 13:59 |
| sdague | I realized that we do a giant job storm on requirements proposal bot because we propose requirements changes to all the projects and they test their tempest configs... right after we just tested that tempest config | 14:00 |
| sdague | would be nice to say that if it was the proposal bot proposing, don't run the tempest jobs | 14:00 |
| sdague | just the unit test ones | 14:01 |
| *** hashar is now known as hasharCall | 14:01 | |
| sdague | it would save a couple hundred nodes | 14:01 |
| *** baoli has quit IRC | 14:01 | |
| sdague | we'll see one of those storms here in a second | 14:01 |
| *** jistr has joined #openstack-infra | 14:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch oslo.vmware from py33 to py34 https://review.openstack.org/135822 | 14:03 |
| *** koolhead17 has quit IRC | 14:03 | |
| *** julim has joined #openstack-infra | 14:03 | |
| openstackgerrit | Merged openstack/requirements: Revert "Exclude testtools 1.4.0" https://review.openstack.org/135828 | 14:04 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/135955 | 14:04 |
| *** jistr is now known as jistr|mtg | 14:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Add python34 check for taskflow https://review.openstack.org/135829 | 14:05 |
| *** sarob has joined #openstack-infra | 14:05 | |
| *** AlexF has joined #openstack-infra | 14:06 | |
| sdague | clarkb: can you re +2 this - https://review.openstack.org/#/c/135949/ ? | 14:06 |
| *** hasharCall has quit IRC | 14:06 | |
| *** hashar has joined #openstack-infra | 14:06 | |
| anteaya | clarkb: we are operating on the theory that manage-projects occasionally fails due to a mysql port being closed | 14:06 |
| *** eharney has quit IRC | 14:07 | |
| *** rushiagr_away is now known as rushiagr | 14:07 | |
| anteaya | clarkb: have we done anything to address that, or just spinning the wheel when we approve new repo creation? | 14:07 |
| clarkb | sdague done. was waiting for test results then got distracted | 14:07 |
| clarkb | anteaya I havent done anything the work around isnt bad so I think we can spin the wheel | 14:08 |
| anteaya | okay | 14:08 |
| * anteaya spins | 14:08 | |
| anteaya | now if I were feeling like taking a stab at fixing the issue, what could I do that might be useful? | 14:09 |
| anteaya | I don't know anything about mysql ports and the timing of there closure | 14:09 |
| anteaya | their | 14:09 |
| *** hdd has quit IRC | 14:10 | |
| clarkb | we probably should start with a simple test case of opening a connectuon, letting it idle for 10 minutes, then attempt to use it | 14:10 |
| clarkb | I can try to make time for that today | 14:11 |
| *** zul has joined #openstack-infra | 14:11 | |
| *** changbl has quit IRC | 14:12 | |
| *** bswartz has quit IRC | 14:12 | |
| anteaya | thanks | 14:13 |
| *** mriedem has joined #openstack-infra | 14:13 | |
| clarkb | but I need to go make breakfast nowish probably afk until typical pacific time | 14:13 |
| anteaya | would I be able to do anything useful besides watch? | 14:13 |
| *** viglesias has quit IRC | 14:13 | |
| anteaya | clarkb: I'm still surprised you are jet lagged | 14:14 |
| *** miqui has quit IRC | 14:14 | |
| clarkb | if you have a rax account tou could spin up trove db and test that | 14:14 |
| anteaya | unless you like the relative quiet of first thing in the eastern time morning | 14:14 |
| *** sarob has quit IRC | 14:14 | |
| *** miqui has joined #openstack-infra | 14:14 | |
| anteaya | I don't think I have a rax account, but I can see about getting one | 14:14 |
| clarkb | anteaya I dont deal with it well and larissa is sick so its just bad | 14:14 |
| *** viglesias has joined #openstack-infra | 14:14 | |
| anteaya | :( | 14:14 |
| *** miqui_ has joined #openstack-infra | 14:15 | |
| *** afazekas has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove unnecessary test-requirements install https://review.openstack.org/135876 | 14:15 |
| *** rfolco has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack-infra/project-config: fix glance_store testing https://review.openstack.org/135949 | 14:16 |
| *** miqui_ has quit IRC | 14:17 | |
| *** unicell has joined #openstack-infra | 14:17 | |
| *** david-lyle has quit IRC | 14:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Create congressmiddleware repo https://review.openstack.org/133567 | 14:18 |
| *** dustins has joined #openstack-infra | 14:19 | |
| openstackgerrit | Merged openstack-infra/puppet-httpd: Add rake "lint" target, fix errors https://review.openstack.org/135788 | 14:19 |
| rcarrillocruz | clarkb: hi there, could you please review https://review.openstack.org/#/c/132194/ and https://review.openstack.org/#/c/132283/ ? similar changes, awaiting for another +2 | 14:26 |
| sdague | clarkb: https://review.openstack.org/#/c/134283/ could use eyes as well | 14:30 |
| sdague | turns out that merge-check misses a lot of nova right now, because it has more than 500 open reviews | 14:31 |
| *** teran has quit IRC | 14:31 | |
| *** rfolco has quit IRC | 14:32 | |
| *** banix has joined #openstack-infra | 14:32 | |
| *** nosnos has quit IRC | 14:34 | |
| rcarrillocruz | thx fungi | 14:34 |
| *** dangers_away is now known as dangers | 14:34 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 14:37 |
| *** hdd has joined #openstack-infra | 14:37 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 14:38 |
| *** ryanpetrello has joined #openstack-infra | 14:40 | |
| *** Sincler has joined #openstack-infra | 14:40 | |
| *** masayukig has quit IRC | 14:41 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/infra-specs: Make metadata.json all lowercase https://review.openstack.org/136009 | 14:41 |
| *** bswartz has joined #openstack-infra | 14:45 | |
| *** masayukig has joined #openstack-infra | 14:45 | |
| *** koolhead17 has joined #openstack-infra | 14:46 | |
| *** jgjl has joined #openstack-infra | 14:46 | |
| *** unicell has quit IRC | 14:46 | |
| mtreinish | no another one, it looks like today taskflow added a new req which broke icehouse again | 14:46 |
| *** r-daneel has joined #openstack-infra | 14:46 | |
| *** bradjones has joined #openstack-infra | 14:47 | |
| *** xyang0 has joined #openstack-infra | 14:47 | |
| *** links has quit IRC | 14:48 | |
| *** yamamoto has joined #openstack-infra | 14:48 | |
| *** zz_gondoi is now known as gondoi | 14:48 | |
| *** otherwiseguy has joined #openstack-infra | 14:49 | |
| openstackgerrit | Merged openstack-infra/system-config: Expose status_url parameter in zuul_prod/dev class signature https://review.openstack.org/132194 | 14:49 |
| *** hashar has quit IRC | 14:50 | |
| *** ddieterly has joined #openstack-infra | 14:50 | |
| *** hashar has joined #openstack-infra | 14:50 | |
| *** tkelsey has joined #openstack-infra | 14:51 | |
| *** links has joined #openstack-infra | 14:51 | |
| *** mjturek has joined #openstack-infra | 14:52 | |
| openstackgerrit | Merged openstack-infra/system-config: Expose projects_config parameter in review/review_dev classes https://review.openstack.org/132283 | 14:52 |
| *** eharney has joined #openstack-infra | 14:53 | |
| openstackgerrit | Merged openstack-infra/release-tools: Fix specs repo location calculation https://review.openstack.org/135652 | 14:53 |
| *** unicell has joined #openstack-infra | 14:53 | |
| *** yaguang has quit IRC | 14:53 | |
| *** pradk has joined #openstack-infra | 14:54 | |
| *** hashar has quit IRC | 14:54 | |
| *** hashar has joined #openstack-infra | 14:54 | |
| anteaya | EmilienM: this repo has 12 branches and 7 tags: https://github.com/redhat-openstack/puppet-openstack-cloud | 14:56 |
| anteaya | EmilienM: did you want to tidy it up a bit before it seeds the stackforge repo? | 14:56 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add stable compat jobs to oslo libs in stable requirements https://review.openstack.org/136017 | 14:56 |
| EmilienM | anteaya: of course! | 14:56 |
| *** markmcclain has joined #openstack-infra | 14:57 | |
| mtreinish | anteaya, sdague: ^^^ 2x in 2 days is too many breaks | 14:57 |
| EmilienM | anteaya: we are using this module for a while... that's why | 14:57 |
| *** viglesias has quit IRC | 14:57 | |
| *** salv-orlando has joined #openstack-infra | 14:57 | |
| *** viglesias has joined #openstack-infra | 14:57 | |
| anteaya | EmilienM: thanks | 14:57 |
| anteaya | EmilienM: makes sense, let's trim it down for the travel | 14:58 |
| anteaya | mtreinish: I agree | 14:58 |
| sdague | mtreinish: yeh, works for me | 14:58 |
| sdague | mtreinish: though how they get out of the hole isn't well established at this point | 14:59 |
| mtreinish | sdague: what do you mean? | 14:59 |
| sdague | how does taskflow add a requirement in a future version | 14:59 |
| anteaya | dhellmann: do you have a minute for 136017? | 14:59 |
| *** yamahata has quit IRC | 14:59 | |
| *** otherwiseguy has quit IRC | 14:59 | |
| *** miqui has quit IRC | 15:00 | |
| anteaya | dhellmann: I'd like you to at least be aware of the new jobs before I approve | 15:00 |
| *** markmcclain1 has joined #openstack-infra | 15:00 | |
| mtreinish | sdague: they have to add it to stable branches too, or cap the g-r in stable | 15:00 |
| *** yamahata has joined #openstack-infra | 15:00 | |
| mtreinish | which I think dhellmann is working on figuring out if that's doable or not | 15:00 |
| sdague | clarkb: any idea where gerrit review --abandon --message foo seems to post the message twice? | 15:00 |
| mtreinish | sdague: I mean right now I'm pushing backports to stable reqs to unbreak tempest's gate | 15:01 |
| fungi | sdague: sounds like a git-review bug | 15:01 |
| sdague | it's not git-review | 15:01 |
| sdague | it's raw ssh | 15:01 |
| *** markmcclain has quit IRC | 15:01 | |
| anteaya | mtreinish: I'm onboard in theory, I just want to hear from dhellmann before adding the new jobs | 15:01 |
| openstackgerrit | Merged openstack-infra/project-config: Fix grenade direction for grenade-dsvm-ironic https://review.openstack.org/135790 | 15:01 |
| fungi | sdague: d'oh, you said gerrit review | 15:01 |
| *** teran has joined #openstack-infra | 15:01 | |
| *** asselin has joined #openstack-infra | 15:01 | |
| fungi | sdague: yeah, then sounds like a bug in gerrit | 15:01 |
| *** Murad has quit IRC | 15:02 | |
| mtreinish | anteaya: yeah, no worries that's fine | 15:02 |
| anteaya | mtreinish: thanks | 15:02 |
| *** teran_ has joined #openstack-infra | 15:02 | |
| mtreinish | for those who have stable +2: https://review.openstack.org/136018 to unbreak the stable gates | 15:02 |
| *** unicell has quit IRC | 15:02 | |
| *** nelsnelson has joined #openstack-infra | 15:02 | |
| *** nadya_ has joined #openstack-infra | 15:02 | |
| anteaya | is that dims and jungleboyj | 15:02 |
| *** nelsnelson has quit IRC | 15:03 | |
| *** jungleboyj has joined #openstack-infra | 15:03 | |
| *** nelsnelson has joined #openstack-infra | 15:04 | |
| EmilienM | anteaya: I've deleted all the branches | 15:04 |
| mtreinish | anteaya: it's anyone in stable-maint I think | 15:04 |
| EmilienM | anteaya: can we keep the tag? | 15:04 |
| dims | anteaya: i don't have karma | 15:04 |
| anteaya | EmilienM: you can keep anything you need to keep | 15:04 |
| anteaya | dims: ah sorry, I am mistaken | 15:05 |
| EmilienM | anteaya: thanks :) | 15:05 |
| anteaya | EmilienM: of course | 15:05 |
| anteaya | :D | 15:05 |
| *** isviridov is now known as isviridov_away | 15:05 | |
| *** markmcclain1 has quit IRC | 15:05 | |
| anteaya | EmilienM: so 4 branches, 7 tags are what you want to keep? | 15:05 |
| *** markmcclain has joined #openstack-infra | 15:05 | |
| *** miqui has joined #openstack-infra | 15:05 | |
| *** teran has quit IRC | 15:06 | |
| *** liam_ has joined #openstack-infra | 15:07 | |
| EmilienM | anteaya: one branch finally, and 7 tags. Yes | 15:07 |
| *** liam_ is now known as Guest73106 | 15:07 | |
| anteaya | oh you are still deleting, okay can you add a comment to the patch once you are done, confirming the repo is in the state you wish | 15:08 |
| *** lttrl has quit IRC | 15:09 | |
| EmilienM | anteaya: done: https://review.openstack.org/#/c/135729/4 | 15:10 |
| sdague | are we rerunning tests again on restore? | 15:10 |
| *** liam__ has joined #openstack-infra | 15:10 | |
| jkt | hmmm, so I proposed a change for fixing zuul's gear binding to a wrong socket, and it seems that it failed with an unrelated error | 15:10 |
| jkt | https://review.openstack.org/#/c/135837/ is what I'm talkiing about, and https://review.openstack.org/#/c/114510/ and https://review.openstack.org/#/c/135332/ appear to be hit as well | 15:11 |
| jkt | I suppose I should file a bug, right? :) | 15:11 |
| *** Guest73106 has quit IRC | 15:11 | |
| *** liam__ has quit IRC | 15:11 | |
| anteaya | EmilienM: thank you! | 15:12 |
| fungi | sdague: i don't think tests rerun on restore events. can't remember if zuul knows how to trigger on those, in which case it might just need configuring in layout | 15:12 |
| clarkb | jkt its a known issue. gitpython released and broke zuul tests. there are a couple patches up to fix it | 15:12 |
| fungi | jkt: known issue with latest gitpython release. dims has a zuul change proposed to fix that | 15:12 |
| clarkb | jkt you can rebase onto dims' change to pin that dep | 15:12 |
| clarkb | so nodepool OS upgrade is a little more complicated than I initially thought | 15:13 |
| clarkb | going to need to write a change or two and make sure my puppet dev env is working | 15:14 |
| jkt | ah, thanks | 15:15 |
| *** jcoufal has quit IRC | 15:15 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: spec2bp v2 -- support in-review specs https://review.openstack.org/108041 | 15:16 |
| jkt | and https://review.openstack.org/#/c/135921/ has a nicer commit message :) | 15:16 |
| *** pcrews has joined #openstack-infra | 15:17 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/zuul: Gear: bind to a specified address https://review.openstack.org/135837 | 15:17 |
| clarkb | I am not sure that is the actual issue though | 15:18 |
| clarkb | jedimike ^ | 15:18 |
| *** AlexF has quit IRC | 15:18 | |
| *** miqui_____ has quit IRC | 15:18 | |
| clarkb | gitdb 0.6.0 is available... problem is the interfaces have changed (exceptions thrown instead of false values returned) | 15:19 |
| *** tonytan4ever has joined #openstack-infra | 15:20 | |
| *** yamamoto_ has joined #openstack-infra | 15:20 | |
| jedimike | clarkb, i'm happy to work on fixing it for 0.6.0, might be best to pin GitPython first though as it's my first time looking at how zuul uses git so might take me a little longer than others to make a fix | 15:20 |
| clarkb | yes we should pin | 15:20 |
| *** JayJ has quit IRC | 15:21 | |
| clarkb | just pointing out I find the commit message misleading | 15:21 |
| *** yamamoto has quit IRC | 15:21 | |
| clarkb | gitdb versions dont really matter. gitpythons public interface is what broke | 15:22 |
| *** amitgandhinz has joined #openstack-infra | 15:22 | |
| *** miqui_ has joined #openstack-infra | 15:22 | |
| jedimike | ok, i'l modify the comment message | 15:22 |
| clarkb | well we should merge dims change | 15:22 |
| clarkb | it was first | 15:22 |
| dims | clarkb: +1 :) | 15:22 |
| jedimike | ok, no problem :) | 15:22 |
| jedimike | I just found that GitPython 0.3.2.1 with gitdb 0.5.x worked, so I figured it was a gitdb change that broke it rather than a GitPython change | 15:23 |
| *** otherwiseguy has joined #openstack-infra | 15:23 | |
| clarkb | the distinction there to me is gitdb versions shouldnt matter if gitpython maintains compat in public interfaces | 15:25 |
| jedimike | yeah they shouldn't | 15:25 |
| *** miqui has quit IRC | 15:25 | |
| clarkb | so I think still likely a gitpython bug especially if they semver | 15:25 |
| jedimike | but, if you use GitPython 0.3.2 with gitdb 0.6.0, zuul breaks. If you remove gitdb 0.6.0 and install the 0.5 version, it works | 15:25 |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/zuul: Process event multiple times. https://review.openstack.org/136027 | 15:26 |
| jedimike | either way, dims fix was first :) | 15:28 |
| *** atan8 has joined #openstack-infra | 15:28 | |
| *** dustins_ has joined #openstack-infra | 15:30 | |
| sdague | mtreinish: you need to update templates | 15:31 |
| mtreinish | sdague: on 136017? | 15:31 |
| sdague | yep | 15:31 |
| *** miqui has joined #openstack-infra | 15:31 | |
| sdague | look at your jenkins results | 15:32 |
| sdague | also, you can actually run those tests locally now | 15:32 |
| *** miqui has quit IRC | 15:32 | |
| *** dustins has quit IRC | 15:33 | |
| mtreinish | oh fun, are there just no stable job definitions for those projects? Or is the naming off | 15:33 |
| *** miqui has joined #openstack-infra | 15:34 | |
| *** jamesmcarthur has quit IRC | 15:34 | |
| mtreinish | yeah I just need to add them to projects.yaml it looks like | 15:34 |
| *** miqui has quit IRC | 15:35 | |
| *** jamesmcarthur has joined #openstack-infra | 15:36 | |
| *** atan8 has quit IRC | 15:36 | |
| *** jgrimm is now known as zz_jgrimm | 15:36 | |
| *** wenlock has joined #openstack-infra | 15:39 | |
| *** otter768 has joined #openstack-infra | 15:39 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add stable compat jobs to oslo libs in stable requirements https://review.openstack.org/136017 | 15:41 |
| mtreinish | sdague: ^^^ I got impatient waiting on the venv, but I think that's all that was needed | 15:42 |
| *** markmcclain has quit IRC | 15:43 | |
| *** zul has quit IRC | 15:43 | |
| *** ryanpetrello has quit IRC | 15:44 | |
| *** otter768 has quit IRC | 15:44 | |
| *** banix has quit IRC | 15:45 | |
| *** zul has joined #openstack-infra | 15:46 | |
| *** ryanpetrello has joined #openstack-infra | 15:46 | |
| *** markmcclain has joined #openstack-infra | 15:46 | |
| *** markmcclain has quit IRC | 15:46 | |
| *** markmcclain has joined #openstack-infra | 15:47 | |
| anteaya | mtreinish: am watching the patch, waiting for doug | 15:50 |
| *** JayJ has joined #openstack-infra | 15:50 | |
| *** uvirtbot has joined #openstack-infra | 15:52 | |
| *** uvirtbot has quit IRC | 15:52 | |
| *** AlexF has joined #openstack-infra | 15:54 | |
| *** dprince has quit IRC | 15:54 | |
| *** banix has joined #openstack-infra | 15:55 | |
| *** bhunter71 has joined #openstack-infra | 15:55 | |
| *** jlibosva has quit IRC | 15:57 | |
| *** jlibosva has joined #openstack-infra | 15:59 | |
| dstufft | fungi: are you still getting https://github.com/pypa/pip/issues/1562 or can you reproduce it at all? I can't seem to do so and i don't see anything in the code that looksl ike it'd be causing it :/ | 16:00 |
| *** viglesias has quit IRC | 16:01 | |
| *** tnovacik has joined #openstack-infra | 16:01 | |
| *** viglesias has joined #openstack-infra | 16:02 | |
| *** andreykurilin_ has joined #openstack-infra | 16:02 | |
| *** david-lyle has joined #openstack-infra | 16:02 | |
| *** tsg_ has joined #openstack-infra | 16:03 | |
| *** rmcall has joined #openstack-infra | 16:03 | |
| fungi | dstufft: good question. i likely found some workaround... now if only i'd left myself some breadcrumb indicating how i encountered that error. let me see if i can find it in irc logs | 16:03 |
| *** dimtruck is now known as zz_dimtruck | 16:04 | |
| *** erikwilson has joined #openstack-infra | 16:04 | |
| *** Ark has joined #openstack-infra | 16:05 | |
| *** jgjl has quit IRC | 16:05 | |
| *** xyang0 has quit IRC | 16:05 | |
| *** Ark is now known as Guest56789 | 16:05 | |
| *** armax has joined #openstack-infra | 16:05 | |
| mtreinish | fungi: can you do a quick check and see if there are any errors in the logs for the one subunit-gearman worker still running? | 16:05 |
| *** marun has joined #openstack-infra | 16:05 | |
| mtreinish | I'm suspicious that it's been able to keep up with the load | 16:06 |
| *** hashar is now known as hasharNature | 16:06 | |
| *** zz_dimtruck is now known as dimtruck | 16:06 | |
| fungi | checking | 16:07 |
| *** koolhead17 has quit IRC | 16:08 | |
| jeblair | fungi, clarkb, mordred: http://paste.openstack.org/show/135369/ | 16:09 |
| jeblair | found that in my email from a while ago (still catching up) | 16:10 |
| jeblair | explains what happened to the afs server right before summit time | 16:10 |
| fungi | jeblair: wow, indeed it does! | 16:10 |
| *** belmoreira has quit IRC | 16:10 | |
| *** thedodd has joined #openstack-infra | 16:10 | |
| jeblair | also, first time i've heard the recommendation to raid across volumes... i wonder how we could ensure that we get actual diversity when doing that? | 16:10 |
| jeblair | (i'm pretty sure we can't which is why we don't) | 16:11 |
| fungi | jeblair: yeah, unless they guarantee anti-affinity for volumes | 16:11 |
| *** alexpilotti has joined #openstack-infra | 16:11 | |
| fungi | which i can't really see given they can't predict usage patterns and you could have way more volumes than they have arrays | 16:11 |
| *** amitgandhinz has quit IRC | 16:12 | |
| *** achuprin_ has quit IRC | 16:12 | |
| fungi | are there maybe cinder api calls you can make to set (anti-)affinity rules? | 16:12 |
| *** _buzz has joined #openstack-infra | 16:13 | |
| clarkb | I am really quickly running a local dry run on trusty for centos7 image builds using latest dib with a moved tmpdir and cachedir | 16:13 |
| fungi | mtreinish: i see a lot of noop and sleep in the log | 16:13 |
| clarkb | while that is running I will spin up the change to move tmpdir and cachedir in our nodepool config | 16:13 |
| clarkb | then I can boot the new node using that change via my puppet dev env | 16:13 |
| *** stevemar has joined #openstack-infra | 16:13 | |
| clarkb | also we will have to wait for new images to build on new side | 16:13 |
| *** hasharNature is now known as hashar | 16:13 | |
| clarkb | are we ok with that delay? it could be up to about an hour | 16:14 |
| *** andreykurilin_ has quit IRC | 16:14 | |
| *** andreykurilin__ has joined #openstack-infra | 16:14 | |
| fungi | mtreinish: no, wait, i found an event which triggered log retrieval and conversion | 16:14 |
| fungi | mtreinish: so i expect it's working | 16:14 |
| fungi | mtreinish: did you merge changes to properly filter down to just gate jobs? | 16:15 |
| mtreinish | fungi: hmm, ok I remain skeptical but ok | 16:15 |
| mtreinish | fungi: no not yet it's still under review | 16:15 |
| mtreinish | fungi: https://review.openstack.org/135493 | 16:15 |
| fungi | mtreinish: i can check the db for signs of recent additions if that helps | 16:15 |
| mtreinish | fungi: sure, that would be useful | 16:15 |
| mtreinish | you can use the runs table it has a run_at column which should be the timestamp of when the stream is added to the db | 16:16 |
| *** arxcruz has quit IRC | 16:18 | |
| *** __TheDodd__ has joined #openstack-infra | 16:18 | |
| fungi | select run_at from runs order by run_at; | 16:18 |
| fungi | 6351 rows in set, 2014-11-20 16:17:55 is the most recent | 16:18 |
| fungi | mtreinish: ^ | 16:18 |
| fungi | 16:19:04 now | 16:19 |
| *** thedodd has quit IRC | 16:19 | |
| fungi | it's active for sure | 16:19 |
| *** dannywilson has joined #openstack-infra | 16:19 | |
| mtreinish | fungi: ok cool, and can you check the count on the test_runs table | 16:20 |
| mtreinish | it should be roughly 2000 X 6000 | 16:20 |
| mtreinish | because that table is where things were hanging up for me locally before | 16:20 |
| fungi | 5184819 | 16:21 |
| fungi | order of magnitude looks about right | 16:22 |
| mtreinish | ok, yeah that's about right | 16:22 |
| mtreinish | this is roughly the point I stopped testing locally too | 16:22 |
| mtreinish | I guess it's working fine | 16:22 |
| mtreinish | I'm very surprised | 16:22 |
| *** ildikov has quit IRC | 16:23 | |
| fungi | it's always great when you discover that the thing you wrote which you assumed was utter crap is actually efficient and working | 16:23 |
| fungi | savor the moment ;) | 16:24 |
| *** dannywilson has quit IRC | 16:24 | |
| mtreinish | heh, well it's still too soon to say that I think :) but I will | 16:24 |
| *** dannywilson has joined #openstack-infra | 16:24 | |
| *** vigneshvar has joined #openstack-infra | 16:24 | |
| clarkb | fungi: I never believe it | 16:25 |
| openstackgerrit | Merged openstack-infra/project-config: Import puppet-openstack-cloud to Stackforge https://review.openstack.org/135729 | 16:25 |
| *** achuprin_ has joined #openstack-infra | 16:27 | |
| clarkb | a murder of crows has taken up shop in the tree across the alley | 16:27 |
| clarkb | I count about 18 | 16:27 |
| *** otherwiseguy has quit IRC | 16:28 | |
| clarkb | anteaya: is congress middleware the project you added? | 16:28 |
| *** afazekas has quit IRC | 16:28 | |
| clarkb | it seems to hae broken my test image build | 16:29 |
| * clarkb goes to investigate | 16:29 | |
| anteaya | it was added yes | 16:29 |
| anteaya | I broke image builds :( | 16:29 |
| *** zz_jgrimm is now known as jgrimm | 16:29 | |
| anteaya | and a stackforge repo addition just merged too | 16:29 |
| anteaya | I'm sorry clarkb | 16:29 |
| anteaya | how will I know in future when not to approve new repo creation | 16:29 |
| anteaya | so as to not break your image builds? | 16:30 |
| clarkb | anteaya: image uilds happen at 1400UTC | 16:30 |
| clarkb | and can take a little over an hour | 16:30 |
| clarkb | I think I fixed congressmiddleware | 16:30 |
| anteaya | thanks | 16:30 |
| anteaya | sorry about that | 16:30 |
| clarkb | anteaya: what is the other one? | 16:30 |
| anteaya | puppet-openstack-cloud | 16:30 |
| anteaya | it just merged | 16:30 |
| clarkb | and I should probably prioritize debugging of this issue since it impacts nodepool things | 16:30 |
| clarkb | oh ya I see it there | 16:31 |
| fungi | an unforseen downside to moving image updates to around the time infra root admins are coming online for the day, it's also about the time we all start approving random things ;) | 16:31 |
| anteaya | so no new repo creation from 1300 to 1500 utc? | 16:31 |
| clarkb | anteaya: probably until 1600 if you want to be safe | 16:31 |
| clarkb | sometimes they run long | 16:31 |
| anteaya | and andreas isn't online | 16:32 |
| anteaya | I can post to the infra ml if you want | 16:32 |
| jeblair | why don't we fix the image builds? | 16:32 |
| *** ashaeron has quit IRC | 16:32 | |
| jeblair | or, well, is anything really broken? :) | 16:33 |
| jeblair | clarkb: what actually failed? | 16:33 |
| clarkb | jeblair: you can't git clone a repo that doesn't have its default branch | 16:33 |
| clarkb | fatal: Couldn't find remote ref master | 16:34 |
| jeblair | clarkb: at what point do we have a git repo without a default branch? | 16:34 |
| *** __TheDodd__ has quit IRC | 16:34 | |
| clarkb | jeblair: right now when creating new projects because creating new projects is broken | 16:34 |
| *** dustins_ has quit IRC | 16:35 | |
| jeblair | clarkb: bug? | 16:35 |
| clarkb | we create the empty repo in gerrit then fail to push into it | 16:35 |
| clarkb | ya | 16:35 |
| jeblair | clarkb: sorry, meant, do you have a link to the bug? | 16:35 |
| clarkb | fungi: ^ | 16:35 |
| jeblair | (i've clearly missed some context and need to catch up) | 16:35 |
| clarkb | fungi was debugging it | 16:35 |
| clarkb | I don't know if there is a storyboard story yet | 16:35 |
| jeblair | clarkb: i believe the correct course of action is to stop creating new projects until the bug is fixed | 16:36 |
| *** baoli has joined #openstack-infra | 16:36 | |
| *** dustins has joined #openstack-infra | 16:36 | |
| *** imcsk8 has quit IRC | 16:37 | |
| *** dustins has quit IRC | 16:37 | |
| *** imcsk8 has joined #openstack-infra | 16:37 | |
| clarkb | I think the reason fungi didn't want to go that route is we have little to go off of. manually running manage projects just works | 16:38 |
| *** jgallard_ has joined #openstack-infra | 16:38 | |
| clarkb | so tickling it has required puppet to run. Theory is that a long lived connection to mysql is being killed which breaks updating the groups file | 16:38 |
| fungi | ahh sorry, yes | 16:38 |
| fungi | jeblair: clarkb: http://paste.openstack.org/show/134438/ | 16:38 |
| clarkb | oh you actually get the gone away error | 16:39 |
| fungi | it looks like if manage-projects runs too long, we get disconnected from the trove remote socket | 16:39 |
| *** jgallard has quit IRC | 16:39 | |
| fungi | i was going to pick brains of people with more mysql experience as to what the best way to work around that is (keepalives? don't reuse the socket? retry mechanism?) | 16:40 |
| jeblair | this bug, just like the last on in m-p, has ripple effects that cause the rest of our systems to break. given that, i think we need to prioritize fixing it, and suspend new project creation until it is fixed | 16:40 |
| anteaya | jeblair: want me to compose a ml post to infra in an etherpad? | 16:40 |
| jeblair | m-p really needs to be a lot more robust than it is :( | 16:40 |
| anteaya | it gets exercised several times a day | 16:41 |
| jeblair | anteaya: if i can convince clarkb and fungi :) | 16:41 |
| * anteaya stands by | 16:41 | |
| *** mmaglana has joined #openstack-infra | 16:41 | |
| *** dmellado has quit IRC | 16:41 | |
| *** ChuckC has quit IRC | 16:41 | |
| fungi | color me convinced | 16:42 |
| fungi | i wasn't sure what the frequency with which we were hitting it was | 16:42 |
| fungi | sounds like frequently now? | 16:42 |
| clarkb | yes puppet-openstack-cloud hit it too | 16:42 |
| jeblair | fungi: can you file a bug for it? | 16:42 |
| fungi | since that's twice in two days | 16:42 |
| clarkb | I think it must be every run | 16:42 |
| fungi | will do | 16:42 |
| anteaya | and there was the notion that paste.o.o might also be hitting the same thing | 16:42 |
| * anteaya opens an etherpad | 16:43 | |
| anteaya | https://etherpad.openstack.org/p/TNfXmwbOPb to compose a ml post to -infra | 16:43 |
| fungi | "By default, the server closes the connection after eight hours if nothing has happened. You can change the time limit by setting the wait_timeout variable when you start mysqld." http://dev.mysql.com/doc/refman/5.0/en/gone-away.html | 16:44 |
| *** jgallard_ has quit IRC | 16:44 | |
| fungi | phschwartz: do you know anyone supporting the trove backends at rax whose brains we can pick? | 16:44 |
| clarkb | we are very clearly overriding that value then :) | 16:45 |
| *** Longgeek has quit IRC | 16:45 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for wrapt stable global reqs issues https://review.openstack.org/136049 | 16:45 |
| mtreinish | mriedem: ^^^ | 16:45 |
| *** tnovacik has quit IRC | 16:45 | |
| clarkb | I think the most direct fix is likely to not have a long lived connection | 16:46 |
| mriedem | looking | 16:46 |
| clarkb | just reconnect for each project as necessary | 16:46 |
| mtreinish | jenkins is back, I'm just going to fast approve my reqs backport to unblock the icehouse gate | 16:47 |
| clarkb | and we can probably take git-review's test harness and throw it into jeepyb | 16:47 |
| clarkb | it may not catch all of these issues (due to mysql being sane and local not trove) but it will be better than our current testing ability | 16:48 |
| *** omrim has quit IRC | 16:48 | |
| fungi | clarkb: jeblair: https://storyboard.openstack.org/#!/story/2000017 | 16:48 |
| jeblair | fungi: thanks... now we just need to trick someone into owning it :) | 16:50 |
| *** otherwiseguy has joined #openstack-infra | 16:50 | |
| *** mkolesni__ has joined #openstack-infra | 16:50 | |
| *** mkolesni__ has quit IRC | 16:51 | |
| anteaya | initial draft of email here: https://etherpad.openstack.org/p/TNfXmwbOPb | 16:51 |
| *** harlowja_at_home has joined #openstack-infra | 16:52 | |
| fungi | i've assigned it to me, but could use some input on the best direction to take the fix. whatever i propose will likely have a "big hammer" style implementation | 16:52 |
| clarkb | fungi: my suggestion is new connection per project or operatin | 16:53 |
| clarkb | opening a tcp socket is relatively cheap | 16:53 |
| fungi | clarkb: easy enough | 16:53 |
| fungi | (though it's not just a new tcp socket, it's authentication and whatever else that entails too) | 16:53 |
| clarkb | mysql, authentication? :P | 16:54 |
| *** rfolco has joined #openstack-infra | 16:54 | |
| *** nikhil_k is now known as nikhil_k|afk | 16:54 | |
| fungi | yeah, or whatever they call authentication anyway | 16:54 |
| fungi | though i've heard through the grapevine that rax is working on implementing x509 key authentication for trove connections | 16:54 |
| *** liam__ has joined #openstack-infra | 16:55 | |
| *** liam__ has quit IRC | 16:55 | |
| *** ildikov has joined #openstack-infra | 16:55 | |
| fungi | along with tls encryption of the socket | 16:55 |
| fungi | it's not network isolation, but it's a start | 16:55 |
| clarkb | that would be nice to have | 16:55 |
| fungi | they apparently lack a new enough trove to support connecting into arbitrary neutron tenant networks yet | 16:56 |
| *** amitgandhinz has joined #openstack-infra | 16:56 | |
| fungi | which is ultimately what we want, i think | 16:56 |
| clarkb | maybe, I think we would have to redeploy all our nodes to make use of that | 16:57 |
| clarkb | which we may do for trusty anyways | 16:57 |
| fungi | well, yeah, probably. or at least the ones we want talking to trove databases | 16:57 |
| *** emagana has joined #openstack-infra | 16:57 | |
| clarkb | (you cannot add a new port to a running VM with neutron aiui) | 16:57 |
| fungi | not all our instances use trove | 16:58 |
| *** shashankhegde has joined #openstack-infra | 16:58 | |
| clarkb | thats true. the number that do is relatively small | 16:58 |
| fungi | just, you know, the painful ones to replace | 16:58 |
| *** hdd has quit IRC | 16:58 | |
| clarkb | :) | 16:58 |
| *** hdd has joined #openstack-infra | 17:00 | |
| harlowja_at_home | mtreinish, https://review.openstack.org/#/c/136058/ hopefully should do the trick for taskflow also | 17:00 |
| *** bdpayne has joined #openstack-infra | 17:00 | |
| mtreinish | harlowja_at_home: well that's actually not exactly needed for this particular case, because wrapt is in stable/juno reqs | 17:01 |
| *** yfried has joined #openstack-infra | 17:01 | |
| *** andreykurilin__ has quit IRC | 17:01 | |
| harlowja_at_home | did it just get there? | 17:01 |
| mtreinish | harlowja_at_home: but long term that's probably better | 17:01 |
| mtreinish | harlowja_at_home: it just blew up on icehouse | 17:01 |
| *** atiwari has joined #openstack-infra | 17:02 | |
| *** amitgandhinz has quit IRC | 17:02 | |
| harlowja_at_home | right, but icehouse does have a version cap, so that confuses me, ha | 17:02 |
| mtreinish | harlowja_at_home: it looks like you added wrapt back in may | 17:02 |
| harlowja_at_home | https://github.com/openstack/requirements/blob/stable/icehouse/global-requirements.txt#L127 | 17:02 |
| mtreinish | harlowja_at_home: devstack is still installing taskflow from git on icehouse | 17:03 |
| harlowja_at_home | ah | 17:03 |
| *** amitgandhinz has joined #openstack-infra | 17:03 | |
| *** viglesias has quit IRC | 17:03 | |
| harlowja_at_home | is that the intention? | 17:04 |
| *** tsg_ has quit IRC | 17:04 | |
| *** Guest27879 is now known as redrobot | 17:04 | |
| *** viglesias has joined #openstack-infra | 17:04 | |
| *** mudassirlatif has joined #openstack-infra | 17:05 | |
| anteaya | fungi: for added fun after puppet-openstack-cloud merged EmilienM offered a patch which is now in merge conflict: https://review.openstack.org/#/c/136055/ | 17:05 |
| ryanpetrello | ever since the wheel distribution stuff happened, I've noticed that pecan no longer has pypy distributions uploaded | 17:05 |
| *** Guest63270 has quit IRC | 17:05 | |
| ryanpetrello | anybody know what i need to do to make this happen again? | 17:05 |
| anteaya | neither EmilienM nor I know what to do in this situation | 17:05 |
| *** MaxV has quit IRC | 17:06 | |
| ryanpetrello | an OpenStack project using pecan has failing pypy tests because it the last pypy upload of pecan from a few months back has a bug in it | 17:06 |
| *** markmcclain has quit IRC | 17:06 | |
| ryanpetrello | (which I've since fixed, but no new pypy distro uploaded) | 17:06 |
| clarkb | ryanpetrello: I would have to check but we only do universal wheels or py2k wheels | 17:06 |
| clarkb | did we ever do pypy specific wheels? | 17:06 |
| ryanpetrello | dunno | 17:07 |
| ryanpetrello | I don't think so? | 17:07 |
| *** MaxV has joined #openstack-infra | 17:07 | |
| ryanpetrello | hrm... | 17:07 |
| fungi | anteaya: EmilienM: abandon that. we don't update project configurations through the gerrit ui, we do it through configuration management deployed from files in the openstack-infra/project-config repo (look in the gerrit subdirectory for the acl you need to edit) | 17:07 |
| ryanpetrello | actually there's a source distribution uploaded for the latest pecan | 17:07 |
| clarkb | ryanpetrello: but pypy should use sdists | 17:07 |
| ryanpetrello | I wonder why pypy won't use it... | 17:07 |
| ryanpetrello | right | 17:07 |
| ryanpetrello | weird... | 17:07 |
| *** banix has quit IRC | 17:07 | |
| anteaya | fungi: ah it was the contents of the patch, not the timing | 17:08 |
| *** changbl has joined #openstack-infra | 17:08 | |
| ryanpetrello | okay, I'll keep digging | 17:08 |
| clarkb | ryanpetrello: ya I think figuring out why an old wheel is prefered to current sdist would be important | 17:08 |
| ryanpetrello | it's not | 17:08 |
| fungi | anteaya: yeah, the "branch: refs/meta/config" is what you have to keep an eye out for there | 17:08 |
| *** uvirtbot has joined #openstack-infra | 17:08 | |
| ryanpetrello | I was mistaken in thinking that a sdist wasn't uploaded | 17:08 |
| ryanpetrello | a sdist *is* preferable | 17:09 |
| clarkb | oh gotcha | 17:09 |
| anteaya | fungi: ah ha, I missed that, sorry | 17:09 |
| ryanpetrello | I need to work w/ this project and figure out why their tox isn't grabbing the latest sdist for pypy tests | 17:09 |
| ryanpetrello | maybe they mistakenly have something pinned | 17:09 |
| *** uvirtbot has quit IRC | 17:09 | |
| *** harlowja_at_home has quit IRC | 17:09 | |
| *** yamamoto_ has quit IRC | 17:09 | |
| openstackgerrit | Merged openstack-infra/zuul: Pin GitPython version to prevent test failure https://review.openstack.org/135782 | 17:10 |
| anteaya | jeblair clarkb waiting for your thoughts on this email when you have a moment: https://etherpad.openstack.org/p/TNfXmwbOPb | 17:10 |
| openstackgerrit | Merged openstack-infra/zuul: Add gerrit owner information to status.json https://review.openstack.org/135386 | 17:10 |
| *** rfolco has quit IRC | 17:10 | |
| *** markmcclain has joined #openstack-infra | 17:11 | |
| clarkb | anteaya: it looks ok. we might specifically ask for people with rax trove implementation knowledge to chime in but I THink we can work around this without that info | 17:11 |
| jeblair | anteaya: lgtm, thanks! | 17:11 |
| anteaya | clarkb: do you want to add that in? | 17:12 |
| anteaya | jeblair: how should it be signed? | 17:12 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for wrapt stable global reqs issues https://review.openstack.org/136049 | 17:12 |
| *** uvirtbot has joined #openstack-infra | 17:12 | |
| *** tsg_ has joined #openstack-infra | 17:12 | |
| clarkb | anteaya: nah I don't think its necessary | 17:13 |
| *** uvirtbot has quit IRC | 17:14 | |
| anteaya | clarkb: okay | 17:14 |
| anteaya | jeblair: from me, from me on behalf of infra? | 17:14 |
| *** tgohad has joined #openstack-infra | 17:14 | |
| anteaya | from someone not me? | 17:14 |
| jeblair | anteaya: i'd just sign it with my own name, but if you want to invoke some more authority, feel free to say 'on behalf of the infrastructure team' or something [i personally do not think that's necessary for you] | 17:14 |
| anteaya | ha ha ha | 17:14 |
| anteaya | okay my name it is | 17:15 |
| *** markmcclain has quit IRC | 17:15 | |
| *** uvirtbot has joined #openstack-infra | 17:15 | |
| asselin | anyone seen this on icehouse devstack setup? 'wrapt' is not a global requirement but it should be,something went wrong | 17:16 |
| *** yamahata has quit IRC | 17:16 | |
| *** zul has quit IRC | 17:16 | |
| *** andreykurilin_ has joined #openstack-infra | 17:16 | |
| *** uvirtbot has quit IRC | 17:17 | |
| mtreinish | asselin: the fix is in the gate currently | 17:17 |
| asselin | mtreinish, ok thanks. | 17:17 |
| mtreinish | asselin: https://review.openstack.org/136018 | 17:17 |
| *** tsg_ has quit IRC | 17:17 | |
| *** uvirtbot has joined #openstack-infra | 17:18 | |
| *** zul has joined #openstack-infra | 17:18 | |
| *** ChuckC has joined #openstack-infra | 17:19 | |
| *** sabeen2 has quit IRC | 17:19 | |
| *** uvirtbot has quit IRC | 17:19 | |
| *** timrc is now known as timrc-afk | 17:19 | |
| *** gyee has joined #openstack-infra | 17:20 | |
| *** sabeen1 has joined #openstack-infra | 17:20 | |
| *** uvirtbot has joined #openstack-infra | 17:20 | |
| *** uvirtbot has quit IRC | 17:21 | |
| *** vrutkovs has quit IRC | 17:22 | |
| *** uvirtbot has joined #openstack-infra | 17:23 | |
| *** dkranz has quit IRC | 17:23 | |
| *** dtantsur is now known as dtantsur|afk | 17:24 | |
| *** gyee has quit IRC | 17:24 | |
| *** uvirtbot has quit IRC | 17:24 | |
| *** gyee has joined #openstack-infra | 17:24 | |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/zuul: Process event multiple times. https://review.openstack.org/136027 | 17:24 |
| *** Viswanath has joined #openstack-infra | 17:24 | |
| *** jpich has quit IRC | 17:25 | |
| *** gyee has quit IRC | 17:25 | |
| *** gyee has joined #openstack-infra | 17:26 | |
| *** gyee has quit IRC | 17:26 | |
| *** gyee has joined #openstack-infra | 17:26 | |
| *** baoli has quit IRC | 17:26 | |
| *** shashankhegde has quit IRC | 17:27 | |
| *** emagana has quit IRC | 17:27 | |
| jogo | clarkb: http://logs.openstack.org/31/135831/1/experimental/check-tempest-dsvm-aiopcpu/1109b15/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-11-20_03_49_01_648 | 17:27 |
| *** Viswanath has quit IRC | 17:27 | |
| jogo | looks like we need to set update the host file | 17:27 |
| *** emagana has joined #openstack-infra | 17:27 | |
| clarkb | jogo: ya lifeless said we would have to | 17:27 |
| *** uvirtbot has joined #openstack-infra | 17:27 | |
| jogo | clarkb: I was hoping the clouds set up DNS for us | 17:28 |
| clarkb | lol | 17:28 |
| jogo | but its openstack so nope | 17:28 |
| *** uvirtbot has quit IRC | 17:28 | |
| *** emagana has quit IRC | 17:28 | |
| *** nelsnels_ has joined #openstack-infra | 17:28 | |
| clarkb | I would append to /etc/hosts on each file in a way similar to how the ssh known hosts file is populated | 17:28 |
| *** nelsnels_ has quit IRC | 17:28 | |
| clarkb | make a master list on the primary node then dump that into etc hosts everywhere | 17:28 |
| *** emagana has joined #openstack-infra | 17:29 | |
| *** nelsnels_ has joined #openstack-infra | 17:29 | |
| jeblair | clarkb, jogo: why are we using hostnames? | 17:29 |
| *** nelsnelson has quit IRC | 17:29 | |
| jogo | jeblair: because libvirt? | 17:30 |
| jeblair | jogo: ip addresses don't work? | 17:30 |
| jogo | jeblair: not even sure if we can set that option | 17:30 |
| jogo | I'll ask around | 17:30 |
| jogo | as that would make more sense | 17:30 |
| jeblair | jogo: normally i would just expect an ip address to work in place of a hostname wherever that's being set | 17:31 |
| jogo | jeblair: I would to, but not sure where its set | 17:32 |
| jogo | jeblair: looking into it now | 17:32 |
| jogo | jeblair: as that is a useful thing in general not just in our test env | 17:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Set dib vars to override tmp and cache locations https://review.openstack.org/136069 | 17:33 |
| clarkb | jeblair: fungi ^ so I tested using both of those vars with latest dib on my local test box. it seems happy | 17:33 |
| jogo | jeblair clarkb: talking to danpb in -nova about it | 17:33 |
| clarkb | jeblair: lifeless explicitly said ips don't work (he may be wrong but current working data is lolibvirt) | 17:34 |
| *** banix has joined #openstack-infra | 17:34 | |
| *** hashar has quit IRC | 17:34 | |
| clarkb | also vpc images are huge | 17:34 |
| openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/135955 | 17:34 |
| *** nikhil_k|afk is now known as nikhil_k | 17:35 | |
| *** dimtruck is now known as zz_dimtruck | 17:35 | |
| *** otherwiseguy has quit IRC | 17:35 | |
| *** otherwiseguy has joined #openstack-infra | 17:36 | |
| *** dkranz has joined #openstack-infra | 17:36 | |
| *** baoli has joined #openstack-infra | 17:37 | |
| emagana | pleia2: Just wanted to say Hi! | 17:37 |
| *** MaxV has quit IRC | 17:37 | |
| pleia2 | emagana: hey :) I'm in Jamaica right now, flying home soon! I should be around more tomorrow as I wade through vacation email backlog | 17:38 |
| pleia2 | no wifi on my flights today, like the dark ages (United) | 17:38 |
| emagana | pleia2: Have a nice and safe trip back! | 17:38 |
| pleia2 | thank you | 17:38 |
| sdague | pleia2: do we have to give you vacation lessons as well :) | 17:39 |
| anteaya | emagana is going to help pleia2 and I admin the third-party-announce list | 17:39 |
| emagana | anteaya: I am glad to help.. | 17:40 |
| jogo | jeblair clarkb: looks like we can make nova use IP addresses (with a nova patch or two) | 17:40 |
| pleia2 | sdague: haha, right right, getting back to catching up on some fiction books | 17:40 |
| *** otter768 has joined #openstack-infra | 17:40 | |
| anteaya | emagana: thank you! | 17:40 |
| clarkb | sdague: see comment on https://review.openstack.org/#/c/134283/ | 17:42 |
| *** matrohon has quit IRC | 17:43 | |
| sdague | clarkb: yep, I believe you are correct | 17:44 |
| *** tonytan4ever has quit IRC | 17:44 | |
| *** otter768 has quit IRC | 17:44 | |
| openstackgerrit | Sean Dague proposed openstack-infra/zuul: implement gerrit pagination in simplequery https://review.openstack.org/134283 | 17:45 |
| *** jamesmcarthur has left #openstack-infra | 17:47 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/gear: Support binding to a specified host name or address https://review.openstack.org/135832 | 17:47 |
| *** AlexF has quit IRC | 17:47 | |
| *** bhunter71 has quit IRC | 17:48 | |
| *** viglesias has quit IRC | 17:49 | |
| *** viglesias has joined #openstack-infra | 17:50 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/zuul: Gear: bind to a specified address https://review.openstack.org/135837 | 17:50 |
| *** bhuvan has joined #openstack-infra | 17:52 | |
| *** yfried has quit IRC | 17:53 | |
| *** dustins has joined #openstack-infra | 17:54 | |
| *** pradk has quit IRC | 17:54 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:55 | |
| *** alazarev has left #openstack-infra | 17:55 | |
| *** AlexF has joined #openstack-infra | 17:56 | |
| *** emagana has quit IRC | 17:56 | |
| *** Guest56789 has quit IRC | 17:57 | |
| *** emagana has joined #openstack-infra | 17:57 | |
| *** rmcall has quit IRC | 17:57 | |
| *** rmcall has joined #openstack-infra | 17:58 | |
| *** derekh has quit IRC | 17:58 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add plugins_info to module registry object. https://review.openstack.org/132927 | 17:59 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8 https://review.openstack.org/135843 | 17:59 |
| *** achanda has joined #openstack-infra | 17:59 | |
| *** Ark has joined #openstack-infra | 18:01 | |
| *** Ark has quit IRC | 18:01 | |
| *** emagana has quit IRC | 18:01 | |
| *** harlowja_away is now known as harlowja | 18:02 | |
| *** koolhead17 has joined #openstack-infra | 18:02 | |
| *** koolhead17 has joined #openstack-infra | 18:02 | |
| *** jlibosva has quit IRC | 18:03 | |
| *** shashankhegde has joined #openstack-infra | 18:03 | |
| *** e0ne has quit IRC | 18:03 | |
| *** dprince has joined #openstack-infra | 18:03 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:03 | |
| *** amuller has quit IRC | 18:04 | |
| *** emagana has joined #openstack-infra | 18:05 | |
| *** ivar-lazzaro has quit IRC | 18:05 | |
| *** viglesias has quit IRC | 18:05 | |
| *** rfolco has joined #openstack-infra | 18:05 | |
| *** viglesias has joined #openstack-infra | 18:06 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:06 | |
| *** tkelsey has quit IRC | 18:07 | |
| anteaya | nibalizer: can you expand the commit message on https://review.openstack.org/#/c/133252/1 to say why auto upgrading puppetboard is useful and also why it would be safe to do so | 18:07 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/jeepyb: Reconnect to Gerrit database if needed https://review.openstack.org/136079 | 18:08 |
| *** hdd has quit IRC | 18:09 | |
| *** yamamoto has joined #openstack-infra | 18:10 | |
| *** mriedem has quit IRC | 18:10 | |
| *** links has quit IRC | 18:11 | |
| harlowja | so, qq, is there anyway for taskflow to have a CI job (local to taskflow) that tests icehouse as well? that would of avoided being hit by this requirement issue i think | 18:13 |
| nibalizer | anteaya: sure | 18:13 |
| nibalizer | ill need some time though | 18:13 |
| anteaya | nibalizer: np | 18:13 |
| *** tonytan4ever has joined #openstack-infra | 18:13 | |
| anteaya | do you want me to comment on the patch? | 18:13 |
| *** krtaylor has quit IRC | 18:13 | |
| fungi | harlowja: depending on the details of the job you're seeking, we likely already have a job template you can just add | 18:13 |
| *** melwitt has joined #openstack-infra | 18:13 | |
| *** yamamoto has quit IRC | 18:14 | |
| fungi | harlowja: what icehouse projects/interactions are you wanting to test? | 18:14 |
| harlowja | fungi well taskflow (via me) added a requirement for a library that wasn't in icehouse requirements, but it was in juno requirements, and afaik taskflow jobs only go back to juno to see if things are ok, not as far back as icehouse | 18:14 |
| clarkb | yes so sdague has all of this set up | 18:15 |
| clarkb | sdague: ^ do we need to do the lib forward testing thing for taskflow too? | 18:15 |
| harlowja | maybe the solution is different though (i think this was releated to devstack installing from git master on stable branches?) | 18:15 |
| fungi | harlowja: based on what i was reading earlier, it sounds like the only reason it didn't break icehouse is that you added it to global requirements master before icehouse was released | 18:15 |
| fungi | er, s/icehouse/juno/ | 18:15 |
| *** krtaylor has joined #openstack-infra | 18:16 | |
| *** amcrn has joined #openstack-infra | 18:16 | |
| harlowja | gotta add it sometime :-P | 18:16 |
| fungi | harlowja: regardless, i think if it wasn't for that lengthy delay between adding it to global reqs and using it in a taskflow release, juno would have been broken too | 18:17 |
| *** david-lyle has quit IRC | 18:17 | |
| harlowja | ya, review took a damn long time and more than alot of iterations, lol | 18:17 |
| fungi | harlowja: responding to your comment about something being apparently tested on juno but not icehouse | 18:17 |
| *** mriedem has joined #openstack-infra | 18:17 | |
| *** patrickeast has joined #openstack-infra | 18:17 | |
| fungi | i think it's tested on neither and juno just got lucky | 18:17 |
| sdague | clarkb: yes... but I think it's correct | 18:17 |
| *** Ryan_Lane has joined #openstack-infra | 18:17 | |
| harlowja | fungi fair enough, sorta makes sense | 18:18 |
| clarkb | cool so we should catch this in the future then? | 18:18 |
| sdague | clarkb: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1481 | 18:18 |
| sdague | clarkb: not without the stable-compat jobs | 18:19 |
| EmilienM | fungi: ack | 18:19 |
| harlowja | sdague what does 'stable-compat ' do? should that be added? | 18:19 |
| clarkb | sdague: ya just noticed thats missing | 18:19 |
| clarkb | harlowja: it does exactly what you were asking for | 18:19 |
| harlowja | ah | 18:19 |
| harlowja | well sweet then | 18:19 |
| sdague | clarkb: not all the libraries had stable compat jobs previously | 18:20 |
| sdague | and... honestly, I think that in the stable case it makes more sense to pin libraries on semver boundaries | 18:20 |
| *** dizquierdo has quit IRC | 18:20 | |
| clarkb | sdague: ya me too | 18:20 |
| harlowja | https://review.openstack.org/#/c/136058/ hopefully helps in that pinning; although devstack i think is still installing from master on icehouse (which i guess i can change?) | 18:21 |
| *** rfolco has quit IRC | 18:22 | |
| *** emagana has quit IRC | 18:22 | |
| *** ryanpetrello_ has joined #openstack-infra | 18:22 | |
| *** emagana has joined #openstack-infra | 18:23 | |
| *** koolhead17 has quit IRC | 18:23 | |
| *** che-arne has quit IRC | 18:23 | |
| *** wuhg has quit IRC | 18:23 | |
| harlowja | so if i also adjust devstack on icehouse to install from the requirements file (not from git) that would seem to also fix this (and follows more of the semver boundaries idea) | 18:23 |
| *** ryanpetrello has quit IRC | 18:25 | |
| *** ryanpetrello_ is now known as ryanpetrello | 18:25 | |
| *** otherwiseguy has quit IRC | 18:25 | |
| *** sputnik13 has joined #openstack-infra | 18:25 | |
| clarkb | ok my puppet env seems to work properly | 18:25 |
| clarkb | fungi: how were you seeing that fail? | 18:25 |
| clarkb | I can either boot new nodepool using https://review.openstack.org/#/c/136069/1 unmerged or wait for it to merge. it should be safe to put that on old nodepool | 18:26 |
| clarkb | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 387: cd: /opt/stack/new/devstack: No such file or directory <- well :( I guess I should figure out why that is thrashing us | 18:27 |
| fungi | clarkb: i no longer remember now. it was failing to raise puppet syntax errors during launch, which i knew to be present, and wound up only half-installing the subunit2sql worker class as a result | 18:28 |
| *** HeOS has quit IRC | 18:28 | |
| clarkb | its happening in rax and hpcloud so not an image issue unless we have consistently bad images | 18:28 |
| fungi | clarkb: but i didn't dig deeply since i was in a hurry to work around it so i could get to bed | 18:28 |
| *** radez is now known as radez_g0n3 | 18:29 | |
| clarkb | http://logs.openstack.org/62/135962/1/gate/gate-tempest-dsvm-postgres-full/e8d2223/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-11-20_18_16_46_398 | 18:30 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all doesn't look terrible so not a bandwidth thing | 18:30 |
| *** vhoward has left #openstack-infra | 18:31 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=877&rra_id=all is probably the cause | 18:32 |
| *** cnesa2 has quit IRC | 18:32 | |
| clarkb | do we just need more git backends? | 18:32 |
| *** Ark has joined #openstack-infra | 18:32 | |
| *** cnesa has joined #openstack-infra | 18:32 | |
| clarkb | memory doesn't seem to be an issue on 04, no swapping and lots of memory free | 18:33 |
| *** Ark is now known as Guest76265 | 18:33 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=881&rra_id=all hrm maybe it is an issue just not right this second | 18:33 |
| clarkb | (oh ya no swap on these machines) | 18:33 |
| *** vhoward has joined #openstack-infra | 18:33 | |
| clarkb | OOMKiller has been busy | 18:34 |
| fungi | right, the oom killer kicks in when we get spikes like that on the git servers, which results in the socket going away from the client's perspective | 18:34 |
| jeblair | yes, we should add swap :) | 18:34 |
| fungi | sdague had a theory that this was red hat's ci testing afazekas's nodepool prep scripts change to perform all git clone operations in parallel | 18:34 |
| *** dkranz has quit IRC | 18:35 | |
| clarkb | WCPGW | 18:35 |
| jeblair | has that been abandoned yet? | 18:35 |
| fungi | i was not able to isolate a specific offender in the log when i tried to post-mortem analyze the last occurrence though | 18:35 |
| sdague | calling it a theory is strong words | 18:36 |
| fungi | jeblair: yeah, afazekas abandoned it, but doesn't mean they weren't already using it internally | 18:36 |
| *** ZZelle_ has joined #openstack-infra | 18:36 | |
| sdague | I just happened to notice the parallel clone patch was up at the same time | 18:36 |
| fungi | i never managed to get confirmation though, so idle speculation at best | 18:36 |
| clarkb | should I carve up xvde like we normally do and get 8GB of swap and put the rest on /opt? | 18:36 |
| fungi | clarkb: that sounds fine | 18:37 |
| *** sputnik13 has quit IRC | 18:37 | |
| clarkb | ok I will do that on the 5 nodes shortly | 18:37 |
| fungi | thanks | 18:37 |
| fungi | i'm trying to manually recreate the manage-projects failure so i can confirm 136079 solves it | 18:38 |
| clarkb | (rebuilding these nodes should do that automagically so future builds will be ok) | 18:38 |
| *** uvirtbot has joined #openstack-infra | 18:38 | |
| *** uvirtbot has quit IRC | 18:39 | |
| *** koolhead17 has joined #openstack-infra | 18:39 | |
| *** koolhead17 has joined #openstack-infra | 18:39 | |
| *** uvirtbot has joined #openstack-infra | 18:40 | |
| *** uvirtbot has quit IRC | 18:41 | |
| *** uvirtbot has joined #openstack-infra | 18:43 | |
| *** uvirtbot has quit IRC | 18:43 | |
| *** markmcclain has joined #openstack-infra | 18:43 | |
| *** bdpayne has quit IRC | 18:44 | |
| *** rushiagr is now known as rushiagr_away | 18:45 | |
| *** markmcclain has quit IRC | 18:45 | |
| *** dprince has quit IRC | 18:45 | |
| *** viglesias has quit IRC | 18:45 | |
| *** viglesias has joined #openstack-infra | 18:46 | |
| *** uvirtbot has joined #openstack-infra | 18:46 | |
| *** dannywilson has quit IRC | 18:47 | |
| *** uvirtbot has quit IRC | 18:47 | |
| *** dannywilson has joined #openstack-infra | 18:48 | |
| *** markmcclain has joined #openstack-infra | 18:48 | |
| *** nadya_ has quit IRC | 18:48 | |
| *** dkranz has joined #openstack-infra | 18:48 | |
| *** talluri has joined #openstack-infra | 18:48 | |
| *** mpaolino has quit IRC | 18:48 | |
| *** uvirtbot has joined #openstack-infra | 18:48 | |
| clarkb | ok git04 has swap | 18:49 |
| clarkb | started there since it seemed least happy. will now do the others | 18:49 |
| *** uvirtbot has quit IRC | 18:49 | |
| mtreinish | harlowja: fwiw, I also have https://review.openstack.org/136017 up to run the stable tests against master taskflow | 18:50 |
| *** zul has quit IRC | 18:51 | |
| harlowja | mtreinish ah, cool, i was looking at also figuring out how to get devstack to use the bounded requirements, although maybe not needed anymore | 18:51 |
| * harlowja unsure where that override seems to be set, haha | 18:51 | |
| *** andreykurilin_ has quit IRC | 18:52 | |
| mtreinish | harlowja: sdague has done work on devstack master to do that | 18:52 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:52 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=880&rra_id=all is a bit telling | 18:53 |
| clarkb | fungi: sdague ^ and probably does support that something somewhere is going crazy | 18:53 |
| harlowja | mtreinish ya there is a 'TASKFLOW_BRANCH' but if we are restricting requirements to be bounded on stable, it seems like devstack should also respect those same bounds and not checkout from git anymore? (i might be wrong in my understanding, ha) | 18:54 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Neutron-vSphere project to StackForge https://review.openstack.org/136091 | 18:54 |
| *** weshay has quit IRC | 18:55 | |
| mtreinish | harlowja: no it's just that taskflow was configured on devstack in icehouse to install the lib from git, you can just drop that part on icehouse and it should install based on the reqs file | 18:55 |
| *** dannywilson has quit IRC | 18:55 | |
| harlowja | mtreinish hehe, i'm trying to figure out what part to drop :) | 18:55 |
| harlowja | remove all the code! | 18:56 |
| harlowja | ha | 18:56 |
| mtreinish | harlowja: https://github.com/openstack-dev/devstack/blob/stable/icehouse/lib/oslo#L62 | 18:56 |
| *** markmcclain1 has joined #openstack-infra | 18:56 | |
| harlowja | kk | 18:56 |
| harlowja | thx :) | 18:56 |
| harlowja | should that whole function ('install_oslo') exist if we are going to be using stable bounded requirements on stable branches? | 18:57 |
| fungi | clarkb: agreed, but the number of open sockets increasing may be a symptom of performance degredation too (git server taking considerably longer to service requests, hence more open and waiting for a response) | 18:57 |
| *** dannywilson has joined #openstack-infra | 18:57 | |
| morganfainberg | 2014.2 = stable/juno right? | 18:57 |
| clarkb | fungi: good point | 18:57 |
| fungi | morganfainberg: that is correct | 18:57 |
| morganfainberg | fungi, thanks | 18:57 |
| *** markmcclain has quit IRC | 18:57 | |
| *** sarob has joined #openstack-infra | 18:57 | |
| mtreinish | harlowja: that I don't know, I'd talk with dhellmann about how he wants to ensure compat with oslo libs on the stable branches | 18:57 |
| sdague | clarkb: when did it go crazy again? | 18:57 |
| harlowja | mtreinish k | 18:57 |
| fungi | morganfainberg: at least for some definitions of == | 18:57 |
| morganfainberg | fungi, LOL | 18:57 |
| morganfainberg | fungi, doing releasemngmt stuff today in LP | 18:58 |
| * morganfainberg begs Storyboard to open to other projects besides infra ;) | 18:58 | |
| clarkb | fungi: in fact that pileup is likely why the old bandwidth limits were problematic | 18:58 |
| clarkb | sdague: right now | 18:58 |
| anteaya | the patch to remove pypi mirror from static.o.o has 3 +2's and 3 +1's is it waiting for anything other than someone to +A? https://review.openstack.org/#/c/119466/ | 18:58 |
| sdague | I guess this is too offset from requirements flood to link it to that | 18:58 |
| fungi | morganfainberg: some projects now have patches on stable/juno since their 2014.2 tags and hence may not be completely equal, depending on what you're asking | 18:58 |
| morganfainberg | fungi, notably tagging bugs / bps | 18:58 |
| morganfainberg | to the milestones | 18:58 |
| sdague | clarkb: do we have request logging for git? | 18:58 |
| harlowja | dhellmann let me know when u get around, i'm wondering if https://github.com/openstack-dev/devstack/blob/stable/icehouse/lib/oslo#L36 should be needed anymore (being that we are going to bound requirements on stable branches?) | 18:58 |
| sdague | it would be nice to know who's doing that | 18:59 |
| clarkb | sdague: we do | 18:59 |
| clarkb | I just haven't had time to go dig through them yet | 18:59 |
| * dhellmann reads scrollback for harlowja | 18:59 | |
| *** Viswanath has joined #openstack-infra | 19:00 | |
| sdague | harlowja: it will become this - https://github.com/openstack-dev/devstack/blob/stable/juno/lib/oslo#L54 | 19:00 |
| anteaya | dhellmann: an I have a pending request for you once you are done with scrollback | 19:00 |
| harlowja | sdague intersting | 19:00 |
| clarkb | ok swap is achieved | 19:00 |
| sdague | at least if I can manage the backport | 19:00 |
| harlowja | ah cool, so backport will enable this selective turn on/off better | 19:01 |
| dhellmann | harlowja: what sdague said, since we still need to test changes to stable branches of libs somehow | 19:01 |
| harlowja | ok dokie, so seems like its being fixed up/adjusted/made better anyway | 19:01 |
| sdague | I just put in the stable/juno backport of the whole client library stack | 19:01 |
| sdague | I collapsed the patches for that to make it a little easier not to miss a piece after | 19:02 |
| sdague | once that seems right, I'll figure out icehouse, which is going to be a bit more painful | 19:02 |
| sdague | in juno we got oslo in before the cut | 19:02 |
| *** bdpayne has joined #openstack-infra | 19:02 | |
| dhellmann | anteaya: what's up? | 19:03 |
| harlowja | sdague sounds great, thanks for the info | 19:03 |
| *** rmcall has quit IRC | 19:03 | |
| anteaya | dhellmann: this patch adds jobs to oslo repos: https://review.openstack.org/#/c/136017/ | 19:03 |
| anteaya | dhellmann: your opinion is gratefully recieved | 19:04 |
| *** MarkAtwood has joined #openstack-infra | 19:04 | |
| dhellmann | anteaya: looking | 19:05 |
| anteaya | thanks | 19:05 |
| *** rmcall has joined #openstack-infra | 19:05 | |
| *** Viswanath has quit IRC | 19:07 | |
| *** rmcall has quit IRC | 19:08 | |
| *** e0ne has joined #openstack-infra | 19:09 | |
| dhellmann | anteaya: I don't understand what that patch is doing. Does that mean we won't be able to add dependencies to oslo libraries until they are also available in the stable branch? | 19:10 |
| sdague | dhellmann: that is correct, though it's basically always been true | 19:11 |
| dhellmann | sdague: I'm not sure that's true. This seems like it's going to introduce pain. | 19:11 |
| sdague | no, it's always been true | 19:11 |
| sdague | that's actually how the system works now | 19:11 |
| *** otter768 has joined #openstack-infra | 19:12 | |
| dhellmann | does this apply to other projects, too? | 19:12 |
| sdague | which one can argue is wrong, but it's not a new thing | 19:12 |
| sdague | it applies to anything that doesn't have a stable/ branch | 19:12 |
| dhellmann | so it doesn't apply to nova? | 19:12 |
| sdague | correct | 19:12 |
| dhellmann | but it does to the clients? | 19:12 |
| sdague | correct | 19:12 |
| *** salv-orlando has quit IRC | 19:12 | |
| *** weshay has joined #openstack-infra | 19:13 | |
| dhellmann | ok. why is this good? | 19:13 |
| dhellmann | apparently it didn't apply to oslo before because we're adding it now :-) | 19:13 |
| clarkb | ok I have 64899 log lines for connections since 1720 on git04 | 19:13 |
| *** rmcall has joined #openstack-infra | 19:13 | |
| clarkb | time to clean these up and see if I can find anything interesting | 19:13 |
| sdague | dhellmann: it did in principle, not in code | 19:13 |
| harlowja | sdague although if clients, libraries start to add version caps, that cap should help avoid these kinds of issues right? | 19:13 |
| dhellmann | sdague: well, yeah | 19:13 |
| sdague | dhellmann: which is why stable/icehouse requirements change is currently being rushed through | 19:14 |
| dhellmann | harlowja: that's a good point, if we're going to cap the libs do we need this? | 19:14 |
| sdague | because tempest is blocked for the second time this week | 19:14 |
| *** dannywilson has quit IRC | 19:14 | |
| sdague | dhellmann: probably not, but someone needs to actually work out that whole system coherently, and I'm not sure there is yet a volunteer for it | 19:15 |
| *** dannywilson has joined #openstack-infra | 19:15 | |
| sdague | right now there is a lot of "we could", but no real owner on it | 19:15 |
| *** jp_at_hp has quit IRC | 19:15 | |
| *** baoli has quit IRC | 19:15 | |
| *** pelix has quit IRC | 19:15 | |
| dhellmann | sdague: I hate to rehash this, but I'm in a distracting environment right now so I'm having trouble remembering why we are testing requirements updates in this way -- if it's a secondary dependency, why does it matter if we add it to the stable versions? | 19:15 |
| *** patrickeast_ has joined #openstack-infra | 19:16 | |
| *** baoli has joined #openstack-infra | 19:16 | |
| *** baoli has quit IRC | 19:16 | |
| sdague | because you try to do a icehouse job | 19:16 |
| dhellmann | as long as oslo isn't being singled out, I'm ok with the change, so the question is just to help me understand more than anything | 19:16 |
| dhellmann | ? | 19:17 |
| sdague | some library is uncapped in icehouse | 19:17 |
| sdague | but icehouse enforces a requirements list | 19:17 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 19:17 |
| EmilienM | anteaya: I tried something ^ | 19:17 |
| sdague | that library adds something we don't support | 19:17 |
| anteaya | EmilienM: /me looks | 19:17 |
| sdague | we delete the requirement | 19:17 |
| *** patrickeast has quit IRC | 19:17 | |
| *** patrickeast_ is now known as patrickeast | 19:17 | |
| sdague | it blows up on install | 19:17 |
| sdague | the job fails | 19:17 |
| sdague | we can no longer run icehouse jobs | 19:17 |
| dhellmann | sdague: right, but why do we care about the secondary dependency? | 19:17 |
| dhellmann | I understand the effect that's happening, I'm trying to understand why we have that as an explicit test | 19:18 |
| *** mudassirlatif_ has joined #openstack-infra | 19:18 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 19:18 |
| *** mudassirlatif has quit IRC | 19:18 | |
| *** mudassirlatif_ is now known as mudassirlatif | 19:18 | |
| dhellmann | is it just because we want distros to be able to keep track of the dependencies for stable? because we do change that list, right? | 19:18 |
| sdague | because that project signed up for being enforced by global reqs | 19:18 |
| dhellmann | I'm looking for the root of the policy, not the "because I said so" answer :-) | 19:19 |
| *** zz_dimtruck is now known as dimtruck | 19:19 | |
| dhellmann | s/I/we/ | 19:19 |
| sdague | dhellmann: there isn't a giant coherent policy, there is what's managed to work | 19:19 |
| dhellmann | ok | 19:19 |
| sdague | and the reason there isn't a giant coherent policy is that no one has signed up to drive it | 19:20 |
| dhellmann | like I said, I'm in a room full of people all talking so thinking coherently is harder than usual | 19:20 |
| dhellmann | k | 19:20 |
| sdague | dhellmann: yeh, this probably also has a lot of moving parts too | 19:20 |
| dhellmann | yeah | 19:20 |
| dhellmann | ok, well, as long as oslo is not being singled out here I'm OK with that patch | 19:20 |
| *** AlexF has quit IRC | 19:20 | |
| dhellmann | it's going to be a PITA, though | 19:20 |
| sdague | anyway, I'm *not* signing up for fixing this one. I think there are things that could be better, but it needs an owner. | 19:20 |
| * dhellmann touches nose and says "not it" | 19:21 | |
| harlowja | not it! | 19:21 |
| harlowja | lol | 19:21 |
| sdague | yeh, well that's the problem, no one wants to be it on this. So it will be a PITA for a long time :) | 19:22 |
| dhellmann | sdague: is someone working on the changes needed to let us stop using alpha versions safely? I guess we need a devstack change before we can cap libs? | 19:22 |
| dhellmann | sdague: maybe next cycle | 19:22 |
| harlowja | i still wonder about caps just on stable branches, then just install things from pip using that cap, maybe have non-voting jobs that install from master to tell the library/project that something is broken on an older version (maybe due to a new requirement) | 19:22 |
| clarkb | 75.98.92.113 is our worst ddoser with 7k connections | 19:23 |
| *** vigneshvar has quit IRC | 19:23 | |
| sdague | harlowja: that sounds suspiciously like you are signing up to be it | 19:23 |
| clarkb | hopefully that isn't me | 19:23 |
| fungi | nicira | 19:23 |
| harlowja | sdague, lol errrr, maybe, ha, i'll think about its | 19:23 |
| * harlowja more of just brainstorming | 19:23 | |
| anteaya | clarkb: is is just hitting git04? | 19:23 |
| sdague | dhellmann: I'm not sure I know what those changes are | 19:24 |
| clarkb | anteaya: yes, we load balance based on a hash of your IP so all of that IP's connections went to git04 | 19:24 |
| anteaya | ah | 19:24 |
| fungi | anteaya: the load balancer balances to a specific git server based on source ip address | 19:24 |
| clarkb | next closest is 64.119.130.115 with 700 connections | 19:24 |
| fungi | what clarkb said | 19:24 |
| clarkb | so 7k is prett heavy | 19:24 |
| anteaya | agree | 19:24 |
| dhellmann | sdague: the devstack changes you and harlowja were talking about a few minutes ago, I think -- install from libs packages instead of source | 19:24 |
| *** Sukhdev has joined #openstack-infra | 19:24 | |
| sdague | dhellmann: so install from pypi is in master | 19:24 |
| clarkb | fungi: how did you determine its nicira? | 19:25 |
| fungi | clarkb: whois? | 19:25 |
| clarkb | dig -x gave me back pnap.net | 19:25 |
| dhellmann | sdague: right, but we need it on the stable branch before we can cap the libs, otherwise we get the install-then-downgrade dance | 19:25 |
| *** hdd has joined #openstack-infra | 19:25 | |
| fungi | clarkb: yeah, they're hosting with internap apparently, hence the ptr | 19:25 |
| clarkb | oh derp of course whois. I need caffeine | 19:25 |
| *** Guest76265 has quit IRC | 19:25 | |
| anteaya | you were up early | 19:25 |
| fungi | clarkb: internap has subdelegated 75.98.92.112/29 to nicira | 19:25 |
| clarkb | ya I see the customer entry | 19:26 |
| fungi | at least in the whois record but apparently not in their reverse dns zone | 19:26 |
| clarkb | and that one node is 1/10th of the connections to git04 in that timespan | 19:26 |
| clarkb | so ya | 19:26 |
| jeblair | clarkb: what's the timespan? | 19:26 |
| *** viglesias has quit IRC | 19:27 | |
| *** viglesias has joined #openstack-infra | 19:27 | |
| clarkb | jeblair: 1720 to 1904 UTC today | 19:27 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Neutron-vSphere project to StackForge https://review.openstack.org/136091 | 19:27 |
| *** unicell has joined #openstack-infra | 19:28 | |
| harlowja | sdague dhellmann maybe i can start an email about this to the ML, see if we can get some kind of plan together, i think there are pieces that exist in various places, but nothing unified yet | 19:28 |
| anteaya | holy holy | 19:29 |
| harlowja | and seems like it needs a 'strategic sequencing' to work | 19:29 |
| fungi | clarkb: jeblair: should we consider dropping in an iptables rule for that host on git.o.o and seeing if the load trails back off into reasonableness? | 19:29 |
| anteaya | 7k in less than 2 hours | 19:29 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 19:29 |
| sdague | harlowja: I don't think an ML thread is honestly all that useful here | 19:30 |
| jeblair | fungi: thinking | 19:30 |
| clarkb | they are all git protocol connections so logs are sparse on what they are actually fetching | 19:30 |
| clarkb | we do have byte totals though | 19:31 |
| clarkb | (I haven't calculated the byte total) | 19:31 |
| fungi | clarkb: also possibly worth running the same analysis looking at 17:30-18:30 for git01 and git05 which saw bad spikes for a shorter duration | 19:31 |
| sdague | because I think the people who have thoughts about this issue are pretty small, and all here. The big bit is getting the whole picture written down some where so it's clear we're not just trading one set of edge conditions for another | 19:31 |
| sdague | so a spec or etherpad would be better | 19:31 |
| *** teran_ has quit IRC | 19:31 | |
| *** rwsu has quit IRC | 19:32 | |
| fungi | clarkb: er, ~17:10-18:15 for git01 (the spikes aren't completely identical timing apparently) | 19:32 |
| *** jamespage_ has joined #openstack-infra | 19:32 | |
| *** ddieterly has quit IRC | 19:32 | |
| *** emagana has quit IRC | 19:32 | |
| *** rwsu has joined #openstack-infra | 19:33 | |
| *** emagana has joined #openstack-infra | 19:33 | |
| *** Ark has joined #openstack-infra | 19:34 | |
| *** jedimike has quit IRC | 19:34 | |
| *** Ark is now known as Guest20252 | 19:34 | |
| *** vigneshvar has joined #openstack-infra | 19:35 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Streamlined task edit form. https://review.openstack.org/135739 | 19:36 |
| *** jamespage__ has joined #openstack-infra | 19:36 | |
| clarkb | fungi: ya I can run those numbers too, but the sun just came out and I am hungry so I will do that after lunch | 19:36 |
| *** e0ne has quit IRC | 19:37 | |
| *** emagana has quit IRC | 19:38 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 19:40 |
| *** jamespage_ has quit IRC | 19:40 | |
| anteaya | EmilienM: that's it! | 19:40 |
| EmilienM | anteaya: I'm not really good in Gerrit, I should learn more :) | 19:40 |
| *** talluri has quit IRC | 19:40 | |
| EmilienM | anteaya: thank your for your help | 19:41 |
| anteaya | np | 19:41 |
| anteaya | that is what I am here for | 19:41 |
| *** jamespage__ has quit IRC | 19:41 | |
| anteaya | now, do you need to have stable branches for this repo? | 19:41 |
| anteaya | or is tagging sufficient? | 19:41 |
| *** AJaeger has joined #openstack-infra | 19:42 | |
| *** jamespage__ has joined #openstack-infra | 19:42 | |
| anteaya | you will note you got rid of branches and kept tags for the transfer | 19:42 |
| jeblair | clarkb, fungi: i think it might be interesting to see how many simultaneous connections they use | 19:42 |
| EmilienM | anteaya: we would need the same as Puppet modules, so yes we need stable/juno/ etc | 19:43 |
| *** radez_g0n3 is now known as radez | 19:44 | |
| *** tonytan4ever has quit IRC | 19:44 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Neutron-vSphere project to StackForge https://review.openstack.org/136091 | 19:44 |
| fungi | jeblair: according to netstat on git.o.o, 3 right this moment | 19:44 |
| clarkb | jeblair: ya looking at cacti graphs I think that would be relevant. its a little harder to get out of the logs | 19:44 |
| anteaya | EmilienM: okay | 19:45 |
| *** krtaylor has quit IRC | 19:45 | |
| dhellmann | harlowja: see the "solving the client libs stable dilemma" thread, I think | 19:45 |
| *** ddieterly has joined #openstack-infra | 19:46 | |
| *** zul has joined #openstack-infra | 19:48 | |
| *** uvirtbot has joined #openstack-infra | 19:49 | |
| anteaya | if a project was created with an acl url pointing to an acl file named something other than $project.config, can we just remove the acl-config line to get it to point to $project.config after a new patch is merged? | 19:49 |
| clarkb | yup | 19:51 |
| *** unicell has quit IRC | 19:51 | |
| AJaeger | anteaya: but you need to add $project.config as well | 19:51 |
| *** packet has joined #openstack-infra | 19:52 | |
| anteaya | oh yes | 19:52 |
| anteaya | but in gerrit/projects.yaml: https://review.openstack.org/#/c/136098/4/gerrit/projects.yaml | 19:52 |
| anteaya | is removing the acl-config line enough to get to point it to the new .config file | 19:52 |
| fungi | jeblair: according to netstat on git.o.o, checked again and see 10 now | 19:52 |
| fungi | so seems fairly variable | 19:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 19:53 |
| AJaeger | anteaya: I hope that the parser does the right thing - and clarkb seems to confirm this. | 19:54 |
| anteaya | okay, we will find out | 19:54 |
| AJaeger | anteaya: so, let's -1 and ask EmilienM to remove the line | 19:54 |
| anteaya | yes, that was going to be my next move, thanks | 19:55 |
| harlowja | dhellmann k, good point, that one does basically cover the similar/same scenario | 19:55 |
| anteaya | EmilienM: you're up | 19:56 |
| *** mpaolino has joined #openstack-infra | 19:56 | |
| fungi | harlowja: yeah, i meant for it to really be a blanket lib/middleware/client solution, but the clients are the easiest to demonstrate the issue with | 19:56 |
| harlowja | fungi agreed | 19:57 |
| fungi | for a broader audience not steeped in oslo minutiae anyway | 19:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 19:57 |
| EmilienM | anteaya: ^ | 19:58 |
| *** krtaylor has joined #openstack-infra | 19:58 | |
| anteaya | EmilienM: nicely done, +2 | 19:58 |
| EmilienM | cool, thx | 19:58 |
| anteaya | let's see what AJaeger says | 19:59 |
| AJaeger | anteaya: I just wonder about the create line for refs/* - those still confuse me a bit... | 19:59 |
| anteaya | me as well | 19:59 |
| AJaeger | anteaya, did we agree on the solution that is used here? | 19:59 |
| anteaya | but the master config file he is going off of is the puppet-config | 19:59 |
| *** Guest20252 has quit IRC | 19:59 | |
| anteaya | and I do believe it is necessary for branch creation | 20:00 |
| * AJaeger needs to write some documentation for this - let me write an email and then document | 20:00 | |
| anteaya | since there is no create line in the */stable/* permissions | 20:00 |
| anteaya | of course | 20:00 |
| *** bhuvan has quit IRC | 20:00 | |
| AJaeger | anteaya: have to run to my next conference session, will followup later... | 20:00 |
| *** AJaeger has quit IRC | 20:00 | |
| *** stevemar has quit IRC | 20:03 | |
| *** erikwilson has quit IRC | 20:03 | |
| *** stevemar has joined #openstack-infra | 20:03 | |
| *** MarkAtwood has quit IRC | 20:04 | |
| *** emagana has joined #openstack-infra | 20:04 | |
| *** bhuvan has joined #openstack-infra | 20:06 | |
| *** melwitt has quit IRC | 20:06 | |
| *** markmcclain1 has quit IRC | 20:08 | |
| *** markmcclain has joined #openstack-infra | 20:08 | |
| *** melwitt1 has joined #openstack-infra | 20:08 | |
| *** hashar has joined #openstack-infra | 20:09 | |
| *** melwitt1 is now known as melwitt | 20:09 | |
| *** MarkAtwood has joined #openstack-infra | 20:09 | |
| fungi | hrm, salv-orlando and rhsu don't hang out in here i guess | 20:10 |
| fungi | i was hoping one of them might be able to tell us if that's their system | 20:10 |
| anteaya | salv quit irc at 19:12 | 20:14 |
| *** alazarev has joined #openstack-infra | 20:15 | |
| *** ldnunes has quit IRC | 20:15 | |
| anteaya | I just emailed him | 20:16 |
| fungi | was mainly curious if he knew what that was and what it was doing. i'm still not sure that the correlation to open connections is causal, nor that the client which made the most connections over a given span of time necessarily imposed the load we're seeing | 20:17 |
| anteaya | right | 20:17 |
| *** erikwilson has joined #openstack-infra | 20:17 | |
| fungi | i'm trying to put together byte totals per client from the logs now to see if there are other interesting correlations | 20:17 |
| *** erikwilson has quit IRC | 20:17 | |
| anteaya | my email just said nicira is hitting our git farm hard, can he join us and help to figure out if it is their ci | 20:18 |
| anteaya | if he gets the email, he will show up | 20:18 |
| fungi | thanks | 20:18 |
| anteaya | np | 20:18 |
| *** rediskin has left #openstack-infra | 20:18 | |
| *** viglesias has quit IRC | 20:19 | |
| *** viglesias has joined #openstack-infra | 20:20 | |
| *** shashankhegde has quit IRC | 20:20 | |
| mtreinish | fungi: heh, salv-orlando's ci system is what is generating the load on the git servers? | 20:20 |
| *** nadya has joined #openstack-infra | 20:20 | |
| anteaya | we don't know | 20:20 |
| fungi | mtreinish: no idea, just trying to collect more data | 20:20 |
| anteaya | but we thought we would ask the question | 20:20 |
| *** nadya is now known as Guest54107 | 20:20 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch dsvm-docker as a Nova check jobs https://review.openstack.org/136115 | 20:22 |
| mtreinish | anteaya: https://review.openstack.org/136017 got doug's ok if you wanted to +A it | 20:23 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch dsvm-docker as a Nova check jobs https://review.openstack.org/136115 | 20:24 |
| anteaya | mtreinish: thanks, on it goes | 20:25 |
| *** hdd has quit IRC | 20:25 | |
| anteaya | salv says he will be back after dinner as he is out right now, I told him to enjoy his dinner as this isn't a fire | 20:26 |
| fungi | yep, awesome | 20:26 |
| anteaya | he suggested rhsu or arosen neither of whom are here either | 20:27 |
| *** eharney has quit IRC | 20:29 | |
| *** jamespage__ has quit IRC | 20:31 | |
| *** nelsnels_ has quit IRC | 20:34 | |
| *** viglesias has quit IRC | 20:35 | |
| *** viglesias has joined #openstack-infra | 20:36 | |
| *** melwitt has quit IRC | 20:36 | |
| *** melwitt has joined #openstack-infra | 20:37 | |
| *** vigneshvar has quit IRC | 20:38 | |
| *** viglesias has quit IRC | 20:39 | |
| *** pc_m has quit IRC | 20:39 | |
| *** viglesias has joined #openstack-infra | 20:39 | |
| *** nelsnelson has joined #openstack-infra | 20:39 | |
| *** viglesias has quit IRC | 20:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Add stable compat jobs to oslo libs in stable requirements https://review.openstack.org/136017 | 20:40 |
| *** viglesias has joined #openstack-infra | 20:40 | |
| dhellmann | mtreinish: could you announce ^^ with a description of the implications on the -dev ML to avoid surprises amongst oslo devs, please? | 20:40 |
| *** jamespage__ has joined #openstack-infra | 20:42 | |
| *** jamespage__ has quit IRC | 20:44 | |
| *** salv-orlando has joined #openstack-infra | 20:44 | |
| *** HeOS has joined #openstack-infra | 20:45 | |
| * harlowja time to recheck some active reviews to see what happens :-/ | 20:48 | |
| *** patrickeast has quit IRC | 20:48 | |
| *** otherwiseguy has joined #openstack-infra | 20:48 | |
| *** mudassirlatif has quit IRC | 20:48 | |
| mtreinish | dhellmann: sure I'll do that when I get home | 20:48 |
| dhellmann | mtreinish: thanks! | 20:48 |
| *** dannywilson has quit IRC | 20:49 | |
| *** patrickeast has joined #openstack-infra | 20:49 | |
| *** andreykurilin_ has joined #openstack-infra | 20:50 | |
| *** dannywilson has joined #openstack-infra | 20:50 | |
| *** atiwari has quit IRC | 20:50 | |
| *** Guest54107 has quit IRC | 20:51 | |
| *** koolhead17 has quit IRC | 20:51 | |
| *** mudassirlatif has joined #openstack-infra | 20:51 | |
| *** teran has joined #openstack-infra | 20:51 | |
| *** banix has quit IRC | 20:53 | |
| *** banix has joined #openstack-infra | 20:53 | |
| *** rlandy has quit IRC | 20:53 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 20:53 | |
| *** r1chardj0n3s has left #openstack-infra | 20:54 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/hacking: Enforce import group ordering https://review.openstack.org/54403 | 20:56 |
| dougwig | jeblair: got a minute for a gertty question? i seem to lose my inline comments whenever reviewing with gertty. does this ring any bells? | 20:57 |
| *** sarob has quit IRC | 20:57 | |
| salv-orlando | hi anteaya | 20:58 |
| *** yfried has joined #openstack-infra | 20:59 | |
| *** dimtruck is now known as zz_dimtruck | 20:59 | |
| fungi | salv-orlando: trying to figure out if 75.98.92.113 in nicira is your ci... saw 7000 connections from it in the span of a couple hours | 20:59 |
| salv-orlando | So today I’ve increased the code paths that we pick for the NSX CI. However, this did not result in a big increase in the number of jobs executed (about 60). And most importantly we use an internal git mirror | 20:59 |
| gondoi | does anyone know what is being used to filter out talk proposals when voting goes out? in other words the web tool that is tallying votes, etc | 20:59 |
| salv-orlando | oh well our addresses are Natted | 21:00 |
| *** emagana has quit IRC | 21:01 | |
| *** emagana has joined #openstack-infra | 21:01 | |
| *** sarob has joined #openstack-infra | 21:02 | |
| salv-orlando | fungi: the address might be on of ours | 21:02 |
| anteaya | salv-orlando: I hope you didn't rush your dinner | 21:03 |
| anteaya | gondoi: talk submissions and voting are handled by the foundation | 21:03 |
| *** emagana has quit IRC | 21:04 | |
| *** kgiusti has left #openstack-infra | 21:04 | |
| *** emagana has joined #openstack-infra | 21:04 | |
| anteaya | gondoi: looks like info@openstack.org might be a place to begin: http://www.openstack.org/foundation/staff | 21:04 |
| gondoi | anteaya: thanks | 21:04 |
| anteaya | np | 21:04 |
| *** MarkAtwood has quit IRC | 21:05 | |
| *** shashankhegde has joined #openstack-infra | 21:06 | |
| fungi | clarkb: summarizing the logfile by the bytes_read column it looks like between 17:30 and 19:30 on git04 75.98.92.113 came in sixth | 21:07 |
| *** weshay has quit IRC | 21:08 | |
| *** Bart__ has joined #openstack-infra | 21:09 | |
| *** mrmartin has quit IRC | 21:09 | |
| *** Bart has quit IRC | 21:09 | |
| salv-orlando | fungi: we have identified an issue an a CI somebody in my team is developing. For a couple of hours it kept pulling nodepool continuosly. Now it’s been stopped. | 21:09 |
| anteaya | salv-orlando: thank you | 21:10 |
| *** e0ne has joined #openstack-infra | 21:10 | |
| fungi | salv-orlando: okay, that almost certainly explains the burst we saw from that address. thanks a bunch! | 21:10 |
| *** arosen has joined #openstack-infra | 21:10 | |
| salv-orlando | fungi: np, we’ll switch that system to our git mirror as well | 21:10 |
| *** zz_dimtruck is now known as dimtruck | 21:10 | |
| anteaya | awesome | 21:10 |
| anteaya | turn around time is faster when you ddos yourself | 21:11 |
| anteaya | :D | 21:11 |
| fungi | salv-orlando: does the timing mostly coincide with the elevated usage seen on the graph at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=878&rra_id=all | 21:11 |
| arosen | fungi: sorry about that. That was my nodepool node that i was working on getting dib working and it kept failing and doing the same thing over again | 21:11 |
| arosen | and i left it running :( | 21:11 |
| fungi | arosen: hah, oops! | 21:11 |
| arosen | I'm suprised you guys noticed | 21:11 |
| *** david-lyle has joined #openstack-infra | 21:11 | |
| arosen | it's pulling everything serially so i didn't thing it's really that much data per say | 21:11 |
| arosen | i figured some CI systems do a fresh clone anyways | 21:12 |
| *** david-lyle has quit IRC | 21:12 | |
| arosen | so i figure you guys have a huge volume of traffic anyways | 21:12 |
| fungi | arosen: unfortunately our current git replication mechanism doesn't ensure synchronization across our load-balanced pool, so we persist requests with a source address affinity | 21:12 |
| *** yamamoto has joined #openstack-infra | 21:12 | |
| fungi | thus increased load from a single source all slams one pool member | 21:12 |
| fungi | rather than being spread evenly across the farm | 21:13 |
| arosen | gotcha | 21:13 |
| *** david-lyle has joined #openstack-infra | 21:13 | |
| arosen | fungi: random question: so I think i tried what you were suggesting to get my public_key in the puppet scripts by changing prepare-node but i'm getting this: http://paste.openstack.org/show/135496/ | 21:14 |
| *** bhuvan has quit IRC | 21:16 | |
| fungi | arosen: it's just ssh_key there http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/single_use_slave.pp#n20 | 21:16 |
| *** banix has quit IRC | 21:16 | |
| arosen | fungi: so I should just change this line to say ssh_key instead and that's it? | 21:16 |
| fungi | arosen: yeah, i think that should work | 21:17 |
| *** yamamoto has quit IRC | 21:17 | |
| arosen | fungi: awesome thanks let me give that a try :) | 21:17 |
| fungi | arosen: i'm still not convinced that sustained nicira connection burst was the culprit for elevated load on our git04 server (or perhaps not solely responsible) but if the impact seen on http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=878&rra_id=all roughly lines up with your failure period, then maybe | 21:18 |
| fungi | it had the highest number of connections opened during that timeframe, but not the highest number of bytes transferred | 21:19 |
| arosen | yea it was running for probably the last 14 hours running and failing on the dib creation | 21:19 |
| arosen | should only be one connection at a time w/e nodepool does for git cloning | 21:20 |
| anteaya | the 12 hours prior to the 2 hour burst show normal activity levels on the graph | 21:20 |
| arosen | fungi: anyways sorry about that | 21:20 |
| anteaya | arosen: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=878&rra_id=all | 21:20 |
| arosen | that's pretty awesome that you guys are monitoring all of that and exposing it :) | 21:21 |
| fungi | arosen: we do that so that _you_ can help _us_ diagnose our problems! ;) | 21:21 |
| arosen | makes sense :) | 21:21 |
| *** viglesias has quit IRC | 21:21 | |
| *** nolafleur has quit IRC | 21:22 | |
| clarkb | fungi what nodes pulled more bytes? maybe we have really stale images? | 21:23 |
| clarkb | and ya the 7k connsctions may not be related | 21:23 |
| anteaya | well johnthetubaguy posted in channel a day or so ago that rax dfw images were broken | 21:24 |
| anteaya | I'm not sure if that is related or if that is still the case | 21:24 |
| *** Sukhdev has quit IRC | 21:24 | |
| fungi | clarkb: top 20 byte consumers for git04 between 17:30 and 20:20 (which basically covers the prolonged elevation of usage we see in cacti) http://paste.openstack.org/show/135507/ | 21:24 |
| fungi | clarkb: i'm about to start checking those against nodepool logs now | 21:25 |
| *** mpaolino has quit IRC | 21:25 | |
| clarkb | sounds good | 21:25 |
| fungi | clarkb: but if you want to play around with other analyses, that script in my homedir is only a couple dozen lines | 21:26 |
| *** markvan has quit IRC | 21:26 | |
| *** banix has joined #openstack-infra | 21:27 | |
| *** markvan has joined #openstack-infra | 21:27 | |
| fungi | looks like for that span, 75.98.92.113 in nicira was third place still | 21:27 |
| fungi | clarkb: of the top 20 listed there, 3 show up in our nodepool debug log | 21:31 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for testtools/tempest bug 1394747 https://review.openstack.org/136131 | 21:31 |
| uvirtbot | Launchpad bug 1394747 in tempest "gate-tempest-dsvm-neutron-heat-slow fails with "The test run didn't actually run any tests"" [Critical,Fix released] https://launchpad.net/bugs/1394747 | 21:31 |
| *** fifieldt__ has joined #openstack-infra | 21:31 | |
| clarkb | fungi but only 3 I guess that means our caching mostly works :) | 21:32 |
| *** baoli has joined #openstack-infra | 21:32 | |
| *** baoli has quit IRC | 21:33 | |
| *** baoli has joined #openstack-infra | 21:33 | |
| *** weshay has joined #openstack-infra | 21:34 | |
| fungi | the top two are in rackspace networks, assigned according to whois as Cloud Servers Cell 0001-0003 IAD3 | 21:34 |
| *** fifieldt_ has quit IRC | 21:34 | |
| fungi | as is the fourth | 21:35 |
| fungi | the fifth one is in a rackspace network assigned to OpenStack-DevQE IP Space | 21:36 |
| fungi | in rackspace uk | 21:36 |
| fungi | i'm pretty sure none of that is us | 21:38 |
| clarkb | jeblair: any chance I can get eyes on https://review.openstack.org/#/c/136069/1 ? it is the next step in nodepool dib work (needed prior to redeploying on trusty performance node) | 21:38 |
| clarkb | fungi: is any of them nodepool.o.o? | 21:38 |
| clarkb | hrm not if iad | 21:38 |
| fungi | clarkb: nope | 21:38 |
| fungi | clarkb: also that spike i'm analyzing began well after image updates would have completed | 21:39 |
| clarkb | fungi: ya I noticed that too | 21:39 |
| clarkb | 1700ish is ~2 hours after image builds should complete | 21:40 |
| *** Ark has joined #openstack-infra | 21:40 | |
| *** Ark is now known as Guest40820 | 21:40 | |
| mtreinish | clarkb, fungi: https://review.openstack.org/#/c/136018/ bounced off the gate (because of a swift failure which is rare) can you do me a favor and just ninja merge it? | 21:41 |
| *** _buzz_ has joined #openstack-infra | 21:41 | |
| mtreinish | it's blocked the tempest queue all day | 21:41 |
| mtreinish | as well as stable | 21:41 |
| asselin | clarkb, blocking our stuff too | 21:43 |
| *** _buzz has quit IRC | 21:43 | |
| clarkb | so | 21:44 |
| clarkb | I am not huge fan of adding a new dep to stable branches | 21:44 |
| clarkb | I thought we were fixing this with an older version of taskflow? | 21:45 |
| *** larsks has left #openstack-infra | 21:45 | |
| mtreinish | clarkb: it's a devstack thing, it's installing from git which is pulling in the new req | 21:45 |
| clarkb | these are the sorts of things that get you yelled at by zigo | 21:45 |
| clarkb | mtreinish: but we can make it not do that | 21:45 |
| clarkb | (and have for a bunch of other libs) | 21:45 |
| mtreinish | zigo: said it's already in debian so he was good with it | 21:46 |
| mtreinish | clarkb: sdague is working on a backport to do that | 21:46 |
| anteaya | clarkb: zigo commented on the patch | 21:46 |
| clarkb | anteaya: yes and said what I am saying | 21:46 |
| clarkb | "isn't the solution to cap taskflow" | 21:46 |
| mtreinish | clarkb: it is capped | 21:46 |
| *** melwitt has quit IRC | 21:47 | |
| clarkb | fungi: ^ do you have an opinion? | 21:47 |
| *** MarkAtwood has joined #openstack-infra | 21:48 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for testtools/tempest bug 1394747 https://review.openstack.org/136131 | 21:49 |
| uvirtbot | Launchpad bug 1394747 in tempest "gate-tempest-dsvm-neutron-heat-slow fails with "The test run didn't actually run any tests"" [Critical,Fix released] https://launchpad.net/bugs/1394747 | 21:49 |
| jeblair | clarkb: looking | 21:50 |
| *** e0ne has quit IRC | 21:50 | |
| fungi | mtreinish: clarkb: so, yeah, that's a huge mess we've created i guess... right now we can't cap any libs in stable if they're libs devstack installs from git | 21:51 |
| *** hashar has quit IRC | 21:51 | |
| fungi | mtreinish: clarkb: that workaround definitely feels terrible, but i guess the alternative is a significant rearchitecture of our devstack-based jobs which would block things for much longer | 21:52 |
| clarkb | fungi: ya, I think that terriblness makes me wary of force merging | 21:52 |
| jeblair | clarkb: dib change lgtm | 21:52 |
| clarkb | jeblair: danke | 21:52 |
| clarkb | jeblair: ^ you might be interseted in this discussion too | 21:52 |
| mtreinish | clarkb: well I can recheck it and we'll wait another ~12 hours to work through the gate (or I guess we can promote it to the top too which would only be 1hr assuming no random ceilo failures again) | 21:54 |
| fungi | end control over which libs are installed from git and which from pip are in the hands of devstack, right? so overriding the taskflow install behavior would have to be done in devstack stable/icehouse? | 21:55 |
| clarkb | ya I think that is what sdague is backporting | 21:55 |
| *** JayJ has quit IRC | 21:55 | |
| mtreinish | fungi: yes sdague has the patches on master he just started working it backwards | 21:56 |
| mtreinish | icehouse will probably be a bit of work to get backported | 21:56 |
| jeblair | clarkb: i agree that we should not force-merge a change that touches dependency handling (unless we have no other choice) because i'm pretty sure we never know the full extent of the repercussions from something like that | 21:56 |
| *** baoli has quit IRC | 21:57 | |
| fungi | i guess this seems like a not-entirely-terrible temporary workaround until icehouse devstack catches up with our newer testing design, so i'm okay with reenqueuing and promoting that change for now to get stuff in stable flowing again | 21:57 |
| *** emagana has quit IRC | 21:58 | |
| mtreinish | fungi: it's more than just stable though, it's any project that has a stable job in check or gate, which is a lot | 21:58 |
| mtreinish | but I'll run a recheck now so you guys can promote it | 21:58 |
| *** emagana has joined #openstack-infra | 21:58 | |
| fungi | mtreinish: no need to recheck | 21:58 |
| *** baoli has joined #openstack-infra | 21:59 | |
| mtreinish | oops too late | 21:59 |
| *** emagana has quit IRC | 22:01 | |
| *** baoli has quit IRC | 22:01 | |
| *** baoli has joined #openstack-infra | 22:01 | |
| *** emagana has joined #openstack-infra | 22:01 | |
| openstackgerrit | Merged openstack-infra/system-config: Set dib vars to override tmp and cache locations https://review.openstack.org/136069 | 22:02 |
| dtroyer | really I think the fix for added dependencies that aren't in global like this should be to revert the dep-add review for not following process | 22:02 |
| *** dkehnm has quit IRC | 22:03 | |
| *** bhuvan has joined #openstack-infra | 22:03 | |
| *** dkehn has quit IRC | 22:03 | |
| *** dkehnm has joined #openstack-infra | 22:03 | |
| mtreinish | dtroyer: oh, yeah that would work too | 22:03 |
| mtreinish | harlowja: ^^^ | 22:03 |
| *** julim has quit IRC | 22:04 | |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add Castellan project https://review.openstack.org/136140 | 22:04 |
| *** dkehn has joined #openstack-infra | 22:05 | |
| *** jgrimm is now known as zz_jgrimm | 22:06 | |
| *** denis_makogon_ has joined #openstack-infra | 22:06 | |
| *** bswartz has quit IRC | 22:06 | |
| *** banix has quit IRC | 22:07 | |
| *** denis_makogon has quit IRC | 22:08 | |
| *** denis_makogon_ is now known as denis_makogon | 22:09 | |
| *** dmakogon_ has joined #openstack-infra | 22:09 | |
| *** melwitt has joined #openstack-infra | 22:09 | |
| *** sarob has quit IRC | 22:10 | |
| clarkb | my phone just updated to lollipop and I am super distracted by the bright colors | 22:12 |
| *** radez is now known as radez_g0n3 | 22:12 | |
| *** rwsu has quit IRC | 22:12 | |
| lifeless | clarkb: isn't it annoying | 22:12 |
| lifeless | also the new icons are WAT. | 22:12 |
| clarkb | lifeless: they are playstation icons | 22:12 |
| *** sarob has joined #openstack-infra | 22:12 | |
| clarkb | triangle, circle, square | 22:13 |
| lifeless | clarkb: with totally different meanings AFAICT. | 22:13 |
| *** markmcclain has quit IRC | 22:13 | |
| *** cnesa has quit IRC | 22:13 | |
| clarkb | ya triangle is ps circle | 22:13 |
| clarkb | and square acts like the home buttun | 22:13 |
| clarkb | and circle acts like start | 22:13 |
| lifeless | thus WAT | 22:14 |
| clarkb | thankfully they didn't move them too | 22:14 |
| lifeless | someone let 'designers' into Google. | 22:14 |
| mtreinish | clarkb: heh, yeah I got a nexus 9 and I was distracted for a bit by that too | 22:14 |
| *** radez_g0n3 is now known as radez | 22:14 | |
| *** rwsu_ has joined #openstack-infra | 22:15 | |
| *** ryanpetrello has quit IRC | 22:15 | |
| *** mjturek has quit IRC | 22:15 | |
| *** stevemar2 has joined #openstack-infra | 22:17 | |
| *** stevemar has quit IRC | 22:17 | |
| *** tykeal has joined #openstack-infra | 22:18 | |
| *** stevemar2 has quit IRC | 22:18 | |
| tykeal | heya mestery | 22:19 |
| *** stevemar2 has joined #openstack-infra | 22:19 | |
| *** rwsu_ has quit IRC | 22:19 | |
| mestery | tykeal: Hi there :) | 22:19 |
| tykeal | ok... who do I need to give cookie pictures to get the ODL CI re-instated? ;) | 22:19 |
| *** packet has quit IRC | 22:20 | |
| mestery | anteaya can likely help us here :) | 22:20 |
| *** sputnik13 has joined #openstack-infra | 22:20 | |
| mestery | anteaya: We're trying to get the ODL CI voting on the sandbox repo, but since our ssh access was disabled it's proving challenging :) | 22:20 |
| * tykeal notes he did just reply to the third-party-announce CI disabled mail about this | 22:20 | |
| mestery | fungi: Also see ^^^ | 22:21 |
| harlowja | mtreinish i can revert if needed; the functionatliy wrapt was providing was just reducing code in taskflow that was doing something similar | 22:21 |
| anteaya | tykeal: okay so the way forward is to trigger a run or two with your ci by using your private gerrit account to listen to the stream | 22:22 |
| mtreinish | harlowja: ok, well the req patch is at the top of the queue now, if it bounces again we can fast path the revert | 22:22 |
| harlowja | mtreinish sounds good to me | 22:22 |
| tykeal | anteaya: ah... I see | 22:22 |
| *** baoli has quit IRC | 22:22 | |
| anteaya | ssh -p 29418 <personal gerrit account username>@review.openstack.org gerrit stream-events | 22:23 |
| *** baoli has joined #openstack-infra | 22:23 | |
| tykeal | that is going to prove difficult for me :-/ the SSH key I used for my personal access is on a hardware token. Guess I'll have to add a disk based key | 22:23 |
| anteaya | then with the ip address of those logs in hand, reply to the list and two operators verify | 22:23 |
| *** weshay has quit IRC | 22:23 | |
| anteaya | tykeal: do you have a personal gerrit account? | 22:23 |
| tykeal | anteaya: are normal accounts granted stream rights? | 22:23 |
| anteaya | different ssh keys | 22:23 |
| anteaya | yes | 22:23 |
| * tykeal has to dig up what his account is | 22:24 | |
| anteaya | you ssh in with the key for your personal account | 22:24 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Switched use of "Resource.read()" to "Resource.get()" https://review.openstack.org/136148 | 22:24 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 22:24 |
| mestery | anteaya: Thanks for the help here :) | 22:24 |
| *** jamespage_ has joined #openstack-infra | 22:24 | |
| anteaya | different from the ssh key for the jenkins | 22:24 |
| krotscheck | There, that should speed things up a bit. | 22:24 |
| anteaya | mestery: of course | 22:24 |
| tykeal | anteaya: ok, let me see what I can do | 22:24 |
| clarkb | krotscheck: are you optimizing my apps too? | 22:24 |
| anteaya | tykeal: thanks, you only need one or two runs | 22:24 |
| clarkb | google just did that on my phone | 22:24 |
| mestery | tykeal: Also thanks to you :) | 22:24 |
| krotscheck | clarkb: Just a minute. | 22:24 |
| krotscheck | clarkb: Optimizing your apps: 10% | 22:25 |
| anteaya | tykeal: something for 2 other ci operators to look at and say yes, you should be re-enabled | 22:25 |
| krotscheck | clarkb: Optimizing your apps: 22% | 22:25 |
| krotscheck | clarkb: Optimizing your apps: 44% | 22:25 |
| anteaya | tykeal: otherwise it is mayhem we don't know who is actually up and who isn't | 22:25 |
| krotscheck | clarkb: Segmentation Fault: Core dumped. | 22:25 |
| tykeal | anteaya: totally understand | 22:25 |
| *** dustins has quit IRC | 22:25 | |
| fungi | i'm going to go find food and then come back in a while | 22:25 |
| clarkb | krotscheck: damn | 22:25 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1394774 https://review.openstack.org/136151 | 22:26 |
| anteaya | tykeal: thanks for understanding | 22:26 |
| uvirtbot | Launchpad bug 1394774 in tempest "Handling notifications from swift middleware causing tempest timeouts" [Undecided,New] https://launchpad.net/bugs/1394774 | 22:26 |
| * tykeal runs the Gerrit system for ODL and completely understands | 22:26 | |
| anteaya | tykeal: thank you | 22:26 |
| anteaya | :D | 22:26 |
| anteaya | tykeal: we used to take people at their word | 22:27 |
| *** jistr|mtg has quit IRC | 22:27 | |
| anteaya | until we couldn't anymore :( | 22:27 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Networking-vSphere project to StackForge https://review.openstack.org/136091 | 22:27 |
| *** dkliban is now known as dkliban_afk | 22:28 | |
| openstackgerrit | Merged openstack-infra/puppet-httpd: Add .gitreview https://review.openstack.org/135781 | 22:29 |
| clarkb | fungi: git04 looks like it may be spiking again | 22:29 |
| clarkb | load average is 110 and climbing | 22:30 |
| anteaya | armax: so that patch looks good to me | 22:30 |
| armax | anteaya: k, tnx | 22:31 |
| anteaya | armax: I just -2'd it so it wouldn't merge since right now infra is not merging repo creation patches | 22:31 |
| anteaya | armax: once we can I'll free it up and we can move forward | 22:31 |
| armax | anteaya: yes, mestery gave me a heads-up | 22:31 |
| anteaya | thank you | 22:31 |
| clarkb | man lsof -> no manpage for lsof. want to buy real os | 22:31 |
| anteaya | great | 22:31 |
| mestery | armax: Ack | 22:31 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 22:31 |
| ianw | fungi / clarkb : can you help get this one out of my queue : https://review.openstack.org/#/c/115168/ | 22:31 |
| mestery | I'll have one of those same commits hopefully tonight as well armax. | 22:31 |
| *** groknix has joined #openstack-infra | 22:32 | |
| armax | mestery: kwel | 22:32 |
| clarkb | no lsof | 22:32 |
| anteaya | clarkb: any info on where the traffic increase is coming from? | 22:32 |
| * clarkb is going to rage quick | 22:32 | |
| clarkb | er quit | 22:32 |
| anteaya | rage quit your phone? | 22:32 |
| clarkb | anteaya: centos | 22:32 |
| anteaya | ah | 22:32 |
| *** jamespage_ has quit IRC | 22:35 | |
| anteaya | git01, 04 and 05 | 22:35 |
| anteaya | for some reason 02 and 03 don't show the same levels of activity | 22:35 |
| anteaya | for those following along at home: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2 | 22:35 |
| *** emagana has quit IRC | 22:36 | |
| *** sarob has quit IRC | 22:37 | |
| *** emagana has joined #openstack-infra | 22:37 | |
| harlowja | git under attack? | 22:37 |
| anteaya | unknown | 22:37 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 22:37 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 22:37 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Enable live block migration https://review.openstack.org/135768 | 22:37 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 22:37 |
| anteaya | traffic spikes for some unknown reason | 22:37 |
| harlowja | :-/ | 22:37 |
| anteaya | we have no reason to believe it is malicious | 22:38 |
| *** andreykurilin_ has quit IRC | 22:38 | |
| anteaya | just heavy traffic and we don't know why yet | 22:38 |
| *** dkranz has quit IRC | 22:38 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1394774 https://review.openstack.org/136151 | 22:38 |
| *** sarob has joined #openstack-infra | 22:38 | |
| uvirtbot | Launchpad bug 1394774 in tempest "Handling notifications from swift middleware causing tempest timeouts" [Undecided,In progress] https://launchpad.net/bugs/1394774 | 22:38 |
| anteaya | jogo: do not commit, do not merge, do not review, I'm loving the titles of your commit messages | 22:39 |
| jogo | anteaya: I try :) | 22:39 |
| anteaya | ha ha ha | 22:39 |
| * clarkb learns him an haproxy | 22:39 | |
| anteaya | you bring joy to my day, everyday | 22:39 |
| anteaya | :D | 22:39 |
| *** dmsimard is now known as dmsimard_away | 22:40 | |
| *** emagana has quit IRC | 22:41 | |
| jeblair | clarkb: yum install lsof? | 22:42 |
| *** groknix has quit IRC | 22:42 | |
| *** groknix has joined #openstack-infra | 22:42 | |
| clarkb | echo "show sess" | sudo socat unix-connect:/var/lib/haproxy/stats stdio | grep git04 | sed -n -e's/.*src=\(.*\):[0-9]*.*/\1/p' | uniq -c is the current magic I am playing with | 22:43 |
| clarkb | it basically shows that we have single connections from a few places nothing looks too terrible | 22:43 |
| clarkb | and if I remove the git04 grep only one source has more than one session | 22:43 |
| clarkb | however 4 appears to have a large number of the total connections | 22:44 |
| clarkb | 33/62 for 04 | 22:44 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for bug 1394774 https://review.openstack.org/136151 | 22:46 |
| uvirtbot | Launchpad bug 1394774 in tempest "Handling notifications from swift middleware causing tempest timeouts" [Undecided,New] https://launchpad.net/bugs/1394774 | 22:46 |
| ianw | clarkb: also a ping on https://review.openstack.org/#/c/131921/ (Set useful PATH before starting scripts) : hpcloud centos builds still have non-fatal errors about path | 22:46 |
| *** emagana has joined #openstack-infra | 22:46 | |
| clarkb | one thing I notice in cacti graphs is that we seem to hit hardware IRQs when this happens | 22:47 |
| *** dkehn has quit IRC | 22:48 | |
| tykeal | anteaya: thanks for the help, I'll get the runs in sometime soon. gotta run off to a meeting :-/ | 22:49 |
| tykeal | our jenkins is currently setup to not try and trigger until I can babysit a little bit | 22:50 |
| *** dkehn has joined #openstack-infra | 22:50 | |
| clarkb | jeblair: we did swap a little on 04. hopefully that let us keep the connection alive | 22:50 |
| *** sputnik13 has quit IRC | 22:51 | |
| *** sputnik13 has joined #openstack-infra | 22:52 | |
| anteaya | tykeal: sure, enjoy your meeting | 22:52 |
| *** rmcall has quit IRC | 22:52 | |
| anteaya | tykeal: wise set up | 22:52 |
| *** sputnik13 has quit IRC | 22:53 | |
| *** iain-king has joined #openstack-infra | 22:53 | |
| asselin | HI, I'm having difficulting getting periodic jobs to work in zuul. I have them 'running' and e-mailing me every minute, but nothing is actually running | 22:53 |
| *** sputnik13 has joined #openstack-infra | 22:54 | |
| *** rmcall has joined #openstack-infra | 22:54 | |
| asselin | I guess this stack trace, but not sure it's related: http://paste.openstack.org/show/135539/ | 22:54 |
| asselin | get* | 22:55 |
| asselin | gearman status is mostly 0 0 1 for all, some are 0 0 2 | 22:56 |
| *** zul has quit IRC | 22:56 | |
| anteaya | asselin: have periodic jobs ever worked for you? | 22:56 |
| anteaya | as in is this a new failure? | 22:56 |
| anteaya | or are you just in the process of setting them up for the first time? | 22:56 |
| *** sarob has quit IRC | 22:56 | |
| asselin | anteaya, I just started tyring today. I did get my no-op jobs to work, but they stopped running | 22:56 |
| anteaya | so this is a new config for you | 22:57 |
| asselin | yes | 22:57 |
| *** wenlock has quit IRC | 22:57 | |
| anteaya | do you have a config file to paste for me to stare blankly at? | 22:57 |
| * anteaya is good at starely blankly at things | 22:57 | |
| asselin | :) | 22:57 |
| anteaya | :D | 22:57 |
| *** otherwiseguy has quit IRC | 22:57 | |
| *** sarob has joined #openstack-infra | 22:58 | |
| *** jungleboyj has quit IRC | 22:59 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Install lsof on the git mirror nodes https://review.openstack.org/136168 | 22:59 |
| asselin | and the job the does 'run ' is called 'NA' (in the zuul ui) | 22:59 |
| anteaya | how interesting | 22:59 |
| anteaya | clarkb: upvoting that is better than watching you rage quit | 23:00 |
| *** vhoward has left #openstack-infra | 23:03 | |
| clarkb | mtreinish: fungi: that change failed :/ | 23:05 |
| clarkb | one test out of 2085 | 23:05 |
| mtreinish | clarkb: yeah it looks like while things were broken tempest landed a bad test on icehouse | 23:07 |
| *** viglesias has joined #openstack-infra | 23:07 | |
| harlowja | damn | 23:07 |
| *** FallenPegasus has joined #openstack-infra | 23:07 | |
| *** FallenPegasus has quit IRC | 23:07 | |
| harlowja | shall i do some reverts? | 23:08 |
| mtreinish | harlowja: yeah if you don't mind | 23:08 |
| harlowja | k | 23:08 |
| *** emagana has quit IRC | 23:08 | |
| mtreinish | clarkb: are you comfortable ninja merging the taskflow revert? | 23:08 |
| *** emagana has joined #openstack-infra | 23:08 | |
| clarkb | mtreinish: can we promote that one or will it hit icehouse too? | 23:09 |
| *** Guest40820 has quit IRC | 23:09 | |
| mtreinish | clarkb: well it'll run the same icehouse job, I don't know if it'll trigger the bad test or not | 23:10 |
| *** MarkAtwood has quit IRC | 23:10 | |
| mtreinish | I've got a tempest patch up that should fix it (by skipping on icehouse) but I can't land it until this reqs issue gets resolved | 23:10 |
| clarkb | right | 23:11 |
| clarkb | jeblair: ^ | 23:11 |
| * clarkb is trying to think of alternatives | 23:11 | |
| clarkb | remove icehouse tests from taskflow, then run taskflow revert through normally? | 23:11 |
| *** denis_makogon has quit IRC | 23:11 | |
| *** changbl has quit IRC | 23:11 | |
| *** dims has quit IRC | 23:12 | |
| *** emagana has quit IRC | 23:13 | |
| mtreinish | clarkb: yeah that would probably work | 23:13 |
| *** emagana has joined #openstack-infra | 23:13 | |
| *** yamamoto has joined #openstack-infra | 23:14 | |
| mtreinish | it'll just add more delay to unwedge things | 23:14 |
| jeblair | how did tempest land a bad test on icehouse? | 23:16 |
| *** dkehn has quit IRC | 23:17 | |
| mtreinish | jeblair: asymmetrical check vs gate. The stable-compat jobs just run on check (with clean check and a working gate it's normally fine) | 23:17 |
| harlowja | ok, https://review.openstack.org/#/c/136175/ and https://review.openstack.org/#/c/136174/ | 23:17 |
| harlowja | if needed | 23:18 |
| harlowja | think i did it right (haven't had to revert much) | 23:18 |
| jeblair | mtreinish: how did it pass clean check? (and why not have it gate if it's in check since clean-check makes it equivalent?) | 23:18 |
| *** dkehn has joined #openstack-infra | 23:18 | |
| *** viglesias has quit IRC | 23:19 | |
| *** yamamoto has quit IRC | 23:19 | |
| *** amitgandhinz has quit IRC | 23:19 | |
| mtreinish | it looks like the last clean check was >4 days ago so something slipped in(with a freshness check this wouldn't have landed) | 23:20 |
| jeblair | mtreinish: with a gate check this wouldn't have landed | 23:20 |
| mtreinish | its not a 100% fail | 23:20 |
| jeblair | i think that's what we need to fix | 23:20 |
| mtreinish | jeblair: it's not in gate because if we made every tempest job gating things would back up like crazy (we're at >20) | 23:20 |
| jeblair | mtreinish: you already do if they are in check | 23:21 |
| jeblair | there's no reason to have check!=gate with clean check, and having them be unequal produces this result | 23:22 |
| lifeless | testtools 1.5.0 released now FWIW | 23:22 |
| mtreinish | jeblair: the difference is in priority and the dependencies between jobs. If you want to double the number of jobs in gate go ahead and try but I suspect it'll just back things up | 23:22 |
| *** banix has joined #openstack-infra | 23:22 | |
| jeblair | mtreinish: i'd much rather you fix this problem | 23:23 |
| mtreinish | I'm honestly more inclined to just say tempest stops supporting stable branches and rip out all the jobs because we basically have had a week were nothing has landed because of this | 23:23 |
| anteaya | lifeless: yay | 23:24 |
| mtreinish | leave the distros to make the stable branches work because I'm getting tired of it all | 23:24 |
| *** dimtruck is now known as zz_dimtruck | 23:26 | |
| *** sarob has quit IRC | 23:27 | |
| jeblair | mtreinish: what's the thing that's frustrating you? that you can't land changes to tempest because stable branch jobs that only run in check are failing? | 23:27 |
| *** alexpilotti has quit IRC | 23:27 | |
| jeblair | mtreinish: and you can't realistically ignore those jobs because if you do, tempest could land a change that breaks those branches? | 23:28 |
| jeblair | (just trying to understand the full picture) | 23:28 |
| mtreinish | jeblair: no just that the stable branches keep on breaking, and no one else from stable main seems to be able to do anything about it. It has nothing to do with which queue the jobs are running on (the same problem would exist if the gating were symmetrical) | 23:29 |
| clarkb | ok I think I have everything lined up for a nodepool redeploy onto trusty now. its a bit late to do that today particularly with things in wonky states (as it may take time to build new images). I will plan for an early redeploy tomorrow | 23:29 |
| jeblair | clarkb: ack; i will be afk tomorrow morning so won't be able to help | 23:29 |
| mtreinish | jeblair: it's just the extra maint headache of trying to keep all these things working together is tiresome | 23:30 |
| clarkb | jeblair: ok | 23:30 |
| jeblair | mtreinish: would branches in tempest improve things? | 23:30 |
| clarkb | the rollback story is straightforward so I am not super worried about it | 23:30 |
| mtreinish | jeblair: it would for landing things on master, but that just delays the problems. Back when we had branches the same problems used to crop up they just went ignored until release time | 23:31 |
| dougwig | jeblair: if you've got a minute, can I ask a gertty question? it seems to be eating my inline comments, and I'm wondering what I'm doing wrong. | 23:31 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/devstack-gate: Enable nova-api-metadata for multi host nova-network https://review.openstack.org/136179 | 23:31 |
| jeblair | mtreinish: yes, it makes things easier to ignore... | 23:31 |
| mtreinish | it's actually why I'm on stable-maint because I was fixing them back then too... | 23:32 |
| jeblair | dougwig: that's terrible; i don't know why it would do that | 23:32 |
| anteaya | clarkb: good work | 23:32 |
| *** emagana has quit IRC | 23:32 | |
| dougwig | i've got two issues. first is frequent sync failures, and this in the log: | 23:32 |
| dougwig | https://www.irccloud.com/pastebin/LaWk3Y12 | 23:32 |
| dougwig | the second is it posting the review, but the inline comments are missing (the main review feedback claims they are there.) | 23:32 |
| mtreinish | jeblair: and from a quality perspective I really don't want to move back to branches, because things have gotten better since we moved to branchless | 23:32 |
| *** emagana has joined #openstack-infra | 23:32 | |
| dougwig | an example horked review: https://review.openstack.org/#/c/130424/ | 23:33 |
| dougwig | just to verify, after viewing the diff and commenting, it's just 'Esc' to go back, right? there's not a key needed to save the comments or anything? | 23:33 |
| *** doug-fish has left #openstack-infra | 23:34 | |
| *** yolanda has quit IRC | 23:34 | |
| clarkb | ianw: re 131921 I think the setup scripts should set their path if they need to | 23:34 |
| jeblair | dougwig: that's correct. you should see "(# drafts)" or something like that next to the patchset on the change screen when you go back | 23:34 |
| dougwig | i do see that. | 23:35 |
| clarkb | ianw: I am worried that if we try to do that in nodepool we will aloways be playing catchup with the next item to add to PATH | 23:35 |
| clarkb | ianw: is there a good reason not to update PATH in the script? | 23:35 |
| *** banix has quit IRC | 23:36 | |
| *** dmsimard_away is now known as dmsimard | 23:36 | |
| *** pc_m has joined #openstack-infra | 23:37 | |
| jeblair | dougwig: it looks like gerrit thinks it got a comment with that change too (it says "(1 comment)" in the message) | 23:37 |
| harlowja | ok squashed the revert into 1 commit, https://review.openstack.org/#/c/136174/ ; should be fine if thats desired to go in | 23:37 |
| *** emagana has quit IRC | 23:37 | |
| *** emagana has joined #openstack-infra | 23:39 | |
| mtreinish | harlowja: ok cool, thanks | 23:39 |
| harlowja | np | 23:39 |
| mtreinish | we basically need to land that or the reqs change followed by https://review.openstack.org/136154 | 23:40 |
| mtreinish | clarkb, fungi, jeblair: ^^^ I'll defer to you guys on how best to make this happen | 23:41 |
| *** pc_m has quit IRC | 23:41 | |
| ianw | clarkb: i feel like we should abstract the differences between logging in as root and as a user | 23:41 |
| clarkb | ianw: I don't because the distros basically screw us on that front | 23:42 |
| clarkb | and its easy to deal outside of nodepool | 23:42 |
| clarkb | whereas changes in nodepool require service restarts | 23:42 |
| ianw | clarkb: right, so by setting a sane PATH you don't have to worry about it. what's the point of basically making everyone remember to set it, or debug when they forget | 23:43 |
| clarkb | ianw: because trying to accomodate every distro ever in nodepool is already ugly | 23:44 |
| *** dangers is now known as dangers_away | 23:44 | |
| clarkb | we have to try a half dozen usernames | 23:44 |
| clarkb | and now we may have to set a ton of stuff in PATH that is a super set of the proper subsets that are required on specific nodes | 23:44 |
| *** ryanpetrello has joined #openstack-infra | 23:45 | |
| mtreinish | jeblair: oh the other thing I just remembered even making things symmetric wouldn't have be a guarantee that this wouldn't have slipped through. I don't think it's a 100% failure just a higher % than normally slips through. | 23:45 |
| mtreinish | it's hard to say with so few data points | 23:46 |
| jeblair | mtreinish: yeah | 23:46 |
| clarkb | ianw: I just think this stuff shouldn't live in nodepool propre as much as possible | 23:46 |
| clarkb | because I dislike restarting services to pull in what is essentially config | 23:47 |
| ianw | clarkb: i think the fact that we're logging in without an interactive session, thus falling back to the compiled in path defaults of ssh, means it's reasonable to say "the PATH you get when running a nodepool script is ..." | 23:47 |
| jeblair | dougwig: ok i looked at your change in gertty, and the comment was on the old side of the commit message; i'm not sure gerrit displays that normally | 23:47 |
| jeblair | dougwig: yeah, i think that's a "feature" of the gerrit api that only gertty lets you access | 23:48 |
| dougwig | the old side of the commit message? i didn't parse that. | 23:48 |
| jeblair | dougwig: so on other changes, is it possible that you also left your comments there? if not, do you have an example of another change | 23:48 |
| jeblair | dougwig: in gerrit you can comment on the old or new versions of each file | 23:48 |
| jeblair | dougwig: (if you use side-by-side diffs, there's the red (old) side on the left and the green (new) side on the right) | 23:49 |
| dougwig | ahh, ok. so, i scrolled down to diff, hit enter, scrolled into a real file, hit enter, typed some stuff, hit escape, then review. which part of that sequence do i need to change? | 23:49 |
| clarkb | ianw: I don't feel so strngly that we shouldn't take the change. just trying to work through if we should | 23:49 |
| jeblair | dougwig: hit 'tab' to change between leaving your comment on the old file or new file | 23:50 |
| dougwig | ahh, got it. i prefer unified anyway, so let me try that. | 23:50 |
| clarkb | ianw: we should however put local before /bin and /usr/bin imo | 23:50 |
| clarkb | ianw: same for sbin. typiclly you expect the local paths to override non local paths | 23:50 |
| jeblair | dougwig: i think in cases where there is no old file, the gerrit webui will prevent you from leaving a comment on the old side. gertty probably should do that too | 23:50 |
| *** baoli has quit IRC | 23:50 | |
| *** reed has quit IRC | 23:51 | |
| ianw | clarkb: that is fair, in fact it makes more sense for it to be --with-superuser-path | 23:51 |
| *** ryanpetrello has quit IRC | 23:52 | |
| jeblair | dougwig: if you are using unified with gertty, it's not 'tab' to change between them, but rather select a line from the new file (or the new filename itself) | 23:52 |
| *** viglesias has joined #openstack-infra | 23:53 | |
| *** gondoi is now known as zz_gondoi | 23:53 | |
| dougwig | in unified, are unchanged lines new or old? | 23:53 |
| *** mriedem has quit IRC | 23:53 | |
| clarkb | jeblair: for untangling tempest/taskflow what do you think of removing the icehouse test from taskflow, then approving+promoting/enqueuing the taskflow revert? | 23:53 |
| jeblair | clarkb: removing the test == zuul config change? | 23:54 |
| clarkb | jeblair: ya | 23:54 |
| *** baoli has joined #openstack-infra | 23:54 | |
| clarkb | (we just added them today I think... before we were actually ready to test on that) | 23:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Set useful PATH before starting scripts https://review.openstack.org/131921 | 23:54 |
| jeblair | clarkb: that sounds very sensible in that case | 23:54 |
| ianw | clarkb: ^ updated, with maybe a clearer justification | 23:54 |
| clarkb | mtreinish: harlowja: one of you want to propose the infra side revert? | 23:55 |
| mtreinish | clarkb: one sec I'll push up a change to remove stable compat jobs from taskflow | 23:55 |
| *** ryanpetrello has joined #openstack-infra | 23:55 | |
| clarkb | ianw: +2 | 23:55 |
| *** bswartz has joined #openstack-infra | 23:56 | |
| *** sarob has joined #openstack-infra | 23:57 | |
| jeblair | dougwig: i think they should be new | 23:58 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Temporarily to remove stable compat jobs on taskflow https://review.openstack.org/136185 | 23:59 |
| ianw | fungi: https://review.openstack.org/#/c/131921/ <- PATH nodepool thing, if you get a chance. | 23:59 |
| mtreinish | clarkb: ^^^ | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!