| *** wenlock has joined #openstack-infra | 00:00 | |
| asselin | i.e fixed the conflict | 00:00 |
|---|---|---|
| clarkb | I wonder if we can stop using that db connection entirely in manage projects | 00:00 |
| clarkb | oh no because races nevermind | 00:00 |
| clarkb | anyways +2 | 00:00 |
| *** banix has quit IRC | 00:00 | |
| sweston | asselin: ok, glad you found it :-) | 00:00 |
| fungi | clarkb: we could switch it to sqlalchemy at some point which does a ping(True) behind the scenes if we got really bored | 00:01 |
| *** doug-fish has quit IRC | 00:01 | |
| *** doug-fish has joined #openstack-infra | 00:02 | |
| *** Guest8043 has quit IRC | 00:02 | |
| *** shashankhegde has joined #openstack-infra | 00:04 | |
| *** teran has joined #openstack-infra | 00:04 | |
| clarkb | fungi: ya it started as a hack and probably should consider less hacky methods | 00:07 |
| *** teran_ has quit IRC | 00:07 | |
| *** achanda has quit IRC | 00:08 | |
| clarkb | but we haven't heard of any psycopg2 ping timeouts so it may be a non issue there | 00:09 |
| clarkb | would potentially explain why they don't have a similar method too | 00:09 |
| fungi | perhaps | 00:12 |
| clarkb | AJaeger: when I wake up tomorrow I would like to try and get python26 changes sorted before I start the long weekend | 00:13 |
| clarkb | AJaeger: I will do my best to start the day early :) | 00:13 |
| clarkb | that way you are still around :) | 00:13 |
| *** JayJ has quit IRC | 00:14 | |
| asselin | sweston, interesting, I had to do this again, so something must be installing it and causing the conflict: sudo apt-get remove python-jenkins | 00:14 |
| *** wenlock has quit IRC | 00:16 | |
| fungi | clarkb: i'll be around to help with those for the first half of my day if needed | 00:16 |
| *** emagana has joined #openstack-infra | 00:16 | |
| clarkb | fungi: mostly I think we just need to rebase and rereview so that stuff is ready for end of weekend | 00:16 |
| clarkb | since AJaeger hsa changes written iirc | 00:16 |
| fungi | yep | 00:17 |
| fungi | and sent the reminder to the ml too | 00:17 |
| sweston | asselin: i'll bet you a million dollars it's puppet ;-) | 00:18 |
| *** ChuckC_ has quit IRC | 00:18 | |
| JayF | I have a change that's been up since Monday that has never had jenkins run on it, and although it's +A'd, doesn't appear to be merging | 00:19 |
| JayF | https://review.openstack.org/#/c/136867/ is the one I'm talking about | 00:19 |
| fungi | JayF: leave a recheck comment on it | 00:20 |
| clarkb | Monday the day of doom | 00:20 |
| *** Sukhdev has joined #openstack-infra | 00:20 | |
| sweston | yup | 00:20 |
| fungi | JayF: it was likely uploaded while zuul was offline so it missed the patchset uploaded event | 00:20 |
| sweston | asselin: it's in the job_builder.pp file | 00:20 |
| JayF | Cool. I avoid spamming a recheck in unless I know it's the real solution to the problem :) | 00:20 |
| fungi | JayF: recheck will cause it to run through check and, if jobs succeed, will automatically enqueue it into the gate since it's already approved | 00:20 |
| asselin | sweston, notice: /Stage[main]/Jenkins::Job_builder/Package[python-jenkins]/ensure: ensure changed 'purged' to 'present' | 00:21 |
| hashar | hello there! | 00:22 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Remove logic from puppet.conf https://review.openstack.org/137246 | 00:22 |
| clarkb | hashar: hello | 00:22 |
| hashar | I think i remember a spec / doc that proposed to run tests with the latest version of dependencies provided by Pypi , or at least test out with some libs different than the global requirements. Does it sounds familiar to anyone? | 00:22 |
| *** emagana has quit IRC | 00:23 | |
| *** HACKING-FACEBOOK has quit IRC | 00:23 | |
| sweston | asselin: should be here: /opt/config/environment/modules/jenkins/manifests/job_builder.pp | 00:23 |
| clarkb | hashar: I seem to remeber the suggsetion sdague made about ways to get around upstreams beraking us | 00:23 |
| *** emagana has joined #openstack-infra | 00:23 | |
| clarkb | hashar: and locking down global requirements then running a job like you describe was one of the ideas thrown out there for doing that | 00:23 |
| asselin | sweston, don't have that | 00:24 |
| clarkb | hashar: but I don't think it got as far as a spec | 00:24 |
| hashar | clarkb: agh I have been searching for an half an hour and was wondering whether I dreamed about it :] | 00:24 |
| JayF | fungi: how long will it take to queue the chceks? I'm used to seeing a message pretty quickly after posting a recheck comment | 00:24 |
| sweston | asselin: you must have it somewhere | 00:25 |
| *** ChuckC_ has joined #openstack-infra | 00:25 | |
| asselin | sweston, looking | 00:25 |
| clarkb | hashar: if ou dig through the openstack-dev mail list you should find the thread | 00:25 |
| clarkb | hashar: it would be from the last couple weeks | 00:25 |
| sweston | asselin: puppet is hella annoying, but he's not magic | 00:25 |
| *** sputnik13 has quit IRC | 00:26 | |
| *** achanda has joined #openstack-infra | 00:26 | |
| hashar | clarkb: thanks, will dig in and if all fails will ask sean :] | 00:26 |
| *** sputnik13 has joined #openstack-infra | 00:26 | |
| fungi | JayF: the zuul status page http://status.openstack.org/zuul/ shows it running in the check pipeline currently with 1 hour 18 minutes to go before it finishes and possibly enqueues into the gate. at that point zuul will update the change with comments about the check results and that it's being chucked into the gate pipeline (if all the voting ones pass) | 00:26 |
| JayF | fungi: did the "Check jobs starting..." gerrit messages get nuked then? | 00:27 |
| *** emagana has quit IRC | 00:28 | |
| asselin | sweston, but I see it in the manifest: https://github.com/openstack-infra/puppet-jenkins/blob/master/manifests/job_builder.pp#L21 | 00:28 |
| fungi | JayF: yes, it was necessary noise we endured while running gerrit 2.4 but once we upgraded to 2.8 we were able to stop zuul from needing to leave that comment | 00:28 |
| JayF | fungi: aha, that's where my confusion was coming from then, I was looking for that message. I guess this always just happens in the background -- that I don't usually have to think about it is a testament to you folk :P | 00:29 |
| fungi | JayF: yeah the theory was that you should be able to just assume it happens automatically. except you know when we have a catastrophe and then we sort of undermine that expectation. hopefully that will be less and less frequent as time goes on | 00:30 |
| *** dims_ has quit IRC | 00:30 | |
| sweston | asselin: find / -name job_builder.pp -print | 00:31 |
| sweston | asselin: does that do anything for you | 00:31 |
| asselin | sweston, yes, but when I find it, then what? | 00:32 |
| *** HACKING-FACEBOOK has joined #openstack-infra | 00:32 | |
| *** doug-fish has left #openstack-infra | 00:32 | |
| sweston | there will be a line in there that tells puppet to install the package, | 00:33 |
| asselin | sweston, yes, but why does it cause the conflict? <asselin> sweston, but I see it in the manifest: https://github.com/openstack-infra/puppet-jenkins/blob/master/manifests/job_builder.pp#L21 | 00:34 |
| asselin | sweston, so the manifest on master is out of sync with the JJB on master? | 00:35 |
| sweston | asselin: If I remember correctly, it installs a dependency that overrides a library that jjb needs for pyaml parsing | 00:35 |
| *** emagana has joined #openstack-infra | 00:35 | |
| *** emagana has quit IRC | 00:35 | |
| *** emagana has joined #openstack-infra | 00:35 | |
| sweston | asselin: I might just remove it, install jenkins from source | 00:35 |
| *** achanda has quit IRC | 00:37 | |
| asselin | sweston, but it is installed from the source | 00:38 |
| sweston | asselin: the os packages often lag way being the current development pip bundles, it has always been a hot spot for issues | 00:38 |
| asselin | sweston, no, it's installed here from source: https://github.com/openstack-infra/puppet-jenkins/blob/master/manifests/job_builder.pp#L26 | 00:38 |
| asselin | sweston, I think it's a real bug | 00:39 |
| sweston | asselin: jjb is, yes, but what about Jenkins? | 00:39 |
| *** doug-fish1 has joined #openstack-infra | 00:39 | |
| asselin | sweston, jenkins is fine :) | 00:39 |
| *** julim has joined #openstack-infra | 00:40 | |
| asselin | sweston, jenkins is java | 00:40 |
| sweston | asselin: didn't you say you fixed the jjb issue when you removed the python-jenkins package? | 00:41 |
| asselin | sweston, yes | 00:42 |
| *** HACKING-FACEBOOK has quit IRC | 00:42 | |
| *** doug-fish2 has joined #openstack-infra | 00:42 | |
| *** doug-fish2 has quit IRC | 00:43 | |
| *** doug-fish1 has quit IRC | 00:44 | |
| sweston | asselin: then I must not be following you, seems like the logical solution would be to remove it from the puppet manifest that is installing it | 00:44 |
| *** yamamoto has joined #openstack-infra | 00:44 | |
| asselin | sweston, yes, I am trying that | 00:44 |
| sweston | asselin: ok, then I am following you, carry on :-) | 00:45 |
| *** koolhead17 has joined #openstack-infra | 00:45 | |
| asselin | sweston, so when I delete it, the problem is fixed. | 00:46 |
| *** erikwilson has joined #openstack-infra | 00:46 | |
| asselin | sweston, I'll propose the solution, but I don't think that's correct still...because there's a comment saying other packages may need it as well... | 00:46 |
| *** erikwilson has left #openstack-infra | 00:47 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova libvirt bug 1372670 https://review.openstack.org/137248 | 00:47 |
| *** ZZelle_ has quit IRC | 00:47 | |
| uvirtbot | Launchpad bug 1372670 in nova "libvirtError: operation failed: cannot read cputime for domain" [High,Confirmed] https://launchpad.net/bugs/1372670 | 00:47 |
| sweston | asselin: let me dig through some logs, before you propose it | 00:48 |
| asselin | sweston, ok | 00:48 |
| *** Ryan_Lane has joined #openstack-infra | 00:49 | |
| *** davideagnello has quit IRC | 00:50 | |
| *** alexpilotti has quit IRC | 00:51 | |
| *** david-lyle is now known as david-lyle_afk | 00:51 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-jenkins: Update to direct storyboard link https://review.openstack.org/137249 | 00:51 |
| *** NobodyCam has quit IRC | 00:56 | |
| *** mpavlase has quit IRC | 00:56 | |
| *** baoli has joined #openstack-infra | 00:56 | |
| *** achanda has joined #openstack-infra | 00:57 | |
| *** emagana has quit IRC | 00:57 | |
| *** NobodyCam has joined #openstack-infra | 00:58 | |
| *** juice has quit IRC | 00:58 | |
| *** shashankhegde has quit IRC | 00:58 | |
| *** emagana has joined #openstack-infra | 00:58 | |
| *** yamahata has joined #openstack-infra | 01:01 | |
| *** juice has joined #openstack-infra | 01:01 | |
| *** emagana has quit IRC | 01:02 | |
| asselin | sweston, going home. I'll just submit it and abandon it later. | 01:03 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-jenkins: Puppet-Jenkins not compatible with JJB https://review.openstack.org/137250 | 01:04 |
| sweston | asselin: ok, I am not finding it in my logs, too long ago | 01:04 |
| asselin | should really be a bug | 01:05 |
| *** stevemar has quit IRC | 01:05 | |
| *** davideagnello has joined #openstack-infra | 01:05 | |
| asselin | but I gotta go. good night | 01:05 |
| sweston | asselin: we might be able to isolate it in the future. i'll keep my eye out for dependencies. have a good night | 01:06 |
| *** gokrokve has quit IRC | 01:06 | |
| zul | mordred: what did i do? | 01:06 |
| *** shashankhegde has joined #openstack-infra | 01:06 | |
| *** hashar has quit IRC | 01:07 | |
| *** salv-orlando has quit IRC | 01:07 | |
| *** teran_ has joined #openstack-infra | 01:08 | |
| *** yamamoto has quit IRC | 01:08 | |
| oomichi | hi, some patches cannot be kicked for the gate job even if getting approvals. e.g) https://review.openstack.org/#/c/118934 , https://review.openstack.org/#/c/68560/ | 01:09 |
| jeblair | mordred, clarkb, fungi: when i used an instance variable in the packet read function, i made it 25x slower. | 01:09 |
| jeblair | caching it in a local var makes it better again | 01:09 |
| fungi | jeblair: that's a substantial magnification | 01:09 |
| clarkb | jeblair: wow | 01:10 |
| clarkb | oomichi: looking | 01:10 |
| *** bo_sh has left #openstack-infra | 01:10 | |
| *** amotoki has joined #openstack-infra | 01:10 | |
| *** shashankhegde has quit IRC | 01:10 | |
| oomichi | clarkb: thanks :) | 01:10 |
| *** teran has quit IRC | 01:11 | |
| *** mmaglana has quit IRC | 01:14 | |
| clarkb | jeblair: fungi: looks like https://review.openstack.org/#/c/68560/ did not go from check to gate properly | 01:18 |
| clarkb | oomichi: I think if you recheck it again it will properl go to the gate. not currently sure hwy it didn't automatically | 01:19 |
| *** baoli has quit IRC | 01:19 | |
| fungi | clarkb: looks like it was probably in the gate pipeline when we had zuul going up and down | 01:20 |
| oomichi | clarkb: thanks, rechecks are done on both. | 01:20 |
| fungi | jeblair: any chance that was left out of your list of changes to reenqueue in the gate? | 01:21 |
| jeblair | fungi: i think that was from just a few hours ago | 01:22 |
| fungi | d'oh, you're right | 01:22 |
| fungi | i got my dates screwed up | 01:22 |
| *** mmaglana has joined #openstack-infra | 01:22 | |
| fungi | i'm looking in the zuul log now | 01:22 |
| *** achanda has quit IRC | 01:22 | |
| clarkb | 2014-11-26 00:39:43,118 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added openstack/nova master 68560,19> I think that is where it should figure out that it can gate on it | 01:24 |
| fungi | no, i was right on the dates... mikal approved patchset 19 a little over 24 hours ago | 01:26 |
| clarkb | fungi: but there was a reverify this afternoon | 01:27 |
| clarkb | that came back +1 but did not enqueue to the gate | 01:27 |
| jeblair | did we remove reverify? | 01:27 |
| fungi | zuul would have been offline at the time. then it was rechecked and passed at 04:15 | 01:27 |
| clarkb | jeblair: no it reran the jobs properly | 01:27 |
| fungi | but never enqueued when the check results came back at that point | 01:27 |
| clarkb | the layout.yaml looks correct | 01:29 |
| clarkb | requires jenkins to +1 or +2 and triggers on comment added +1 by jenkins | 01:29 |
| clarkb | or and require workflow +1 | 01:29 |
| *** dimsum__ has joined #openstack-infra | 01:30 | |
| clarkb | oh I know | 01:30 |
| *** baoli has joined #openstack-infra | 01:30 | |
| clarkb | maybe | 01:30 |
| clarkb | if you look at the jenkins comments only the really early one says +1 | 01:30 |
| *** mmaglana has quit IRC | 01:30 | |
| clarkb | I wonder if the events neglect to say +1 in the event if it isn't a state change | 01:30 |
| clarkb | because gerrit | 01:30 |
| fungi | d'oh, right | 01:30 |
| clarkb | and so we get an event that says "looks good" but without the +1 and zuul does nothing | 01:31 |
| *** achanda has joined #openstack-infra | 01:31 | |
| clarkb | and its back in check not gate | 01:31 |
| clarkb | so I think what we need is to approve that change again | 01:31 |
| clarkb | and maybe to cry about gerrit | 01:31 |
| *** teran has joined #openstack-infra | 01:31 | |
| *** MaxV_ has quit IRC | 01:31 | |
| jeblair | the latest recheck should do it, right? | 01:31 |
| clarkb | jeblair: no | 01:31 |
| clarkb | because there is no state change | 01:32 |
| jeblair | er, why isn't that a trigger for gate? | 01:32 |
| clarkb | because gerrit only emits vote values if they change | 01:32 |
| *** r-daneel has quit IRC | 01:32 | |
| clarkb | yay gerrit | 01:33 |
| jeblair | oh, because we have no reverify now | 01:33 |
| jeblair | i thought we had recheck and reverify as triggers in both pipelines | 01:33 |
| clarkb | oh ya we don't put those in gate they are in check because assumption is you reverify or recheck, get the +1 then that enqueues | 01:33 |
| clarkb | no just check because ^ but that appears to be a bad assumption | 01:33 |
| *** otter768 has joined #openstack-infra | 01:34 | |
| *** teran_ has quit IRC | 01:34 | |
| *** achanda has quit IRC | 01:35 | |
| *** dimsum__ has quit IRC | 01:36 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Make agent_ssh check job voting for IPA https://review.openstack.org/134436 | 01:37 |
| *** dimsum__ has joined #openstack-infra | 01:37 | |
| *** guest764 has quit IRC | 01:37 | |
| clarkb | git log -p is failing me | 01:37 |
| jeblair | yeah i'm having trouble too | 01:37 |
| clarkb | trying to find the point where we removed reverify | 01:37 |
| jeblair | i think something broke when the file was moved | 01:38 |
| JayF | I changed the scope of https://review.openstack.org/#/c/134436/5 to make the tempest job only vote for IPA, not Ironic, if others want to take a look I'd appreciate it. | 01:38 |
| *** otter768 has quit IRC | 01:38 | |
| clarkb | JayF: ya | 01:39 |
| clarkb | er jeblair yup | 01:39 |
| * JayF messing up your j<tab> sorted by recent nicks | 01:40 | |
| *** hdd has joined #openstack-infra | 01:40 | |
| *** guest764 has joined #openstack-infra | 01:40 | |
| clarkb | JayF: indeed | 01:40 |
| *** dimsum__ has quit IRC | 01:41 | |
| adam_g | fungi, hmph so raising the oslo.vmware dependency to a higher line in requirements.txt doesn't seem to help here :| http://logs.openstack.org/38/136538/4/check/check-grenade-dsvm/951b9b3/logs/grenade.sh.txt.gz#_2014-11-26_01_13_37_040 | 01:41 |
| jeblair | clarkb: https://review.openstack.org/#/c/114626/ | 01:42 |
| clarkb | jeblair: that already had it removed from gate though | 01:42 |
| clarkb | so we need to look before that | 01:43 |
| jeblair | oh right | 01:43 |
| clarkb | https://review.openstack.org/#/q/project:openstack-infra/system-config+message:recheck+status:merged,p,002ed22a0001b259 is what I am looking through | 01:43 |
| clarkb | its between that and https://review.openstack.org/#/c/111098/ somewhere | 01:43 |
| clarkb | oh wait no its even earlier | 01:44 |
| *** patrickeast has quit IRC | 01:44 | |
| jeblair | clarkb, fungi: after addressing the instance variable issue, the client performance is back up to par, but i'm a bit unhappy with the server performance; i may give this another pass and make it simpler | 01:45 |
| clarkb | before https://review.openstack.org/#/c/92271/ even | 01:46 |
| clarkb | jeblair: hack away :) very interested to see the final result though | 01:46 |
| jeblair | and my best guess as to what was going on is very bad performance; i can't reproduce the same numbers we saw in production locally, but i can get some really bad ones :) | 01:46 |
| *** shashankhegde has joined #openstack-infra | 01:48 | |
| clarkb | jeblair: https://review.openstack.org/#/c/73418/3 I think I found it | 01:48 |
| fungi | that does appear to be where we dropped it, yes | 01:50 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 01:52 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 01:52 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 01:52 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add ssh key to root user https://review.openstack.org/137176 | 01:52 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Enable live block migration https://review.openstack.org/135768 | 01:52 |
| *** wuhg has joined #openstack-infra | 01:52 | |
| *** dimsum__ has joined #openstack-infra | 01:53 | |
| *** marun has quit IRC | 01:55 | |
| *** mudassirlatif has quit IRC | 01:55 | |
| *** fandi has quit IRC | 01:55 | |
| *** mmaglana has joined #openstack-infra | 01:56 | |
| *** mmaglana has quit IRC | 01:58 | |
| *** ayoung-dad-mode is now known as ayoung | 02:01 | |
| adam_g | fungi, you mentioned some concern about capping oslo libs in stable, care to expand? bumping oslo.vmware higher in the req list didn't help at all and i'm thinking we may need https://review.openstack.org/#/c/136879/ | 02:02 |
| fungi | adam_g: because devstack doesn't (yet) know how to do that and ends up installing from the tip of master anyway. or did until recently but maybe sdague's changes make that possible now? | 02:04 |
| adam_g | fungi, the devstack changes to install from released versions got backported, which is kinda how we've ended up here | 02:04 |
| *** emagana has joined #openstack-infra | 02:06 | |
| fungi | adam_g: which dependency is it that was not getting upgraded even after reordering oslo.vmware to appear earlier in the requirements list? | 02:06 |
| adam_g | fungi, eventlet | 02:06 |
| adam_g | fungi, its still ending up with the dependency satisfied by the distro package (0.13.0) | 02:06 |
| adam_g | i cant seem to reproduce this locally, though | 02:07 |
| *** emagana_ has joined #openstack-infra | 02:07 | |
| *** yaguang has joined #openstack-infra | 02:08 | |
| *** dannywilson has quit IRC | 02:08 | |
| fungi | adam_g: so in this case it looks like keystone is being installed by devstack as the first thing to request eventlet and asks for eventlet>=0.13.0 | 02:08 |
| *** yaguang_ has joined #openstack-infra | 02:09 | |
| adam_g | right | 02:09 |
| *** dannywilson has joined #openstack-infra | 02:09 | |
| fungi | swift does the same... | 02:09 |
| fungi | then glance | 02:10 |
| fungi | cinder, nova... | 02:10 |
| *** emagana has quit IRC | 02:10 | |
| *** Ark has joined #openstack-infra | 02:11 | |
| *** Ark is now known as Guest34138 | 02:12 | |
| adam_g | during the glance install, prior to the eventlet pass, oslo.vmware 0.7.0 gets downloaded but its eventlet requirement seems to be ignored | 02:12 |
| adam_g | i believe thats the first occurence of oslo.vmware being hit | 02:12 |
| *** yaguang has quit IRC | 02:12 | |
| fungi | yeah, looking at the bit where it's installing glance, it installs oslo.vmware 0.7.0 but does not go directly to installing its requirements | 02:13 |
| *** dannywilson has quit IRC | 02:13 | |
| adam_g | yeah--wth? | 02:13 |
| adam_g | i've spent some time reproducing what devstacks doing with different variations of a requirements.txt and can't reproduce it | 02:14 |
| fungi | aha, it _is_ breadth-first not depth-first | 02:14 |
| fungi | look down toward the end and you'll see it starting to install the transitive dependencies if needed | 02:14 |
| *** nosnos has joined #openstack-infra | 02:14 | |
| fungi | so the "list it earlier" trick only works for competing transitive dependencies | 02:15 |
| fungi | not for a transitive dependency which needs to meet more specific criteria than a directly listed version of the same requirement | 02:15 |
| adam_g | fungi, hmmm | 02:15 |
| fungi | blah blah (from oslo.vmware>=0.2->glance==2014.1.4.dev6.gef77c79) | 02:16 |
| fungi | doesn't mention eventlet, presumably because it already checked eventlet earlier where it was listed as a requirement of glance | 02:16 |
| fungi | adam_g: so yeah, you can float an oslo.vmware version cap in stable global-requirements.txt and see if it passes tests i guess | 02:17 |
| adam_g | https://review.openstack.org/#/c/136879/ is in check queue now | 02:18 |
| *** bo_sh1 has joined #openstack-infra | 02:18 | |
| fungi | since pip is satisfying reqs breadth-first for projects, there's really no way with reordering you're going to be able to get a transitive req to trump a direct req i don't think | 02:18 |
| *** banix has joined #openstack-infra | 02:18 | |
| adam_g | unfortunately, the grenade job that would let a fix for this be tested pre-merge is busted on a devstack bug introduced in that backport i mentioned, and can't be fixed till this is resolved :| | 02:19 |
| fungi | quite the pity | 02:19 |
| *** bo_sh1 has left #openstack-infra | 02:20 | |
| *** Sukhdev has quit IRC | 02:20 | |
| *** Sukhdev has joined #openstack-infra | 02:21 | |
| *** yamamoto has joined #openstack-infra | 02:22 | |
| *** asettle is now known as asettle-optometr | 02:25 | |
| *** asettle-optometr is now known as asettle-glasses | 02:25 | |
| *** NobodyCam has quit IRC | 02:26 | |
| *** NobodyCam has joined #openstack-infra | 02:27 | |
| *** MarkAtwood has quit IRC | 02:28 | |
| *** sdake_ has joined #openstack-infra | 02:31 | |
| *** sdake_ has quit IRC | 02:31 | |
| *** sdake_ has joined #openstack-infra | 02:31 | |
| *** MaxV has joined #openstack-infra | 02:32 | |
| *** vipul has quit IRC | 02:33 | |
| *** jerryz has joined #openstack-infra | 02:35 | |
| *** dmsimard_away is now known as dmsimard | 02:37 | |
| *** MaxV has quit IRC | 02:37 | |
| *** vipul has joined #openstack-infra | 02:37 | |
| *** MarkAtwood has joined #openstack-infra | 02:38 | |
| *** koolhead_ has joined #openstack-infra | 02:45 | |
| *** wenlock has joined #openstack-infra | 02:45 | |
| *** koolhead17 has quit IRC | 02:48 | |
| *** shashankhegde has quit IRC | 02:49 | |
| *** sputnik13 has quit IRC | 02:51 | |
| *** gokrokve has joined #openstack-infra | 02:52 | |
| *** gokrokve has quit IRC | 02:52 | |
| *** sputnik13 has joined #openstack-infra | 02:52 | |
| *** dimsum__ has quit IRC | 02:52 | |
| *** gokrokve has joined #openstack-infra | 02:52 | |
| *** gokrokve has quit IRC | 02:53 | |
| *** gokrokve has joined #openstack-infra | 02:53 | |
| *** banix has quit IRC | 02:54 | |
| *** gokrokve has quit IRC | 02:54 | |
| *** gokrokve has joined #openstack-infra | 02:55 | |
| *** guest764 has quit IRC | 02:58 | |
| *** sputnik13 has quit IRC | 02:58 | |
| *** gokrokve has quit IRC | 02:59 | |
| *** gokrokve has joined #openstack-infra | 02:59 | |
| *** gokrokve has quit IRC | 02:59 | |
| *** MarkAtwood has quit IRC | 02:59 | |
| *** gokrokve has joined #openstack-infra | 02:59 | |
| *** gokrokve has quit IRC | 03:00 | |
| *** gokrokve has joined #openstack-infra | 03:00 | |
| *** nosnos has quit IRC | 03:01 | |
| *** fifieldt has joined #openstack-infra | 03:01 | |
| *** Sukhdev has quit IRC | 03:01 | |
| adam_g | fungi, ugh yeah--https://github.com/pypa/pip/blob/master/pip/req.py#L1266 ... transitive dependencies are skipped entirely if one by the same name exists in the top-level or has already been added by a previously processed requirement | 03:02 |
| *** Ryan_Lane has quit IRC | 03:03 | |
| *** mmaglana has joined #openstack-infra | 03:03 | |
| *** hdd has quit IRC | 03:04 | |
| *** gokrokve has quit IRC | 03:05 | |
| *** gokrokve has joined #openstack-infra | 03:05 | |
| *** banix has joined #openstack-infra | 03:06 | |
| *** gokrokve has quit IRC | 03:06 | |
| *** gokrokve has joined #openstack-infra | 03:07 | |
| *** gokrokve has quit IRC | 03:07 | |
| *** gokrokve has joined #openstack-infra | 03:08 | |
| *** gokrokve has quit IRC | 03:08 | |
| *** gokrokve has joined #openstack-infra | 03:12 | |
| *** gokrokve has quit IRC | 03:17 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 03:17 |
| *** ryanpetrello has joined #openstack-infra | 03:18 | |
| *** dmsimard is now known as dmsimard_away | 03:21 | |
| *** harlowja is now known as harlowja_away | 03:29 | |
| *** wenlock has quit IRC | 03:30 | |
| *** hdd has joined #openstack-infra | 03:35 | |
| *** otter768 has joined #openstack-infra | 03:35 | |
| *** jerryz1 has joined #openstack-infra | 03:35 | |
| *** jerryz has quit IRC | 03:37 | |
| *** otter768 has quit IRC | 03:40 | |
| *** ryanpetrello has quit IRC | 03:41 | |
| *** sarob has joined #openstack-infra | 03:45 | |
| *** amcrn has quit IRC | 03:47 | |
| *** emagana_ has quit IRC | 03:48 | |
| *** emagana has joined #openstack-infra | 03:48 | |
| *** emagana has quit IRC | 03:49 | |
| *** sarob has quit IRC | 03:49 | |
| *** emagana has joined #openstack-infra | 03:50 | |
| *** unicell has joined #openstack-infra | 03:51 | |
| *** dimsum__ has joined #openstack-infra | 03:52 | |
| *** ivar-lazzaro has quit IRC | 03:53 | |
| *** dimsum__ has quit IRC | 03:57 | |
| *** ryanpetrello has joined #openstack-infra | 04:07 | |
| *** achanda has joined #openstack-infra | 04:12 | |
| *** nosnos has joined #openstack-infra | 04:16 | |
| *** MaxV has joined #openstack-infra | 04:22 | |
| *** ryanpetrello has quit IRC | 04:23 | |
| *** bhunter71 has quit IRC | 04:24 | |
| *** MaxV has quit IRC | 04:25 | |
| *** Sukhdev has joined #openstack-infra | 04:26 | |
| *** AlexF has joined #openstack-infra | 04:26 | |
| *** emagana has quit IRC | 04:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-httpd: Update Rakefile and module info https://review.openstack.org/136959 | 04:29 |
| *** achanda has quit IRC | 04:30 | |
| *** dannywilson has joined #openstack-infra | 04:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add puppet-httpd to modules.env https://review.openstack.org/136962 | 04:31 |
| *** banix has quit IRC | 04:34 | |
| *** weshay has quit IRC | 04:37 | |
| *** sabeen1 has joined #openstack-infra | 04:46 | |
| *** pcrews has quit IRC | 04:48 | |
| *** shashankhegde has joined #openstack-infra | 04:49 | |
| *** ryanpetrello has joined #openstack-infra | 04:49 | |
| *** zz_dimtruck is now known as dimtruck | 04:50 | |
| *** yongli has joined #openstack-infra | 04:51 | |
| *** yfried has joined #openstack-infra | 04:51 | |
| *** tnovacik has joined #openstack-infra | 04:54 | |
| *** ayoung is now known as ayoung_ZZzz_zzZZ | 04:59 | |
| *** AlexF has quit IRC | 05:04 | |
| *** harlowja_at_home has joined #openstack-infra | 05:08 | |
| *** hdd has quit IRC | 05:09 | |
| *** garyh has quit IRC | 05:09 | |
| *** rushiagr_away is now known as rushiagr | 05:10 | |
| *** harlowja_at_home has quit IRC | 05:12 | |
| *** stevemar has joined #openstack-infra | 05:13 | |
| *** asettle-glasses is now known as asettle | 05:20 | |
| *** AlexF has joined #openstack-infra | 05:21 | |
| *** emagana has joined #openstack-infra | 05:29 | |
| *** BharatK has joined #openstack-infra | 05:30 | |
| *** talluri has joined #openstack-infra | 05:30 | |
| *** jerryz1 has quit IRC | 05:32 | |
| *** yfried has quit IRC | 05:34 | |
| *** otter768 has joined #openstack-infra | 05:36 | |
| *** stevemar has quit IRC | 05:36 | |
| *** chandankumar has joined #openstack-infra | 05:37 | |
| *** Longgeek has joined #openstack-infra | 05:38 | |
| *** ujuc has joined #openstack-infra | 05:39 | |
| *** otter768 has quit IRC | 05:40 | |
| *** garyh has joined #openstack-infra | 05:41 | |
| *** michchap has quit IRC | 05:41 | |
| *** Longgeek has quit IRC | 05:44 | |
| *** Guest34138 has quit IRC | 05:47 | |
| *** dannywilson has quit IRC | 05:50 | |
| *** gokrokve has joined #openstack-infra | 05:50 | |
| *** gokrokve has quit IRC | 05:52 | |
| *** dannywilson has joined #openstack-infra | 05:58 | |
| *** rushiagr is now known as rushiagr_away | 05:59 | |
| *** tnovacik has quit IRC | 06:00 | |
| *** stevemar has joined #openstack-infra | 06:01 | |
| *** k4n0 has joined #openstack-infra | 06:03 | |
| *** ryanpetrello has quit IRC | 06:05 | |
| *** talluri has quit IRC | 06:15 | |
| *** Ark has joined #openstack-infra | 06:17 | |
| tchaypo | ttx: we seem to have a conflict between the meetings page and the meetings ical feed as to the time of the tripleo meeting | 06:17 |
| *** Ark is now known as Guest5416 | 06:18 | |
| *** Masahiro has joined #openstack-infra | 06:19 | |
| *** Masahiro has quit IRC | 06:19 | |
| *** Masahiro has joined #openstack-infra | 06:19 | |
| *** dimtruck is now known as zz_dimtruck | 06:21 | |
| *** Sukhdev has quit IRC | 06:22 | |
| *** Longgeek has joined #openstack-infra | 06:22 | |
| *** mrmartin has joined #openstack-infra | 06:23 | |
| *** mudassirlatif has joined #openstack-infra | 06:23 | |
| *** Guest5416 has quit IRC | 06:26 | |
| *** Masahiro has quit IRC | 06:26 | |
| *** Masahiro has joined #openstack-infra | 06:27 | |
| *** dannywilson has quit IRC | 06:28 | |
| *** Masahiro has quit IRC | 06:29 | |
| *** Masahiro has joined #openstack-infra | 06:29 | |
| *** yfried has joined #openstack-infra | 06:31 | |
| *** talluri has joined #openstack-infra | 06:33 | |
| tchaypo | ttx: because the wiki page is wrong. That’s something I can fix, so I did. | 06:34 |
| *** stevemar has quit IRC | 06:34 | |
| *** SumitNaiksatam has quit IRC | 06:38 | |
| *** SumitNaiksatam has joined #openstack-infra | 06:39 | |
| *** guest764 has joined #openstack-infra | 06:41 | |
| *** emagana has quit IRC | 06:43 | |
| *** emagana has joined #openstack-infra | 06:43 | |
| *** belmoreira has joined #openstack-infra | 06:46 | |
| *** emagana has quit IRC | 06:48 | |
| *** pcrews has joined #openstack-infra | 06:48 | |
| *** afazekas has joined #openstack-infra | 06:50 | |
| *** viglesias has quit IRC | 06:51 | |
| *** pcrews has quit IRC | 06:53 | |
| *** viglesias has joined #openstack-infra | 06:56 | |
| *** fandi has joined #openstack-infra | 06:59 | |
| *** yfried is now known as yfried|afk | 06:59 | |
| *** Masahiro has quit IRC | 07:02 | |
| *** Masahiro has joined #openstack-infra | 07:03 | |
| *** yfried|afk is now known as yfried | 07:05 | |
| *** Masahiro has quit IRC | 07:05 | |
| *** rushiagr_away is now known as rushiagr | 07:07 | |
| *** fifieldt has quit IRC | 07:16 | |
| *** ashaeron has joined #openstack-infra | 07:20 | |
| *** yfried is now known as yfried|afk | 07:21 | |
| *** pcrews has joined #openstack-infra | 07:23 | |
| *** Masahiro has joined #openstack-infra | 07:25 | |
| *** talluri has quit IRC | 07:27 | |
| *** ujuc has quit IRC | 07:30 | |
| *** koolhead_ has quit IRC | 07:30 | |
| *** yfried|afk is now known as yfried | 07:31 | |
| *** koolhead17 has joined #openstack-infra | 07:31 | |
| *** achanda has joined #openstack-infra | 07:31 | |
| *** pcrews has quit IRC | 07:31 | |
| *** koolhead17 has quit IRC | 07:35 | |
| *** camunoz has quit IRC | 07:35 | |
| *** achanda has quit IRC | 07:36 | |
| *** Masahiro has quit IRC | 07:36 | |
| *** otter768 has joined #openstack-infra | 07:36 | |
| *** ala_ has joined #openstack-infra | 07:37 | |
| *** e0ne has joined #openstack-infra | 07:38 | |
| *** ociuhandu has quit IRC | 07:38 | |
| *** e0ne has quit IRC | 07:38 | |
| *** teran_ has joined #openstack-infra | 07:38 | |
| *** ildikov has joined #openstack-infra | 07:39 | |
| *** alla__ has joined #openstack-infra | 07:41 | |
| *** zz_dimtruck is now known as dimtruck | 07:41 | |
| *** otter768 has quit IRC | 07:41 | |
| *** unicell1 has joined #openstack-infra | 07:41 | |
| *** teran has quit IRC | 07:41 | |
| *** unicell has quit IRC | 07:43 | |
| *** ala_ has quit IRC | 07:44 | |
| *** SumitNaiksatam has quit IRC | 07:46 | |
| *** sputnik13 has joined #openstack-infra | 07:47 | |
| *** AlexF has quit IRC | 07:47 | |
| *** camunoz has joined #openstack-infra | 07:49 | |
| *** SumitNaiksatam has joined #openstack-infra | 07:49 | |
| *** jaypipes has quit IRC | 07:50 | |
| *** HeOS has quit IRC | 07:50 | |
| *** dimtruck is now known as zz_dimtruck | 07:51 | |
| *** ryanpetrello has joined #openstack-infra | 07:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove python26 jobs from all stackforge projects https://review.openstack.org/129434 | 07:51 |
| oomichi | clarkb: FYI: I could kick the gate job on some patches after rebasing them for the latest master. maybe they got old. | 07:51 |
| AJaeger | clarkb, https://review.openstack.org/#/c/129434/ is the python 2.6 job, I've rebased and added a comment with one question. | 07:53 |
| *** AlexF has joined #openstack-infra | 07:54 | |
| *** armax has quit IRC | 07:55 | |
| *** ryanpetrello has quit IRC | 07:56 | |
| *** salv-orlando has joined #openstack-infra | 08:02 | |
| *** ZZelle has quit IRC | 08:02 | |
| *** ZZelle has joined #openstack-infra | 08:03 | |
| *** amuller has joined #openstack-infra | 08:05 | |
| *** ildikov has quit IRC | 08:07 | |
| *** michchap has joined #openstack-infra | 08:11 | |
| *** dtantsur|afk is now known as dtantsur | 08:13 | |
| *** oomichi has quit IRC | 08:17 | |
| *** henditan has joined #openstack-infra | 08:17 | |
| *** yfried is now known as yfried|afk | 08:17 | |
| *** mudassirlatif has quit IRC | 08:17 | |
| *** yamahata has quit IRC | 08:18 | |
| *** Masahiro has joined #openstack-infra | 08:18 | |
| *** henditan has quit IRC | 08:19 | |
| *** markus_z has joined #openstack-infra | 08:21 | |
| *** arxcruz has joined #openstack-infra | 08:21 | |
| *** ildikov has joined #openstack-infra | 08:22 | |
| *** nadya has joined #openstack-infra | 08:26 | |
| *** nadya is now known as Guest28381 | 08:27 | |
| *** koolhead17 has joined #openstack-infra | 08:28 | |
| *** jcoufal has joined #openstack-infra | 08:29 | |
| *** koolhead_ has joined #openstack-infra | 08:32 | |
| *** achanda has joined #openstack-infra | 08:33 | |
| *** andreykurilin_ has joined #openstack-infra | 08:35 | |
| *** koolhead17 has quit IRC | 08:35 | |
| *** yfried|afk is now known as yfried | 08:38 | |
| *** MaxV has joined #openstack-infra | 08:40 | |
| ttx | tchaypo: thx! Should get reported on the meeting ical when Tony catches up | 08:40 |
| *** talluri has joined #openstack-infra | 08:42 | |
| openstackgerrit | Joakim Löfgren proposed openstack-infra/jenkins-job-builder: Add DRY publisher https://review.openstack.org/126317 | 08:42 |
| *** AlexF has quit IRC | 08:45 | |
| *** koolhead_ has quit IRC | 08:46 | |
| *** shashankhegde has quit IRC | 08:47 | |
| *** teran_ has quit IRC | 08:48 | |
| *** achanda has quit IRC | 08:48 | |
| *** mpavlase has joined #openstack-infra | 08:49 | |
| *** koolhead17 has joined #openstack-infra | 08:50 | |
| *** SumitNaiksatam has quit IRC | 08:51 | |
| *** SumitNaiksatam has joined #openstack-infra | 08:52 | |
| *** koolhead17 has quit IRC | 08:55 | |
| *** SumitNaiksatam has quit IRC | 08:56 | |
| *** berendt has joined #openstack-infra | 08:57 | |
| *** andreykurilin_ has quit IRC | 08:58 | |
| *** andreykurilin_ has joined #openstack-infra | 08:59 | |
| *** mpaolino has joined #openstack-infra | 08:59 | |
| *** jistr has joined #openstack-infra | 08:59 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** jp_at_hp has joined #openstack-infra | 08:59 | |
| *** jp_at_hp has quit IRC | 08:59 | |
| *** jp_at_hp has joined #openstack-infra | 08:59 | |
| *** SumitNaiksatam has joined #openstack-infra | 09:00 | |
| *** HeOS has joined #openstack-infra | 09:06 | |
| openstackgerrit | Joakim Löfgren proposed openstack-infra/jenkins-job-builder: Refactor checkstyle publisher https://review.openstack.org/126304 | 09:07 |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 09:07 |
| *** amotoki has quit IRC | 09:11 | |
| tchaypo | ttx: it’s okay, the ical was correct already, I just made the wiki match the ical :) | 09:11 |
| *** derekh has joined #openstack-infra | 09:12 | |
| *** Masahiro has quit IRC | 09:13 | |
| ttx | tchaypo: oh ok. All good then | 09:14 |
| *** e0ne has joined #openstack-infra | 09:16 | |
| *** bo_sh has joined #openstack-infra | 09:17 | |
| *** ihrachyshka has joined #openstack-infra | 09:19 | |
| *** vigneshvar has joined #openstack-infra | 09:20 | |
| *** Masahiro has joined #openstack-infra | 09:23 | |
| *** nfedotov has joined #openstack-infra | 09:23 | |
| *** sergsh_away is now known as sergsh | 09:25 | |
| *** dannywilson has joined #openstack-infra | 09:29 | |
| *** yamamoto has quit IRC | 09:32 | |
| *** sputnik13 has quit IRC | 09:33 | |
| *** dannywilson has quit IRC | 09:33 | |
| *** afazekas has quit IRC | 09:33 | |
| *** mpavlase has quit IRC | 09:35 | |
| *** otter768 has joined #openstack-infra | 09:37 | |
| *** Longgeek_ has joined #openstack-infra | 09:38 | |
| *** Longgeek has quit IRC | 09:39 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 09:41 |
| *** otter768 has quit IRC | 09:42 | |
| *** jedimike has joined #openstack-infra | 09:42 | |
| *** darvon has quit IRC | 09:43 | |
| *** darvon has joined #openstack-infra | 09:43 | |
| *** pblaho has joined #openstack-infra | 09:43 | |
| *** yamahata has joined #openstack-infra | 09:45 | |
| *** BobBall_AWOL is now known as BobBall | 09:45 | |
| rcarrillocruz | NikitaKonovalov: hi there, is there a python storyboard client code available yet? | 09:45 |
| rcarrillocruz | i'd be interested in get involved in that | 09:45 |
| NikitaKonovalov | rcarrillocruz: not yet | 09:46 |
| NikitaKonovalov | There is a change to set up a repo | 09:46 |
| NikitaKonovalov | https://review.openstack.org/#/c/136004/ | 09:46 |
| NikitaKonovalov | As soon as it is merged, I'll upload initial commits | 09:47 |
| *** Ark has joined #openstack-infra | 09:47 | |
| NikitaKonovalov | Those will contain basic layout, oslo apiclient, and some base classes | 09:48 |
| *** Ark is now known as Guest55395 | 09:48 | |
| *** afazekas has joined #openstack-infra | 09:48 | |
| NikitaKonovalov | I'm now working on setting up a manager for /users endpoint | 09:48 |
| NikitaKonovalov | As soon as it is ready, all other endpoints support will not be a problem | 09:49 |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 09:52 |
| *** andreykurilin_ has quit IRC | 09:52 | |
| *** Guest55395 has quit IRC | 09:53 | |
| *** jcoufal_ has joined #openstack-infra | 09:57 | |
| *** abhishek has joined #openstack-infra | 09:59 | |
| *** ociuhandu has joined #openstack-infra | 09:59 | |
| *** jcoufal has quit IRC | 10:00 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 10:01 |
| *** sasik has joined #openstack-infra | 10:02 | |
| *** sasik has left #openstack-infra | 10:02 | |
| *** ociuhandu_ has joined #openstack-infra | 10:03 | |
| *** ociuhandu has quit IRC | 10:04 | |
| *** ociuhandu_ is now known as ociuhandu | 10:04 | |
| abhishek | hi, does any one has idea why check-grenade-dsvm is failing on stable/juno patches? | 10:05 |
| *** rlandy has joined #openstack-infra | 10:05 | |
| *** afazekas has quit IRC | 10:07 | |
| abhishek | https://review.openstack.org/#/c/131987/ | 10:09 |
| openstackgerrit | Joakim Löfgren proposed openstack-infra/jenkins-job-builder: Refactor checkstyle publisher https://review.openstack.org/126304 | 10:10 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: oslo: add missing Python 2.6 jobs https://review.openstack.org/137321 | 10:11 |
| *** talluri has quit IRC | 10:18 | |
| *** talluri has joined #openstack-infra | 10:18 | |
| *** Guest28381 has quit IRC | 10:20 | |
| *** afazekas has joined #openstack-infra | 10:21 | |
| *** Masahiro has quit IRC | 10:25 | |
| *** andreykurilin_ has joined #openstack-infra | 10:27 | |
| *** mmaglana has quit IRC | 10:28 | |
| *** nadya_ has joined #openstack-infra | 10:29 | |
| *** rediskin has joined #openstack-infra | 10:31 | |
| *** koolhead17 has joined #openstack-infra | 10:31 | |
| *** KanagarajM has joined #openstack-infra | 10:34 | |
| *** ldnunes has joined #openstack-infra | 10:35 | |
| *** nadya_ has quit IRC | 10:38 | |
| abhishek | hi, does any one has idea why check-grenade-dsvm is failing on stable/juno patches? https://review.openstack.org/#/c/131987/ | 10:39 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 10:41 | |
| *** yamahata has quit IRC | 10:43 | |
| *** yamahata has joined #openstack-infra | 10:45 | |
| *** dizquierdo has joined #openstack-infra | 10:52 | |
| *** yfried is now known as yfried|afk | 10:55 | |
| *** jerryz has joined #openstack-infra | 10:58 | |
| *** adalbas has joined #openstack-infra | 10:58 | |
| *** rfolco has joined #openstack-infra | 11:01 | |
| *** maishsk has joined #openstack-infra | 11:02 | |
| maishsk | Anyone awake? | 11:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove python26 jobs from all stackforge projects https://review.openstack.org/129434 | 11:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python 2.6 for tooz https://review.openstack.org/137332 | 11:04 |
| *** nadya_ has joined #openstack-infra | 11:04 | |
| *** ildikov has quit IRC | 11:05 | |
| *** rfolco has quit IRC | 11:05 | |
| *** koolhead17 has quit IRC | 11:06 | |
| AJaeger | jd__, which oslo libs are missing python 2.6 gating? | 11:06 |
| AJaeger | maishsk, hi! | 11:07 |
| *** koolhead17 has joined #openstack-infra | 11:07 | |
| maishsk | AJaeger: Hi | 11:07 |
| maishsk | maybe you can help me out | 11:07 |
| *** rfolco has joined #openstack-infra | 11:07 | |
| AJaeger | maishsk, let's give it a try. What's up? | 11:08 |
| maishsk | So here goes. I am looking to create a easy to deploy environment that will allow people to contribute to Openstack | 11:09 |
| maishsk | i.e. a quick way to get gerrit, git up and running | 11:09 |
| maishsk | easiest way to do this - is a container | 11:10 |
| maishsk | in order to do this I will need a few things - username, email etc... | 11:10 |
| maishsk | but also an SSH key | 11:10 |
| maishsk | the public key is up on gerrit - because I pasted it there | 11:11 |
| *** rfolco has quit IRC | 11:12 | |
| AJaeger | and the private part of the key needs to be in your container - or home directory | 11:12 |
| *** koolhead17 has quit IRC | 11:12 | |
| maishsk | so I have .ssh/id_rsa.pub and .ssh/id_rsa I have | 11:12 |
| AJaeger | So, .pub is the public one, id_rsa the private one. | 11:12 |
| maishsk | now the question - is there a way to programatically create a key on gerrit? | 11:13 |
| maishsk | not through the gui? | 11:13 |
| maishsk | if so how? | 11:13 |
| AJaeger | Check the upstream gerrit documentation at: https://gerrit-documentation.storage.googleapis.com/Documentation/2.9.1/index.html | 11:13 |
| AJaeger | I'm not aware of anything - but I'm not the gerrit expert, those are still asleep ;) | 11:14 |
| maishsk | https://gerrit-review.googlesource.com/Documentation/cmd-set-account.html | 11:15 |
| AJaeger | You could write a script that asks for username etc and generates a key if none exists. | 11:15 |
| maishsk | but I assume I cannot do that from a remote location | 11:15 |
| *** yfried|afk is now known as yfried | 11:15 | |
| maishsk | So I have been looking here | 11:16 |
| maishsk | https://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html | 11:16 |
| AJaeger | I don't think so. And for the set-account command, you need to identify yourself first - so have already one ssh added | 11:16 |
| AJaeger | so, you should be able to upload with your gerrit credentials - good | 11:17 |
| maishsk | I am having difficulty with finding the correct curl commands and the correct endpoint... | 11:17 |
| AJaeger | maishsk, better wait for fungi or clarkb to help you with that... | 11:18 |
| maishsk | gotcha! | 11:19 |
| AJaeger | give them another 3 or 4 hours | 11:19 |
| maishsk | will catch them later when they decide to wake up .... :) | 11:19 |
| *** rfolco has joined #openstack-infra | 11:24 | |
| *** dizquierdo has quit IRC | 11:24 | |
| *** bo_sh has left #openstack-infra | 11:25 | |
| *** yfried is now known as yfried|afk | 11:25 | |
| *** Masahiro has joined #openstack-infra | 11:26 | |
| *** sasik has joined #openstack-infra | 11:28 | |
| *** sasik has left #openstack-infra | 11:28 | |
| *** yfried|afk is now known as yfried | 11:28 | |
| *** mmaglana has joined #openstack-infra | 11:29 | |
| *** salv-orlando has quit IRC | 11:29 | |
| *** mpavlase has joined #openstack-infra | 11:30 | |
| *** Masahiro has quit IRC | 11:30 | |
| *** aysyd has joined #openstack-infra | 11:31 | |
| *** hdd has joined #openstack-infra | 11:35 | |
| *** dizquierdo has joined #openstack-infra | 11:36 | |
| *** mmaglana has quit IRC | 11:36 | |
| *** otter768 has joined #openstack-infra | 11:38 | |
| *** otter768 has quit IRC | 11:43 | |
| *** achuprin_ has quit IRC | 11:46 | |
| *** jamielennox is now known as jamielennox|away | 11:48 | |
| *** hdd has quit IRC | 11:49 | |
| *** dmakogon_ is now known as denis_makogon | 11:49 | |
| *** fandi has quit IRC | 11:50 | |
| yolanda | jeblair, clarkb, i have an issue with nodepool and dib, please ping me when you are available | 11:57 |
| sdague | abhishek: because swift did not start - http://logs.openstack.org/87/131987/1/check/check-grenade-dsvm/6a8d77b/logs/old/screen-s-proxy.txt.gz | 12:03 |
| *** achuprin_ has joined #openstack-infra | 12:03 | |
| ekarlso- | is there anything wrong with devstack these days ? | 12:04 |
| ekarlso- | 2014-11-26 12:01:05.108 31830 TRACE glance os.makedirs(self._signing_dirname, stat.S_IRWXU) | 12:04 |
| ekarlso- | 2014-11-26 12:01:05.108 31830 TRACE glance File "/usr/lib/python2.7/os.py", line 150, in makedirs | 12:04 |
| ekarlso- | 2014-11-26 12:01:05.108 31830 TRACE glance makedirs(head, mode) | 12:04 |
| ekarlso- | 2014-11-26 12:01:05.108 31830 TRACE glance File "/usr/lib/python2.7/os.py", line 157, in makedirs | 12:04 |
| ekarlso- | 2014-11-26 12:01:05.108 31830 TRACE glance mkdir(name, mode) | 12:04 |
| ekarlso- | 2014-11-26 12:01:05.108 31830 TRACE glance OSError: [Errno 13] Permission denied: '/var/cache/glance' | 12:04 |
| ekarlso- | http://paste.ubuntu.com/9248969/ is my local.conf | 12:04 |
| *** fandi has joined #openstack-infra | 12:06 | |
| sdague | ekarlso-: I'm sure there is plenty wrong, however it's running hundreds of successful gate jobs every day | 12:06 |
| ekarlso- | dunno why that happens then on a fresh trusty vm :| | 12:07 |
| *** cnesa has joined #openstack-infra | 12:07 | |
| sdague | your service list looks ancient | 12:09 |
| ekarlso- | sdague: what should they be ? :D | 12:09 |
| *** baoli has quit IRC | 12:10 | |
| sdague | don't know for all things, but don't assign to ENabled_sevices directly, just enable / disable what you need | 12:10 |
| mrmartin | sdague: I was looking for nginx examples in system-config and found only remains of a nginx-remove module. Do we have a policy to not use nginx for some reason? | 12:10 |
| *** ildikov has joined #openstack-infra | 12:11 | |
| chmouel | devananda, sdague: just catched with my highlights, the proposal is up https://review.openstack.org/#/c/137054/ | 12:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Sort taskflow properly https://review.openstack.org/137348 | 12:19 |
| *** salv-orlando has joined #openstack-infra | 12:20 | |
| sdague | mrmartin: no particular policy per say, beyond the fact that the infrastructure runs on apache, and unless there is a really good reason, maintaining a second web server stack is something that should be avoided | 12:26 |
| sdague | chmouel: yeh, did you see my follow up message on list ? | 12:26 |
| chmouel | sdake: i don't think so, is is the part about being able to have integrated projects have a /devstack/ dir ? | 12:27 |
| mrmartin | sdague, ok, beacuse I'm working on this askbot puppetization and the current site is running on nginx, but it is a good argument to use apache to avoid additional maintenance | 12:27 |
| AJaeger | sdague, could you review https://review.openstack.org/#/c/133368/ , please? | 12:28 |
| sdague | chmouel: yeh, I was thinking about the devstack side of the interface | 12:28 |
| *** viktors has joined #openstack-infra | 12:28 | |
| *** KanagarajM has quit IRC | 12:28 | |
| chmouel | sdague: yeah i mentton it in the spec, | 12:28 |
| sdague | ok, I didn't think it was in the first rev, I'll look again | 12:29 |
| sdague | oh... look, more revs :) | 12:29 |
| sdague | sorry, yesterday was a long and busy day | 12:29 |
| chmouel | sdague: ah no worries :) | 12:29 |
| viktors | hi folks! Does anybody knows the current state of the gate-tempest-dsvm-neutron-src-*-icehouse and gate-tempest-dsvm-src-*-icehousegate jobs ? | 12:29 |
| chmouel | sdague: it's just that part http://pastie.org/pastes/9744581/text?key=alsym8lgyztx6sadoozmew | 12:30 |
| sdague | the icehouse jobs should have been fixed yesterday | 12:30 |
| chmouel | sdague: maybe i should enphasis that's the goal of this spec? | 12:30 |
| chmouel | emphasis even | 12:30 |
| viktors | sdague: I'm still observing issues with these jobs =( | 12:30 |
| sdague | chmouel: so, I think the difference that I was thinking was around the plug space | 12:31 |
| sdague | viktors: url ? | 12:31 |
| viktors | https://review.openstack.org/#/c/136207/ | 12:31 |
| viktors | 'doc8' is not a global requirement but it should be,something went wrong | 12:31 |
| sdague | yeh, oslo.db apparently added a new requirement when we weren't enforcing stable requirements | 12:32 |
| sileht | sdague, hi, I have seen the result of the meeting yesterday for pinnig oslolib/client version for stable branches, I have oslo.messaging gate blocked since stable jobs have been enabled since, perhaps we can merged https://review.openstack.org/#/c/134727/ now ? | 12:32 |
| sdague | that's a valid fail in the existing system | 12:32 |
| *** rpodolyaka1 has joined #openstack-infra | 12:32 | |
| viktors | sdague: so should I add this requirement to the stable requirements? | 12:33 |
| sdague | sileht: it's going to need to be done systemically, not one lib at a time, or we're going to lose everything. I'd suggest coordinating with ttx and stable maint on getting that rolling | 12:33 |
| sdague | viktors: no, this is the whole conversation around requirements yesterday | 12:33 |
| AJaeger | sdague, doc8 is in global-requirements for trunk but perhaps not for icehouse? | 12:33 |
| sdague | please take a look at yesterday's project meeting | 12:33 |
| sdague | AJaeger: yep | 12:34 |
| sileht | sdague, ok, thanks | 12:34 |
| viktors | sdague: ok, will look | 12:34 |
| *** xchu has joined #openstack-infra | 12:36 | |
| chmouel | sdake: feel free to comment on the specs, i think it wil be easier there :) | 12:37 |
| *** xchu has quit IRC | 12:37 | |
| sdague | the net of this, oslo libs are going to be blocked until this is resolved. In moving to a new testing scheme we temporarily allowed oslo libs to do a thing that they couldn't before in adding requirements that only exist in master, but that's going to break everything on release. So the capping scheme has to be implemented or the oslo libs need to roll back their dependency changes to release. | 12:38 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/infra-specs: StoryBoard: story types specification https://review.openstack.org/129267 | 12:45 |
| *** rushiagr is now known as rushiagr_away | 12:46 | |
| sdague | chmouel: ok, commented in the spec | 12:47 |
| sdague | consider my -1 advisory of "hey, what about this approach" | 12:47 |
| sdague | it's not something I'm married to | 12:47 |
| *** jcoufal_ has quit IRC | 12:47 | |
| *** hdd has joined #openstack-infra | 12:48 | |
| *** jcoufal has joined #openstack-infra | 12:48 | |
| chmouel | sdague: no i like it :) i wasn't sure if allowing all XX-*.sh would make sense to be allowed but i guess that worked with for sysv-rc scripts for a while | 12:51 |
| sdague | chmouel: yeh, until we write devstack-systemd ;) | 12:52 |
| chmouel | let me try to put a new version of the spec getting your suggestions in there, | 12:52 |
| chmouel | sdague: lol :) | 12:52 |
| *** talluri has quit IRC | 12:54 | |
| *** koolhead17 has joined #openstack-infra | 12:54 | |
| *** baoli has joined #openstack-infra | 12:54 | |
| *** koolhead17 has quit IRC | 12:54 | |
| *** koolhead17 has joined #openstack-infra | 12:54 | |
| jkt | hmm, trying to run the test suite of turbo-hipster on centos 7, I had to modify the test-requirements.txt so that "six" and "unittest2" are listed before "testtools", otherwise I got this: http://fpaste.org/154261/41700678/ | 12:59 |
| jkt | am I doing something wrong? | 12:59 |
| jkt | and I don't see any dependencies listed at http://bazaar.launchpad.net/~testing-cabal/testtools/trunk/files , but maybe I just don't know where to look | 13:00 |
| *** banix has joined #openstack-infra | 13:01 | |
| *** xchu has joined #openstack-infra | 13:01 | |
| jkt | I mean, this is listed at http://bazaar.launchpad.net/~testing-cabal/testtools/trunk/view/head:/README.rst , but I was hoping for something machine-readable | 13:02 |
| sdague | jkt: what version of testtools? | 13:04 |
| *** banix has quit IRC | 13:04 | |
| *** xchu has quit IRC | 13:04 | |
| sdague | jkt: http://bazaar.launchpad.net/~testing-cabal/testtools/trunk/view/head:/setup.py is where the dependencies are listed | 13:04 |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Add more details on event dashboard https://review.openstack.org/135954 | 13:05 |
| maishsk | AJaeger: is there a sandbox area where I can test upload a commit - I don't want to start polluting any repositories... | 13:06 |
| AJaeger | maishsk, sure, it's call sandbox ;) | 13:06 |
| AJaeger | maishsk, http://git.openstack.org/cgit/openstack-dev/sandbox/ | 13:06 |
| AJaeger | PLease abandon patches afterwards so that the review queue is clean | 13:06 |
| maishsk | I wonder why I did not think of that.... :) | 13:06 |
| maishsk | of course... | 13:07 |
| *** xchu has joined #openstack-infra | 13:10 | |
| *** Longgeek_ has quit IRC | 13:11 | |
| jkt | sdague: I do see that in my version of testtools, too (appears to be "1.5.0") | 13:11 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: spec2bp v2 -- support in-review specs https://review.openstack.org/108041 | 13:12 |
| *** Masahiro has joined #openstack-infra | 13:14 | |
| *** xchu has quit IRC | 13:16 | |
| jkt | sdague: I'm gonna try it with upgraded pip because I've seen it doing *crazy* stuff here on centos7 | 13:16 |
| sdague | what pip version? | 13:16 |
| jkt | python-pip-1.3.1-4.el7.noarch | 13:17 |
| jkt | stuff like ignoring any version requirements as long as a wrong version is already in the download cache, etc | 13:17 |
| jkt | ok, no change here :( | 13:17 |
| sdague | oh, yeh, so that's terrible | 13:19 |
| *** yaguang_ has quit IRC | 13:19 | |
| sdague | we are definitely at min req of 1.4 | 13:19 |
| *** julim has quit IRC | 13:19 | |
| *** Masahiro has quit IRC | 13:19 | |
| sdague | expect everything to be broken if you have < 1.4 | 13:20 |
| *** nosnos has quit IRC | 13:20 | |
| *** mmaglana has joined #openstack-infra | 13:20 | |
| *** alexpilotti has joined #openstack-infra | 13:21 | |
| maishsk | AJaeger: it seems this actually works ... https://review.openstack.org/#/c/137355/ | 13:21 |
| *** bswartz has quit IRC | 13:22 | |
| *** jpich has quit IRC | 13:22 | |
| AJaeger | maishsk, congrats! | 13:22 |
| *** frank_zhou has joined #openstack-infra | 13:23 | |
| maishsk | :) | 13:23 |
| *** jpich has joined #openstack-infra | 13:23 | |
| *** mmaglana has quit IRC | 13:25 | |
| jkt | sdague: so I erased the systemwide pip, so that I now have pip from pip (1.5.6), but I still get this: http://jkt.flaska.net/tmp/tox-log | 13:25 |
| sdague | jkt: yeh, honestly, I don't know. I don't use centos and haven't tried to install turbohipster | 13:28 |
| sdague | the pbr < 0.6 seems suspicious to me | 13:28 |
| *** BharatK has quit IRC | 13:29 | |
| *** vigneshvar_ has joined #openstack-infra | 13:31 | |
| *** weshay has joined #openstack-infra | 13:31 | |
| *** alexpilotti has quit IRC | 13:32 | |
| openstackgerrit | Cedric Brandily proposed openstack-infra/git-review: get_remote_url(): honor any "url.<base>.insteadOf" config setting https://review.openstack.org/109851 | 13:33 |
| *** fandi has quit IRC | 13:33 | |
| *** vigneshvar has quit IRC | 13:34 | |
| *** jaypipes has joined #openstack-infra | 13:36 | |
| *** mestery has quit IRC | 13:36 | |
| *** ildikov has quit IRC | 13:36 | |
| *** Longgeek has joined #openstack-infra | 13:37 | |
| *** baoli has quit IRC | 13:37 | |
| *** dizquierdo has quit IRC | 13:38 | |
| *** otter768 has joined #openstack-infra | 13:39 | |
| *** berendt has quit IRC | 13:40 | |
| *** BharatK has joined #openstack-infra | 13:41 | |
| *** ryanpetrello has joined #openstack-infra | 13:42 | |
| *** rushiagr_away is now known as rushiagr | 13:42 | |
| *** dizquierdo has joined #openstack-infra | 13:42 | |
| *** otter768 has quit IRC | 13:44 | |
| *** baoli has joined #openstack-infra | 13:44 | |
| mordred | morning | 13:48 |
| mordred | jkt, sdague: you are saying words that sound like I should look at something | 13:48 |
| *** ildikov has joined #openstack-infra | 13:48 | |
| *** julim has joined #openstack-infra | 13:49 | |
| jkt | what I'm saying is that I have no idea why this is broken, mordred | 13:49 |
| jkt | jhesketh: it seems that there's some non-determinism in the way turbo-hipster's tests are done, see http://fpaste.org/154275/17009697/ | 13:49 |
| jkt | jhesketh: my torbo-hipster has some trivial patches, http://fpaste.org/154276/09787141/ -- the disabling of jjb and mysql is there because I believe that they are not neeed in my setup (are they?) | 13:50 |
| *** yamahata has quit IRC | 13:50 | |
| *** dimsum__ has joined #openstack-infra | 13:51 | |
| *** erw has quit IRC | 13:51 | |
| *** yamahata has joined #openstack-infra | 13:51 | |
| mordred | jkt: can you also pip install unittest2 by hand real quick? | 13:52 |
| *** erw has joined #openstack-infra | 13:52 | |
| mordred | it looks like there is a bug somewhere that is not telling your system that unittest2 is now needed by testtools | 13:53 |
| *** BharatK has quit IRC | 13:53 | |
| *** yfried is now known as yfried|afk | 13:53 | |
| jkt | mordred: while I was trying to fix this, I used yum to install it | 13:53 |
| yolanda | mordred, you there? i have a problem with nodepool and dib image upload, can we talk about it? | 13:53 |
| mordred | either way | 13:53 |
| jkt | mordred: but because this is a virtual env, apparently, I belive that it doesn't make any difference what actually is in the system | 13:53 |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/nodepool: Add per-provider env vars https://review.openstack.org/116993 | 13:53 |
| mordred | ah - yes, in a venv no, that will not have helped | 13:53 |
| *** nadya_ has quit IRC | 13:54 | |
| jkt | mordred: for example, it went trying to compile PyYAML even though it's already present | 13:54 |
| mordred | so install it in to the venv with pip | 13:54 |
| jkt | mordred: how do I do that? | 13:54 |
| mordred | what's the venv? | 13:54 |
| mordred | oh - you're just doing tox? | 13:54 |
| jkt | I have no idea, I just run `tox` in a top-level dir of turbo-hipster | 13:54 |
| jkt | yes | 13:54 |
| fungi | dhellmann: sdague: just a heads up, i'm wondering if https://review.openstack.org/136879 is likely to wedge oslo.vmware icehouse stable-compat jobs? | 13:54 |
| *** che-arn8 has joined #openstack-infra | 13:54 | |
| *** che-arne has joined #openstack-infra | 13:54 | |
| *** che-arn8 has quit IRC | 13:54 | |
| mordred | do this: .tox/py27/bin/pip install unittest2 | 13:55 |
| *** vigneshvar_ has quit IRC | 13:55 | |
| jkt | mordred: it's present now because I added it before. So I'll nuke the env, undo my local change, and try again | 13:56 |
| fungi | jkt: if you pass -r to tox it will recreate its venv(s) | 13:56 |
| sdague | fungi: yeh, possibly, like I said in the project meeting this whole thing is complicated enough to need a champion to navigate through it | 13:56 |
| *** thingee has quit IRC | 13:57 | |
| *** fabo has quit IRC | 13:58 | |
| *** thingee has joined #openstack-infra | 13:59 | |
| jkt | mordred: I don't think it helped. I undid my changes to test-requirements.txt, destroyed the env, run `tox -e py27` which failed, as expected | 14:01 |
| jkt | mordred: then I run `.tox/py27/bin/pip install unittest2` as requested, it installed stuff, and re-run `tox -e py27` again, which failed with the same error as before | 14:02 |
| *** chandankumar has quit IRC | 14:02 | |
| sdague | jkt: what version of pip, what version of tox | 14:02 |
| jkt | mordred: my assumption is that tox notices that the venv is not set up properly, and decided to wipe it out before proceeding | 14:02 |
| *** hdd has quit IRC | 14:02 | |
| sdague | jkt: that is the wrong assumption | 14:02 |
| sdague | tox does nothing to the venv if it exists | 14:02 |
| sdague | evenif it is wrong | 14:02 |
| jkt | sdague: tox 1.4.2, pip 1.5.6 | 14:02 |
| mordred | jkt: I'm trying locally | 14:03 |
| sdague | so at least in the main projects, you need tox 1.6 | 14:03 |
| *** mjturek has joined #openstack-infra | 14:04 | |
| jkt | sdague: how do you explain that "unittest2" is now absent from my ./.tox/py27/lib/python2.7/site-packages/ ? | 14:05 |
| *** mjturek has left #openstack-infra | 14:05 | |
| *** mestery has joined #openstack-infra | 14:05 | |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/system-config: Fix vhost configuration sections https://review.openstack.org/132170 | 14:05 |
| jkt | sdague: OK, I just checked that running `tox -e py27` effectively removed unittest2 from there | 14:06 |
| jkt | I'll try with tox from pip as opposed to my system's copy | 14:06 |
| jkt | maybe rhel7 is just too obsolete for this tuff already :( | 14:06 |
| dteselkin | Hi | 14:07 |
| dteselkin | I have two reviews with 2 +2, but not merged. Could someone take a look please? | 14:07 |
| *** bswartz has joined #openstack-infra | 14:08 | |
| jkt | sdague, mordred: tox 1.8.1 fixes this issue | 14:09 |
| *** thingee has quit IRC | 14:10 | |
| *** AJaeger has quit IRC | 14:11 | |
| mordred | jkt: w00t! | 14:11 |
| *** AJaeger has joined #openstack-infra | 14:11 | |
| *** AJaeger has joined #openstack-infra | 14:11 | |
| sdague | jkt: I wonder if this would fix it for you - https://review.openstack.org/137364 | 14:12 |
| *** yfried|afk is now known as yfried | 14:13 | |
| *** tchaypo has quit IRC | 14:15 | |
| * jkt checks | 14:15 | |
| *** thingee has joined #openstack-infra | 14:15 | |
| dteselkin | It's https://review.openstack.org/133077 and https://review.openstack.org/130811 | 14:16 |
| *** tchaypo has joined #openstack-infra | 14:17 | |
| jkt | sdague: commented in there | 14:18 |
| *** ameade_ has quit IRC | 14:18 | |
| jkt | is there a turbo-hipster channel? #turbo-hipster isn't it, apparently | 14:18 |
| sdague | jkt: those folks mostly are here | 14:19 |
| jkt | ok, I just asked as openstackgerrit hasn't reported your change in here | 14:20 |
| *** ameade_ has joined #openstack-infra | 14:20 | |
| *** thingee has quit IRC | 14:20 | |
| *** thingee has joined #openstack-infra | 14:21 | |
| *** ayoung_ZZzz_zzZZ is now known as ayoung | 14:21 | |
| *** frank_zhou has quit IRC | 14:21 | |
| *** dkliban_afk is now known as dkliban | 14:22 | |
| jkt | jhesketh: OK, I'm definitely seeing test flakiness even with just tests.test_worker_manager.TestWorkerServer.test_zuul_manager_started, see http://fpaste.org/154294/01204914/ for an example of two executions | 14:28 |
| jkt | jhesketh: both invocations reported "passed", but I do not terribly like the backtraces in there | 14:28 |
| *** Ark has joined #openstack-infra | 14:30 | |
| *** Ark is now known as Guest62036 | 14:30 | |
| *** yfried is now known as yfried|afk | 14:31 | |
| *** jgrimm is now known as zz_jgrimm | 14:32 | |
| jkt | seems that the gear server gets shutdown, interesting | 14:32 |
| *** Longgeek has quit IRC | 14:34 | |
| *** Guest62036 has quit IRC | 14:34 | |
| *** nadya_ has joined #openstack-infra | 14:34 | |
| *** armax has joined #openstack-infra | 14:36 | |
| *** thingee has quit IRC | 14:36 | |
| *** r-daneel has joined #openstack-infra | 14:37 | |
| *** Longgeek has joined #openstack-infra | 14:40 | |
| *** mikeit has joined #openstack-infra | 14:42 | |
| *** xyang0 has joined #openstack-infra | 14:43 | |
| *** AlexF has joined #openstack-infra | 14:44 | |
| *** che-arne has quit IRC | 14:46 | |
| *** bhunter71 has joined #openstack-infra | 14:46 | |
| *** mikeit has quit IRC | 14:47 | |
| *** mikeit has joined #openstack-infra | 14:48 | |
| *** mestery has quit IRC | 14:48 | |
| *** kgiusti has joined #openstack-infra | 14:49 | |
| *** dkranz has joined #openstack-infra | 14:50 | |
| *** rediskin has left #openstack-infra | 14:54 | |
| *** AJaeger_ has joined #openstack-infra | 14:55 | |
| *** che-arne has joined #openstack-infra | 14:56 | |
| clarkb | AJaeger thank you. I will review | 14:57 |
| AJaeger_ | morning, clarkb! | 14:57 |
| *** markmcclain has joined #openstack-infra | 14:58 | |
| clarkb | good morning/afternoon | 14:58 |
| AJaeger_ | clarkb: there was only one request for python 2.6 - tooz. I've put this in as additional change. | 14:58 |
| AJaeger_ | clarkb: 4pm for me ;) | 14:58 |
| fungi | well, and jd__'s patch to add 25 back to a lot of the oslo libs they were missing from | 14:58 |
| clarkb | ya I see tooz as the child change | 14:58 |
| fungi | er, s/25/26/ | 14:58 |
| *** skolekonov has joined #openstack-infra | 14:58 | |
| AJaeger_ | fungi: yeah, exactly. | 14:58 |
| clarkb | fungi 2 libs and not really missing | 14:59 |
| AJaeger_ | That patch to remove them was proposed before further discussions happened. | 14:59 |
| clarkb | its the special concurrenct or whatever test | 14:59 |
| fungi | oh, right, most of it's just the new template | 14:59 |
| AJaeger_ | and it was for 2 removed - and many more added now ;) | 14:59 |
| AJaeger_ | fungi: yeah. | 14:59 |
| *** thingee has joined #openstack-infra | 15:00 | |
| clarkb | but I havent made up my mind yet ... | 15:00 |
| fungi | it also sets oslo.log gating on py33 | 15:01 |
| fungi | though it was already voting in check | 15:01 |
| fungi | so that's more just cleanup i guess | 15:01 |
| clarkb | this isnt actually adding any new py26 jobs | 15:01 |
| AJaeger_ | fungi: exactly | 15:02 |
| *** mattfarina has joined #openstack-infra | 15:02 | |
| AJaeger_ | clarkb: it adds back the python jobs removed in my other patch... | 15:02 |
| clarkb | oh I didnt see that | 15:02 |
| fungi | oslo.concurrency | 15:02 |
| fungi | oh, and oslo.log | 15:03 |
| *** Masahiro has joined #openstack-infra | 15:03 | |
| clarkb | so maybe we should revert the other change | 15:04 |
| fungi | so, yeah, adds py26 jobs for oslo.concurrency and oslo.log, adds py33 to the gate for oslo.log though that was already voting on check | 15:04 |
| clarkb | and not have this largee hard to see what is really changing change | 15:04 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/system-config: Support filtering by owner information https://review.openstack.org/137372 | 15:05 |
| fungi | it's raining buckets here. i think someone turned the ocean upside down | 15:05 |
| *** wuhg has quit IRC | 15:05 | |
| *** wuhg has joined #openstack-infra | 15:05 | |
| AJaeger_ | Note the two sets of changes - stackforge and oslo libs - are independend of each other, just related by topic | 15:06 |
| fungi | and i don't think they merge conflict | 15:06 |
| AJaeger_ | fungi: they shouldn't merge conflict. And we can merge oslo now, just wait until Sunday with the stackforge one... | 15:07 |
| *** Masahiro has quit IRC | 15:08 | |
| jd__ | is there a window for our tooz rename/migration to openstack/ ? | 15:08 |
| clarkb | personally I am -1 on that chnage | 15:08 |
| jd__ | and hi btw, I'm rude :) | 15:08 |
| clarkb | we should revert the change that was a problem | 15:08 |
| clarkb | because I could not understand what was actually fixed in the new change | 15:08 |
| fungi | i'm fine with the patch as-is, though i agree separate patches to revert the removal, add teh missing py33 job and templatify py26 would be cleaner | 15:08 |
| clarkb | which may or may not be due to being awake at 7am | 15:09 |
| fungi | yeah, i'm already on coffee #2 so things may make more sense to me briefly than they should too | 15:09 |
| *** tonytan4ever has joined #openstack-infra | 15:09 | |
| *** mmaglana has joined #openstack-infra | 15:09 | |
| AJaeger_ | jd__: we didn't schedule the tooz rename yet AFAIK | 15:09 |
| *** thingee has quit IRC | 15:09 | |
| AJaeger_ | clarkb, fungi: Shall I propose a reversal of https://review.openstack.org/#/c/130444/ and jd__ is adjusting his change as pure cleanup? | 15:10 |
| *** jcoufal has quit IRC | 15:10 | |
| clarkb | AJaeger_: that would be my preference | 15:11 |
| jd__ | wfm | 15:11 |
| fungi | fine here | 15:11 |
| fungi | happy to re-review the decomposition | 15:11 |
| jd__ | AJaeger_: I can git revert + my patch and git review send that? | 15:11 |
| *** ildikov has quit IRC | 15:11 | |
| AJaeger_ | jd__: let's do the revert quickly, I'll propose it and we can merge quickly... | 15:12 |
| fungi | jd__: separating out the py26 templating and the addition of the missing py33 gate job (switching to python3-jobs template for oslo.log) would also probably help | 15:12 |
| *** AJaeger has quit IRC | 15:12 | |
| fungi | jd__: makes it more obvious that was intended and not an accident | 15:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Adjust python 2.6 jobs for oslo libraries" https://review.openstack.org/137373 | 15:13 |
| *** AJaeger has joined #openstack-infra | 15:13 | |
| *** mmaglana has quit IRC | 15:14 | |
| AJaeger_ | clarkb, fungi: since we're discussing oslo - I suggest to merge this one as well, it's a pure rearranging of lines for clarity: https://review.openstack.org/137348 | 15:15 |
| jkt | I don't think this test log is good, http://fpaste.org/154325/01491214/ | 15:15 |
| *** thingee has joined #openstack-infra | 15:16 | |
| *** mestery has joined #openstack-infra | 15:16 | |
| fungi | AJaeger_: it will likely merge-conflict with what jd__ is working on so we likely want to either ask him to rebase once it merges or rebase it on what he proposes | 15:16 |
| clarkb | jkt: it looks like the connection object is going away. | 15:18 |
| clarkb | jkt: possibly just a race between test cleanup and gear sending status updates? | 15:18 |
| clarkb | jkt: but I don't know enough about turbo hipsters tests to say | 15:18 |
| jkt | clarkb: actually, I have multiple logs, and sometimes this pretty trivial test is getting stuck, that's why I'm looking into it | 15:18 |
| fungi | jkt: yeah, that was after gear shutdown, so might just be ignored (though i agree not necessarily very clean) | 15:18 |
| AJaeger_ | fungi: agreed. So, we would schedule the rename at next week's meeting? In that case, I suggest we rebase the tooz patch after mine. jd__ shall I do this for you? | 15:19 |
| jkt | such as this one, http://fpaste.org/154329/17015140/ | 15:19 |
| clarkb | AJaeger_: ya we should schedule soon since we will between holidays | 15:19 |
| clarkb | if we wait too long it will be hard to schedule again | 15:20 |
| fungi | AJaeger_: clarkb: sounds good | 15:20 |
| AJaeger_ | clarkb: so, discuss at next meeting? | 15:20 |
| clarkb | AJaeger_: yup | 15:20 |
| fungi | i'm expecting to be mostly afk starting in a few hours, so i'm going to scramble to try and get a few things knocked out | 15:20 |
| AJaeger_ | Then let me move this up on the agenda | 15:20 |
| AJaeger_ | fungi: for how long? | 15:21 |
| clarkb | fungi: you driving into the mountains? | 15:21 |
| fungi | clarkb: no, have a lot of pies to bake this afternoon, then driving to visit christine's family for the next couple days | 15:21 |
| *** Ark has joined #openstack-infra | 15:21 | |
| fungi | AJaeger_: likely not around much through this weekend | 15:22 |
| clarkb | I am doing stuffing, turkey, and potatoes tomorrow morning | 15:22 |
| *** Ark is now known as Guest10529 | 15:22 | |
| clarkb | AJaeger_: it is a US holiday on thursday and typically people have a 4 day weekend | 15:22 |
| fungi | it is the annual gluttony celebration | 15:22 |
| fungi | recovery usually takes a few days | 15:23 |
| jd__ | AJaeger: it was already supposed to be scheduled yesterday… | 15:23 |
| clarkb | apparently they are getting snow days today in the NE so 5 day weekend for some | 15:23 |
| jd__ | AJaeger: so yeah ASAP :) | 15:23 |
| AJaeger_ | fungi: ah, thanks giving - I know about it, just feared you were gone for much longer... | 15:23 |
| * AJaeger_ already appreciates that my US colleagues are on vacation :) | 15:24 | |
| *** ildikov has joined #openstack-infra | 15:24 | |
| mtreinish | clarkb: yes, it's snowing... | 15:24 |
| AJaeger_ | jd__: I just updated https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting and moved it to the top | 15:24 |
| jd__ | AJaeger: 👍 | 15:25 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 15:27 |
| *** david-lyle_afk is now known as david-lyle | 15:28 | |
| *** dimsum__ has quit IRC | 15:28 | |
| *** stevemar has joined #openstack-infra | 15:28 | |
| AJaeger_ | jd__: I'll base 135215 on top of 137348. Ok? | 15:28 |
| *** dimsum__ has joined #openstack-infra | 15:29 | |
| jd__ | AJaeger: ok | 15:29 |
| sdague | clarkb: well, except for us work from home folks | 15:30 |
| *** gokrokve has joined #openstack-infra | 15:30 | |
| sdague | it does mean I'll have a shoveling break at some point | 15:30 |
| sdague | though my snow blower is also supposed to arrive today | 15:30 |
| *** Sukhdev has joined #openstack-infra | 15:31 | |
| *** marun has joined #openstack-infra | 15:31 | |
| clarkb | sdague: I think you get to roll around in the snow even when you work from home | 15:31 |
| sdague | heh | 15:31 |
| *** radez is now known as radez_g0n3 | 15:32 | |
| clarkb | ryanpetrello: you around? so we can remove py26 from pecan | 15:32 |
| *** dmsimard_away is now known as dmsimard | 15:32 | |
| clarkb | ryanpetrello: or if you prefer we can continue running those jobs for you. With the understanding that we really will remove them in about a year | 15:32 |
| *** dmsimard is now known as dmsimard_away | 15:32 | |
| *** dmsimard_away is now known as dmsimard | 15:33 | |
| sdague | clarkb: https://twitter.com/sdague/status/537630175924879360 - view out my office window | 15:33 |
| clarkb | AJaeger_: In your base change we should remove all of pecan's py26 tests and sqlalchemy migrate's py26 tests | 15:33 |
| clarkb | AJaeger_: we should probably add migrates back in a change since a lot of openstack that does py26 depends on it | 15:34 |
| clarkb | AJaeger_: then hear from ryanpetrello on whether or not he wants us to keep running those jobs | 15:34 |
| clarkb | AJaeger_: does that sound reasonable? | 15:34 |
| AJaeger_ | clarkb: yes, sounds reasonable. But that's a different patch set. Will do. | 15:34 |
| clarkb | sdague: ya family in NJ said that school was cancelled yesterday before any snow had fallen | 15:34 |
| clarkb | AJaeger_: ya will need a new patch I think that is ok | 15:34 |
| *** abhishek has quit IRC | 15:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Adjust python 2.6 jobs for oslo libraries" https://review.openstack.org/137373 | 15:35 |
| clarkb | sdague: on this side of the country the leaves are finally turning | 15:36 |
| sdague | clarkb: yeh, they are calling for 8 - 16" here, glad we are not traveling for turkey day | 15:36 |
| clarkb | it has been a weird fall | 15:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 15:36 |
| sdague | yay! snowblower delivery on it's way | 15:36 |
| *** ddieterly has joined #openstack-infra | 15:36 | |
| sdague | so I won't be shoveling, I'll be blowing | 15:36 |
| fungi | upgrade | 15:37 |
| fungi | now you get to be that annoying neighbor who runs his snow blower at 5am | 15:37 |
| clarkb | fungi: pretty sure that is how you end up disappeared | 15:38 |
| sdague | yeh, well last year we got a 16" snow storm while I was in korea on a trip. So my wife put her foot down that for me to get on a plane this winter, there would be a snow blower. | 15:38 |
| fungi | they find you sticking partway out of a melting drift at some point halfway into the spring thaw | 15:38 |
| *** alexpilotti has joined #openstack-infra | 15:39 | |
| sdague | fungi: they all have them here, so no one cares if you run them early. Honestly, they aren't that loud. | 15:39 |
| clarkb | we are supposed to get biblical amounts of rain | 15:39 |
| mtreinish | sdague: heh, they're delivering it in the nick of time | 15:39 |
| clarkb | I think one forecast called for a foot of rain in the seattle area. slightly less in portland | 15:39 |
| sdague | mtreinish: yep | 15:39 |
| fungi | clarkb: we're getting similar here. the parking area and back yard are now under standing water | 15:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove python26 jobs from all stackforge projects https://review.openstack.org/129434 | 15:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Re-add gate-sqlalchemy-migrate-unittests-26sa07 https://review.openstack.org/137382 | 15:40 |
| *** otter768 has joined #openstack-infra | 15:40 | |
| AJaeger_ | clarkb: does these two jobs do what you proposed? | 15:40 |
| AJaeger_ | sdague: that's a nice snowy view | 15:42 |
| ryanpetrello | clarkb: doesn't matter to me | 15:42 |
| ryanpetrello | if you all want them gone, that's fine w/ me | 15:42 |
| clarkb | ryanpetrello: ok we will remove them then | 15:42 |
| clarkb | AJaeger_: +2 on the first one | 15:42 |
| AJaeger_ | ryanpetrello: removed as part of https://review.openstack.org/129434 - thanks | 15:43 |
| clarkb | AJaeger_: for sqlalchemy migrate https://review.openstack.org/#/c/137382/1/zuul/layout.yaml should we run a normal py26 job there? | 15:43 |
| clarkb | AJaeger_: I guess it wasn't there before | 15:43 |
| sdague | fungi / clarkb - want to +A this - https://review.openstack.org/#/c/137123/ - current and last heat ptl are +1 on it | 15:43 |
| *** BharatK has joined #openstack-infra | 15:43 | |
| AJaeger_ | clarkb: I guess we should, let me add one. | 15:44 |
| *** yamamoto has joined #openstack-infra | 15:44 | |
| *** wenlock has joined #openstack-infra | 15:44 | |
| *** otter768 has quit IRC | 15:44 | |
| *** chandankumar has joined #openstack-infra | 15:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Re-add gate-sqlalchemy-migrate-unittests-26sa07 https://review.openstack.org/137382 | 15:45 |
| AJaeger_ | clarkb: added ^^^ | 15:45 |
| *** jcoufal has joined #openstack-infra | 15:46 | |
| clarkb | AJaeger_: I am going to see if I canfigure out where those py26 jobs were removed | 15:47 |
| AJaeger_ | jd__: if you split and update https://review.openstack.org/#/c/137321/, we can approve quickly | 15:48 |
| fungi | mbayer should probably also weigh in on it | 15:48 |
| AJaeger_ | clarkb: I removed them as part of my first patch | 15:48 |
| clarkb | AJaeger_: oh right cool | 15:48 |
| clarkb | fungi: I don't think he cares | 15:48 |
| *** bo_sh1 has joined #openstack-infra | 15:48 | |
| AJaeger_ | didn't remember ;( | 15:48 |
| * AJaeger_ updates commit message | 15:48 | |
| clarkb | alembic is the tool zzzeek cares about | 15:48 |
| clarkb | but that stack looks good to me now | 15:48 |
| jd__ | AJaeger_: doing | 15:48 |
| fungi | clarkb: good point. the sqlam core group is a grab-bag of volunteers anyway | 15:49 |
| clarkb | fungi: beat me to approving heat slow removal | 15:49 |
| clarkb | fungi: mostly just thinking it is an important piece of many projects particularly in the past as we migrate to alembic | 15:49 |
| clarkb | fungi: so trying to be good and run those tests | 15:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Re-add sqlalchemy-migrate python 2.6 jobs https://review.openstack.org/137382 | 15:50 |
| fungi | clarkb: yep, i'm totally +2 on keeping py26 jobs there | 15:50 |
| jd__ | AJaeger_: I don't think I need to split, now it's just factorizing python26 jobs AFAICS | 15:50 |
| AJaeger_ | jd__: you're adding one new job to the gate | 15:50 |
| AJaeger_ | don't you? | 15:51 |
| jd__ | AJaeger_: not anymore since you reverted | 15:51 |
| clarkb | ya the revert fixed the py26 and py33 issues | 15:51 |
| AJaeger_ | jd__: ah... Ok, I'll review carefully ;) | 15:51 |
| jd__ | AJaeger_: so this patch isn't needed anymore unless you want to use a python26 template | 15:51 |
| clarkb | AJaeger_: ok stack looks good now. As long as we don't make any drastic changes over the rest of the week we should be ready | 15:51 |
| openstackgerrit | Merged openstack-infra/project-config: remove heat-slow job and definition https://review.openstack.org/137123 | 15:51 |
| AJaeger_ | clarkb, sdague, fungi https://review.openstack.org/#/c/137348 could be approved now.... | 15:51 |
| *** JayJ has joined #openstack-infra | 15:52 | |
| AJaeger_ | clarkb: thanks for the reviews! | 15:52 |
| clarkb | jd__: the python26 template is a reasonable thing to have. might make it more readable that we actually want python26 on those projects too | 15:52 |
| jd__ | ok I'll do that clarkb | 15:52 |
| *** alla__ has quit IRC | 15:53 | |
| AJaeger_ | should we have a comment in front of the python26 job that these are deprecated? | 15:53 |
| yolanda | clarkb, mordred, do you have a sec to talk about nodepool upload problem? | 15:53 |
| clarkb | yolanda: sure | 15:53 |
| clarkb | AJaeger_: maybe? | 15:53 |
| yolanda | clarkb, so we detected a problem with our side | 15:53 |
| yolanda | realated to slow image uploads | 15:53 |
| clarkb | AJaeger_: and a note that they have until juno's end of life before removal? | 15:54 |
| yolanda | as our network is not so great, it can take over 2-3 hours to upload an image | 15:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python 2.6 for tooz https://review.openstack.org/137332 | 15:54 |
| yolanda | as it's using providermanager for the upload, it's blocking the rest of tasks | 15:54 |
| fungi | yolanda: oh, ew, that's no good | 15:54 |
| yolanda | we detected an outage today, nearly 3 hours without any new task being processed, until the image upload finished | 15:54 |
| yolanda | and i guess it has happened other times | 15:54 |
| clarkb | weren't we using a glance provider manager? | 15:55 |
| AJaeger_ | fungi: regarding https://review.openstack.org/#/c/129434 - since you have the -2 on it: Will you approve all changes once you're back on Monday? | 15:55 |
| fungi | AJaeger_: absolutely | 15:55 |
| AJaeger_ | thanks, fungi | 15:55 |
| yolanda | so my idea was to split that, create a "UploadManager" similar to the ProviderManager, but only performing image uploads | 15:55 |
| yolanda | what do you think? can you imagine a better approach? | 15:55 |
| fungi | AJaeger_: if you're around monday, ping me once i get online and i'll cram all those through first thing | 15:55 |
| clarkb | oh its a glanceclient in the existing provider managers gotcha | 15:55 |
| jeblair | yolanda: or just have the upload command execute directly instead of going into the task manager queue | 15:56 |
| *** pcrews has joined #openstack-infra | 15:56 | |
| AJaeger_ | fungi: ok, will try to remember ;) | 15:56 |
| fungi | jeblair: as part of the image build thread? | 15:56 |
| * AJaeger_ should be around | 15:56 | |
| jeblair | fungi: yeah, so it will block that thread but nothing else | 15:56 |
| yolanda | fungi, image build is another thread actually | 15:56 |
| yolanda | there are 2 steps: image build and image upload | 15:56 |
| clarkb | yolanda: I think nodepool tasks can run async | 15:56 |
| clarkb | yolanda: I would just do it that way | 15:56 |
| *** tykeal has left #openstack-infra | 15:57 | |
| clarkb | oh I defer to jeblair | 15:57 |
| yolanda | ok that approach is simpler | 15:58 |
| yolanda | image uploads in provider are fine if they take 5 min, but in our case is quite crazy | 15:58 |
| *** afazekas has quit IRC | 15:58 | |
| clarkb | also the spin out of image builder nodes would potentially alleviate this issue a bit if you can run those nodes closer to glance | 15:58 |
| clarkb | but it doesn't fix the problem | 15:58 |
| yolanda | cool, i'll file a change for you to review, is quite an urgent issue for us | 15:59 |
| jeblair | clarkb: good point | 15:59 |
| *** sarob has joined #openstack-infra | 16:00 | |
| yolanda | yes, our nodepool is not in the same network as glance atm | 16:00 |
| *** maishsk has quit IRC | 16:01 | |
| mtreinish | fungi: if you've got a sec: https://review.openstack.org/#/c/135493/ should be pretty quick | 16:01 |
| *** pblaho_ has joined #openstack-infra | 16:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Sort taskflow properly https://review.openstack.org/137348 | 16:02 |
| fungi | yolanda: nor ours... they're separated by a gulf of internets (nodepool in rax, glance in hpcloud) | 16:02 |
| dimsum__ | fungi: the 'owner' information in status.json finally kicked in (zuul restart?) - finally am able to filter the status page by name/email... thanks - https://review.openstack.org/#/c/137372/ | 16:02 |
| clarkb | fungi: we haven't really had issues with uploads though | 16:02 |
| clarkb | fungi: the entire process is faster than a single snapshot build | 16:03 |
| fungi | clarkb: agreed. we have a fair amount of bandwidth and smallish images | 16:03 |
| yolanda | clarkb, i guess is problem with our connectivity | 16:03 |
| *** dimsum__ is now known as dims | 16:03 | |
| fungi | dimsum__: yeah, it got restarted on monday | 16:03 |
| yolanda | and our images are over 5G | 16:03 |
| clarkb | yolanda: ya ours are just under 3GB | 16:03 |
| *** mikeit has quit IRC | 16:04 | |
| *** pblaho has quit IRC | 16:04 | |
| sdague | fungi: oh, in my list of patches that I think are ready, https://review.openstack.org/#/c/134283/ is the pagination for merge-check (clarkb found a real bug) | 16:04 |
| sdague | you had previously +2 | 16:04 |
| yolanda | our speed can vary from 20 min to 3 hours, depends on proxy really... i've done manual uploads so quick and i had to abort others | 16:04 |
| AJaeger_ | clarkb, fungi: I have still two patches to remove the docutils pin - https://review.openstack.org/128394 and https://review.openstack.org/128395 since weeks open. reviews for this housekeeping patches are welcome. | 16:04 |
| *** ryanpetrello has quit IRC | 16:06 | |
| mtreinish | fungi: thanks | 16:07 |
| *** rhe00 has left #openstack-infra | 16:08 | |
| *** jerryz has quit IRC | 16:08 | |
| *** bogdando has quit IRC | 16:09 | |
| fungi | sdague: i'm cool with 134283 but was hoping jeblair might have time to look it over once he wasn't travelling | 16:09 |
| *** mmaglana has joined #openstack-infra | 16:10 | |
| jeblair | yeah, let me look at that since i spent some time with gerrit pagination recently | 16:10 |
| *** bogdando has joined #openstack-infra | 16:11 | |
| openstackgerrit | Merged openstack-infra/system-config: Add support to log gearman client to filter on build-queue https://review.openstack.org/135493 | 16:12 |
| *** JayJ has quit IRC | 16:12 | |
| *** Ark has joined #openstack-infra | 16:12 | |
| *** JayJ has joined #openstack-infra | 16:13 | |
| *** patrickeast has joined #openstack-infra | 16:13 | |
| *** Ark is now known as Guest8638 | 16:13 | |
| *** Guest10529 has quit IRC | 16:13 | |
| *** MarkAtwood has joined #openstack-infra | 16:13 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Use a template for python26 jobs https://review.openstack.org/137321 | 16:14 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move pycadf in Oslo section https://review.openstack.org/137390 | 16:14 |
| *** mmaglana has quit IRC | 16:15 | |
| *** nfedotov has quit IRC | 16:15 | |
| clarkb | jeblair: will there be a gear change today with your stuff from yesterday? (I am mostly curious to see what you changed) | 16:16 |
| jeblair | clarkb: i hope so; will start on it in about 2 hours | 16:16 |
| *** kgiusti has quit IRC | 16:16 | |
| clarkb | awesome | 16:16 |
| *** pblaho_ is now known as pblaho | 16:16 | |
| *** zigo has quit IRC | 16:16 | |
| *** fabo has joined #openstack-infra | 16:17 | |
| fungi | i will try to pop back in at some point this evening to review gear improvements | 16:17 |
| *** JayJ has quit IRC | 16:17 | |
| *** JayJ has joined #openstack-infra | 16:18 | |
| AJaeger_ | jd__: you templatized all the stackforge jobs as well ;( | 16:19 |
| AJaeger_ | that conflicts with my other patches to remove them. | 16:19 |
| jd__ | AJaeger_: is it wrong? | 16:19 |
| jd__ | oh | 16:19 |
| jd__ | whoever goes first win! ;) | 16:19 |
| sdague | sounds good to me | 16:19 |
| AJaeger_ | jd__: see https://review.openstack.org/129434 | 16:19 |
| *** amuller has quit IRC | 16:20 | |
| *** kim__ has joined #openstack-infra | 16:20 | |
| kim__ | Hi, What cause to "Merge Failed." | 16:20 |
| AJaeger_ | jd__: I can rebase if you find two other cores to approve your patch ;) | 16:20 |
| clarkb | kim__: can you link an example? | 16:21 |
| *** shashankhegde has joined #openstack-infra | 16:21 | |
| *** JayJ has quit IRC | 16:22 | |
| *** mmaglana has joined #openstack-infra | 16:22 | |
| *** JayJ has joined #openstack-infra | 16:23 | |
| *** mmaglana has quit IRC | 16:23 | |
| jd__ | AJaeger_: 2 cores? I don't have this amount of money! ;) | 16:23 |
| kim__ | clarkb: I will describe the problem: When you run a new job of a new commit I noticce that there is a failed message in the gerrit: "Merge Failed." Even when the job passed or failed | 16:23 |
| clarkb | kim__: can you link to an example? | 16:24 |
| clarkb | kim__: there are several ways this can happen and iwthout a concrete example it is hard to sya what caused it | 16:24 |
| *** ildikov has quit IRC | 16:24 | |
| openstackgerrit | Merged openstack-infra/system-config: Add a security.openstack.org site for advisories https://review.openstack.org/133302 | 16:25 |
| openstackgerrit | Merged openstack-infra/system-config: Enable SSL in groups-dev.openstack.org https://review.openstack.org/135708 | 16:25 |
| kim__ | clarkb: I found one... for example in this commit there are some Merge failed" | 16:25 |
| kim__ | https://review.openstack.org/#/c/136861/ | 16:25 |
| clarkb | kim__: is "This change depends on a change that failed to merge" the message you are curious about or "Merge Failed" | 16:26 |
| *** zigo has joined #openstack-infra | 16:27 | |
| clarkb | kim__: for the first things it means that a parent change of this change was unable to merge. since this change depends on that parent it cannot merge either | 16:27 |
| clarkb | kim__: for the second it means there is a merge conflict that could not be automatically resolved. You should rebase on the latest version of the target branch and resolve these conflicts manually | 16:27 |
| *** sarob has quit IRC | 16:28 | |
| kim__ | https://review.openstack.org/#/c/136861/:q! | 16:28 |
| jeblair | sdague: lgtm thanks | 16:29 |
| *** gokrokve_ has joined #openstack-infra | 16:30 | |
| *** k4n0 has quit IRC | 16:30 | |
| kim__ | clarkb: I expect it to be automatically | 16:30 |
| *** gokrokve has quit IRC | 16:31 | |
| kim__ | clarkb: I think that the issue can be resolved with the manager parameter in layout.conf | 16:31 |
| *** ryanpetrello has joined #openstack-infra | 16:31 | |
| *** pblaho_ has joined #openstack-infra | 16:31 | |
| sdague | clarkb: ... I have a testr mystery if you have the ability to help deduce | 16:31 |
| clarkb | kim__: no, these are conflicts that are not able to be automatically resolved | 16:31 |
| kim__ | clarkb: It should be manager: dependentPipelineManager.. | 16:31 |
| clarkb | kim__: zuul attempts to do a git merge but runs into merge conflicts | 16:31 |
| clarkb | kim__: it is for the gate pipeline | 16:31 |
| sdague | nova functional tests died in a very odd way that I can't reproduce - https://jenkins07.openstack.org/job/gate-nova-tox-functional/281/consoleFull | 16:32 |
| openstackgerrit | Merged openstack-infra/zuul: implement gerrit pagination in simplequery https://review.openstack.org/134283 | 16:32 |
| kim__ | clarkb: So..Who should fix this? THe commiter? | 16:32 |
| clarkb | kim__: yes | 16:33 |
| kim__ | Thanks | 16:33 |
| *** jcoufal has quit IRC | 16:33 | |
| clarkb | kim__: the message left by jenkins indicates the steps that should be taken | 16:33 |
| *** JayJ has quit IRC | 16:34 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Make agent_ssh check job voting for IPA https://review.openstack.org/134436 | 16:34 |
| clarkb | sdague: the stderr thing about openssl.cnf seems pretty straightforward | 16:34 |
| clarkb | sdague: not sure this is a testr thing | 16:34 |
| sdague | clarkb: well, except, it doesn't happen consistently | 16:34 |
| sdague | and never locally | 16:34 |
| sdague | and never in past runs of that job | 16:35 |
| *** pblaho has quit IRC | 16:35 | |
| clarkb | sdague: the first thing I would do is grab the subunit log for that run | 16:35 |
| clarkb | sdague: do testr load < subunitlog | 16:35 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Make agent_ssh check job voting for IPA https://review.openstack.org/134436 | 16:35 |
| clarkb | then run testr run --analyze-isolation which does bisection to find conflicting tests | 16:35 |
| *** sarob has joined #openstack-infra | 16:35 | |
| *** packet has joined #openstack-infra | 16:35 | |
| *** shashankhegde has quit IRC | 16:36 | |
| sdague | hmmm... no subunit - http://logs.openstack.org/72/136772/7/gate/gate-nova-tox-functional/bd72a22/ | 16:36 |
| clarkb | your job isn't copying the file | 16:36 |
| *** packet has quit IRC | 16:37 | |
| *** kim__ has quit IRC | 16:38 | |
| clarkb | sdague: it tries to but the files are not there | 16:38 |
| clarkb | maybe a PWD thing? | 16:38 |
| *** shashankhegde has joined #openstack-infra | 16:39 | |
| *** asselin_ has joined #openstack-infra | 16:43 | |
| *** gokrokve_ has quit IRC | 16:45 | |
| *** ryanpetrello_ has joined #openstack-infra | 16:45 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager https://review.openstack.org/137401 | 16:47 |
| yolanda | just initial proposal, i need to test in my instances ^ | 16:48 |
| *** ryanpetrello has quit IRC | 16:48 | |
| *** ryanpetrello_ is now known as ryanpetrello | 16:48 | |
| *** bo_sh1 has quit IRC | 16:48 | |
| *** pblaho_ has quit IRC | 16:49 | |
| *** bo_sh has joined #openstack-infra | 16:49 | |
| dteselkin | Does anyone knows anything about 'refs-updated' event from gerrit stream-events? | 16:50 |
| dteselkin | It looks like sometimes gerrit sends *a lot* of such messages with 'failed' status. | 16:50 |
| dteselkin | And that kills my zuul :( | 16:51 |
| clarkb | dteselkin: what version of gerrit? | 16:51 |
| *** emagana has joined #openstack-infra | 16:51 | |
| jeblair | dteselkin: and i don't know what 'failed' means in this context: ref-updated happens when a change is merged or pushed to a ref (eg 'master' or some other branch); so i don't know why it would emit such an event in the case of any kind of failure | 16:52 |
| *** belmoreira has quit IRC | 16:52 | |
| *** Masahiro has joined #openstack-infra | 16:52 | |
| clarkb | dstufft: https://review.openstack.org/Documentation/json.html#refUpdate doesn't have a status field | 16:52 |
| jeblair | tabfail | 16:53 |
| *** ildikov has joined #openstack-infra | 16:53 | |
| *** viktors is now known as viktors|afk | 16:53 | |
| clarkb | gah | 16:54 |
| dteselkin | Events like http://paste.openstack.org/show/138928/ | 16:54 |
| clarkb | dteselkin: ^ | 16:54 |
| dteselkin | Oh, sorry | 16:54 |
| jeblair | oh ref-replicated | 16:54 |
| dteselkin | ref-replicated | 16:54 |
| *** talluri has joined #openstack-infra | 16:54 | |
| clarkb | thats new hence my gerrit version question :) | 16:55 |
| jeblair | dteselkin: i think that zuul should ignore that since it hasn't been programmed to respond to it; you might get a warning in the error log but should be harmless. what affect is it having on your zuul? | 16:55 |
| *** changbl has joined #openstack-infra | 16:56 | |
| jeblair | i'm guessing you get "Received unrecognized event type '%s' from Gerrit." a lot | 16:56 |
| *** Masahiro has quit IRC | 16:56 | |
| dteselkin | jeblair, the problem is that there are *thousands* of such messages, and they flood internal queue | 16:57 |
| dstufft | clarkb: lol I was really confused there for a moment ;) | 16:57 |
| jeblair | dteselkin: ah, so they get enqueued into the trigger queue and take too much time to process? | 16:57 |
| dteselkin | Yep, there are some, but then is just stops further processing | 16:57 |
| *** talluri_ has joined #openstack-infra | 16:58 | |
| dteselkin | Yes, they are received by GerritWatcher | 16:58 |
| dteselkin | But at some point getEvent stops fetching them | 16:58 |
| jeblair | dteselkin: that sounds like a bug :/ we've worked through thousands of events before without problem (it should work through them fairly quickly) | 16:59 |
| dteselkin | Gerrit version 2.9 | 16:59 |
| jeblair | dteselkin: regardless, we have two ways to proceed -- figure out why it gets stuck, or ignore those events | 16:59 |
| jeblair | i lean toward the latter since they don't seem particularly useful | 17:00 |
| *** talluri has quit IRC | 17:00 | |
| jeblair | (and then if your zuul still gets stuck, we'll look into that subsequently) | 17:00 |
| clarkb | in theory you might trigger jobs based on those events | 17:00 |
| clarkb | but it doesn't seem likely | 17:00 |
| dteselkin | I agree that it's better to ignorre them, as early as possible | 17:00 |
| clarkb | thinking of something like "Oh mirror is up to date let me go run thing against the mirror) | 17:00 |
| *** asselin_ has quit IRC | 17:01 | |
| dteselkin | Is it a good way to add a kind of filter to GerritWatcher? | 17:01 |
| jeblair | clarkb: i suppose we could ignore them now and add them back in later if needed? | 17:01 |
| clarkb | jeblair: yup that works too | 17:02 |
| fungi | perhaps zuul grows a configurable blacklist of event types for efficiency, which defaults to things we know to be potentially noisy and of little value | 17:02 |
| fungi | if it turns out people actually want the flexibility to tweak that | 17:02 |
| jeblair | clarkb: (and if we add it back, maybe with a few changes to make it more efficient to receive thousands of events with little impact :) | 17:02 |
| *** mmaglana has joined #openstack-infra | 17:02 | |
| clarkb | ++ | 17:03 |
| jeblair | there's already something that's nearly a whitelist in there... | 17:03 |
| clarkb | in the pipeline managers | 17:03 |
| jeblair | there's a list of events and mapping to the user attribute (since it changes names sometimes) | 17:03 |
| jeblair | no i meant in the gerrit trigger itself | 17:03 |
| clarkb | we list the things we are interested in | 17:04 |
| *** arxcruz has quit IRC | 17:04 | |
| clarkb | jeblair: right I am thinking that we could map managers onto that other mapping and ignore everything else | 17:04 |
| *** dizquierdo has quit IRC | 17:04 | |
| jeblair | clarkb: oh, thinking of maybe taking the list of interested events from the config file and generate a whitelist based on that? | 17:04 |
| clarkb | up | 17:04 |
| clarkb | *yup | 17:04 |
| *** ashaeron has quit IRC | 17:04 | |
| jeblair | that could work; my idea was to take the existing list of events that the trigger knows about and instead of logging an error and continuing if it hits a new one, just ignore it | 17:05 |
| mordred | ++ | 17:05 |
| jeblair | a third option is to make a blacklist | 17:05 |
| dteselkin | http://paste.openstack.org/show/138941/ | 17:05 |
| jeblair | so we're still partially future-compatible, but ignore "annoying" events like ref-replicated | 17:05 |
| dteselkin | Two lines from my log shows how fast events are coming | 17:05 |
| *** ChuckC_ has quit IRC | 17:05 | |
| *** ChuckC has joined #openstack-infra | 17:06 | |
| *** chandankumar has quit IRC | 17:06 | |
| jeblair | dteselkin: ah! tens of thousands! we've never had a queue that deep, so that's pretty much untested | 17:06 |
| dteselkin | That because it's a garbage :) | 17:07 |
| *** MaxV has quit IRC | 17:07 | |
| jeblair | dteselkin: did you add that log message? | 17:08 |
| dteselkin | http://paste.openstack.org/show/138942/ | 17:08 |
| dteselkin | An updated version ^^^ | 17:08 |
| jeblair | (i can't find it in the source) | 17:08 |
| *** patrickeast has quit IRC | 17:08 | |
| dteselkin | Yep, I added that to addEvent | 17:08 |
| *** shashankhegde has quit IRC | 17:08 | |
| *** baoli has quit IRC | 17:08 | |
| dteselkin | I'll show | 17:09 |
| jeblair | ok, hrm. i'd rather not add a filter to that class; i think it's the wrong level of abstraction (that actually should be in gerritlib) | 17:09 |
| jeblair | well | 17:10 |
| dteselkin | http://paste.openstack.org/show/138943/ | 17:10 |
| jeblair | i guess it could be an argument to the constructor | 17:10 |
| fungi | i like clarkb's suggestion | 17:10 |
| fungi | dynamism based on expressed interest in the config makes for the most user-friendly option i think | 17:10 |
| clarkb | I thought about the gerritism of not sending the +1 in the event if the value doesn't change | 17:10 |
| clarkb | you know how we can fix that? have jenkins reset votes when testing starts .... | 17:11 |
| fungi | clarkb: yeah except | 17:11 |
| fungi | clarkb: that leaves a comment, so now we're back to "starting check jobs" | 17:11 |
| jeblair | yeah, so if we do that, we'll need to pass the list all the way into the lib/gerrit class | 17:11 |
| clarkb | fungi: ya | 17:11 |
| clarkb | fungi: I don't like the solution but it should work and may explain why it was working until recently | 17:12 |
| *** ryanpetrello has quit IRC | 17:12 | |
| fungi | clarkb: alternatively, trigger the gate pipeline on any voteless comment left by jenkins and then rely on it querying the change to see if it already has a +1 present | 17:12 |
| fungi | that way it will still only get enqueued under the correct circumstances i think | 17:13 |
| jeblair | dteselkin: are you interested in hacking on a fix? | 17:13 |
| dteselkin | jeblair, yes, definitely | 17:13 |
| *** yamamoto has quit IRC | 17:13 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move pycadf in Oslo section https://review.openstack.org/137390 | 17:14 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Use a template for python26 jobs https://review.openstack.org/137321 | 17:14 |
| clarkb | fungi: ya I think that will require zuul code changes though | 17:14 |
| jeblair | dteselkin: it sounds like we think the best approach is to have trigger/gerrit.py calculate the list of interesting events in the postConfig method, and then somehow pass that into the gerritWatcher in lib/gerrit | 17:15 |
| jeblair | dteselkin: note that will need to update while running to handle live reconfiguration | 17:15 |
| dteselkin | And check it somewhere before https://github.com/openstack-infra/zuul/blob/master/zuul/lib/gerrit.py#L42 ? | 17:15 |
| jeblair | dteselkin: yep | 17:15 |
| *** sergsh is now known as sergsh_away | 17:15 | |
| *** mpaolino has quit IRC | 17:16 | |
| dteselkin | Take whitelist from config ? | 17:16 |
| dteselkin | Or hardcode them for now | 17:17 |
| jeblair | dteselkin: whitelist from config would be best i think | 17:17 |
| jeblair | dteselkin: the postConfig method gets called when the config changes | 17:17 |
| jeblair | dteselkin: so you can inspect it then to generate the whitelist | 17:17 |
| dteselkin | Ok, thanks. Didn't know that yet. | 17:18 |
| jeblair | dteselkin: if you want a quick fix for your zuul, of course, adding something to ignore that event to https://github.com/openstack-infra/zuul/blob/master/zuul/lib/gerrit.py#L42 is probably the thing :) | 17:18 |
| *** dizquierdo has joined #openstack-infra | 17:18 | |
| dteselkin | jeblair, well, it's better to have universal solution than my own fix :) | 17:19 |
| *** bo_sh1 has joined #openstack-infra | 17:19 | |
| jeblair | dteselkin: definitely; just didn't know how bad your fire was :) | 17:19 |
| dteselkin | :) | 17:19 |
| dteselkin | Ok, thanks a lot for help! | 17:19 |
| *** mudassirlatif has joined #openstack-infra | 17:20 | |
| jeblair | dteselkin: thanks for yours! and running into this before we did! :) | 17:20 |
| *** vhoward has left #openstack-infra | 17:20 | |
| *** bo_sh has quit IRC | 17:21 | |
| *** Longgeek has quit IRC | 17:24 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager https://review.openstack.org/137401 | 17:26 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: the src-neutron-icehouse jobs should only run smoke tests https://review.openstack.org/137419 | 17:27 |
| *** rediskin has joined #openstack-infra | 17:27 | |
| *** atan8 has joined #openstack-infra | 17:27 | |
| *** e0ne has quit IRC | 17:28 | |
| *** marcusvrn has quit IRC | 17:29 | |
| *** chandankumar has joined #openstack-infra | 17:31 | |
| pleia2 | good morning | 17:33 |
| *** Sukhdev has quit IRC | 17:34 | |
| *** zul has quit IRC | 17:34 | |
| *** HeOS has quit IRC | 17:34 | |
| *** andreykurilin_ has quit IRC | 17:34 | |
| clarkb | jogo: so you have working migrations now on those multinode tests? | 17:35 |
| *** zul has joined #openstack-infra | 17:35 | |
| *** ryanpetrello has joined #openstack-infra | 17:35 | |
| *** jpich has quit IRC | 17:35 | |
| *** AJaeger_ has quit IRC | 17:36 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add a ci-sandbox project. https://review.openstack.org/137242 | 17:36 |
| *** mudassirlatif has quit IRC | 17:38 | |
| *** kgiusti has joined #openstack-infra | 17:38 | |
| *** chandankumar has quit IRC | 17:39 | |
| *** gokrokve has joined #openstack-infra | 17:39 | |
| *** baoli has joined #openstack-infra | 17:40 | |
| *** otter768 has joined #openstack-infra | 17:41 | |
| *** bdpayne has joined #openstack-infra | 17:41 | |
| *** unicell1 has quit IRC | 17:43 | |
| *** dannywilson has joined #openstack-infra | 17:44 | |
| *** dannywil_ has joined #openstack-infra | 17:44 | |
| *** otter768 has quit IRC | 17:45 | |
| *** talluri_ has quit IRC | 17:46 | |
| *** tonytan4ever has quit IRC | 17:47 | |
| *** dannywilson has quit IRC | 17:48 | |
| clarkb | sdague: ok, your job uses run-tox.sh which does none of the test runner specific actions to get things like subunit logs | 17:48 |
| * clarkb writes a hacky thing that we can argue over | 17:50 | |
| *** jistr has quit IRC | 17:56 | |
| *** derekh has quit IRC | 17:58 | |
| *** markus_z has quit IRC | 17:58 | |
| *** patrickeast has joined #openstack-infra | 17:59 | |
| *** talluri has joined #openstack-infra | 18:00 | |
| *** nadya_ has quit IRC | 18:00 | |
| *** ryanpetrello_ has joined #openstack-infra | 18:01 | |
| *** dannywil_ has quit IRC | 18:03 | |
| jroll | mordred, clarkb: for rax dib stuff, were you planning on using cloud-init to configure the network or? | 18:03 |
| clarkb | I would assume so but am happy to let mordred figure out the best options | 18:03 |
| clarkb | personally I think dhcp should be used | 18:03 |
| clarkb | since it is 30 years old and was created to solve this problem | 18:03 |
| *** harlowja_away is now known as harlowja | 18:04 | |
| jroll | right, assuming people can't be convinced to make that work :/ | 18:04 |
| *** ryanpetrello has quit IRC | 18:04 | |
| *** ryanpetrello_ is now known as ryanpetrello | 18:04 | |
| *** BharatK has quit IRC | 18:04 | |
| jroll | I'm working this from the configdrive route | 18:05 |
| clarkb | if configdrive then cloud init is what I would expect | 18:06 |
| *** tonytan4ever has joined #openstack-infra | 18:06 | |
| jroll | yeah, ok | 18:06 |
| jroll | thanks | 18:07 |
| clarkb | dhcp has the benefit of being pretty well understood everywhere. I know how to debug it but debugging cloud init and configdrive sounds like pain | 18:07 |
| jroll | yeah, agree | 18:08 |
| fungi | unfortunately we'll likely need to shoehorn in our own cloud-init so that we don't preinstall its python dependencies as distro packages | 18:09 |
| jroll | I just think it would be much harder to convince someone to change our network that much, then to dump some data in an iso :P | 18:09 |
| jroll | fungi: are we finally running things in virtualenvs? | 18:10 |
| clarkb | jroll: I have no idea what your network looks like but usually you only need like 2 firewall rules and a dhcp server | 18:10 |
| *** dizquierdo has quit IRC | 18:10 | |
| fungi | clarkb: or nova network or neutron which provide you targeted dnsmasq services | 18:10 |
| jroll | clarkb: across many cells across 7 regions/datacenters... | 18:11 |
| *** bo_sh1 has quit IRC | 18:11 | |
| *** dkliban is now known as dkliban_brb | 18:11 | |
| fungi | jroll: no, pip installing some things system-wide which conflict with other distro-packaged python-based tools pulling in python libs from distro packages | 18:11 |
| jroll | clarkb: and I don't know if there's other things that for whatever reason depend on instances using statoc configs | 18:11 |
| clarkb | jroll: so I did the build entire datacenters without dhcp thing for a couple years | 18:11 |
| jroll | fungi: ah, right | 18:11 |
| clarkb | jroll: I was also the person stuck with fixing it | 18:11 |
| clarkb | jroll: and it was probably the most worthwhile thing I ever did in that job | 18:12 |
| *** mmedvede has quit IRC | 18:12 | |
| jroll | fungi: we have some builders for custom cloud-init packages, you might be able to fork them and do something similar https://github.com/racker/cloud-init-docker-build | 18:12 |
| *** e0ne has joined #openstack-infra | 18:13 | |
| jroll | clarkb: ha, I don't disagree that using dhcp may be better, I just would have no idea where to start talking to people here about doing that :) | 18:13 |
| JayF | Note that the repos ^ that points to has an OnMetal downstream patch in it | 18:13 |
| *** mmedvede has joined #openstack-infra | 18:13 | |
| *** dtantsur is now known as dtantsur|afk | 18:13 | |
| JayF | but it's easy to remove and make it stock again | 18:13 |
| mordred | no | 18:13 |
| mordred | no cloud init | 18:13 |
| mordred | I have NO intent on using cloud init | 18:14 |
| jroll | mordred: how will you be configuring networking, then | 18:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Make run-tox a super set of run-unittests https://review.openstack.org/137433 | 18:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Switch to using run-tox.sh everywhere. https://review.openstack.org/137434 | 18:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Delete run-unittests.sh https://review.openstack.org/137435 | 18:14 |
| fungi | mordred: initscript that adjusts interface configs early in boot? | 18:14 |
| clarkb | sdague: ^ that will fix you. its a bit more terrible than I like but it fixes this permanently | 18:14 |
| *** jedimike has quit IRC | 18:14 | |
| JayF | Ugh, I don't love cloud-init but reinventing the wheel in small scripts over and over isn't always better :( | 18:14 |
| jroll | mordred: better question: do you want the networking data in JSON or debian-interfaces-file formats? | 18:15 |
| jroll | JayF: though they only need to support one or two distros | 18:15 |
| jroll | so not the worst idea | 18:15 |
| *** baoli has quit IRC | 18:15 | |
| clarkb | mordred: so I question that | 18:15 |
| jroll | mordred: (or both formats) | 18:15 |
| clarkb | because cloud init solves this problem | 18:15 |
| clarkb | and is widely used etc | 18:15 |
| mordred | but I'm on a phone ... will word vomit later about plan | 18:15 |
| fungi | though i will agree that cloud-init is no fun to debug | 18:15 |
| mordred | don't care ...json is fine. | 18:16 |
| jroll | ok | 18:16 |
| jroll | thanks, helpful | 18:16 |
| fungi | flemel json | 18:16 |
| fungi | consume yaml | 18:16 |
| jroll | flemel? | 18:16 |
| mordred | clarkb: will argue about it later | 18:17 |
| clarkb | mordred: I don't know that I need to argue but its the same thing with dhcp | 18:17 |
| clarkb | mordred: we need to stop being silly | 18:17 |
| fungi | jroll: sorry, flamel | 18:17 |
| mordred | agree | 18:17 |
| clarkb | and use standard tooling that solves these problem | 18:17 |
| clarkb | s | 18:17 |
| fungi | jroll: https://pypi.python.org/pypi/flamel | 18:17 |
| jroll | oh, neat | 18:17 |
| fungi | jroll: it's a SpamapS special | 18:18 |
| *** e0ne has quit IRC | 18:18 | |
| mordred | I want dhcp, but can't have nice things | 18:18 |
| jroll | I'm going to provide y'all with json, you can flamel it if you like :P | 18:18 |
| jeblair | https://pypi.python.org/pypi/explode/1.0.5 is alo cool | 18:18 |
| *** harlowja has quit IRC | 18:18 | |
| mordred | woot | 18:18 |
| fungi | jeblair: good point | 18:18 |
| jeblair | (and potentially useful here) | 18:18 |
| jroll | mordred: if you want a preview of the format or whatever, spin up an onmetal server, we do this today | 18:18 |
| mordred | cool | 18:19 |
| jroll | it's in vendor_data | 18:19 |
| *** harlowja has joined #openstack-infra | 18:19 | |
| mordred | I'll look at it in a few, and thanks!! | 18:20 |
| jroll | the nova spec to match is https://review.openstack.org/#/c/85673/ | 18:20 |
| jroll | np :) | 18:20 |
| *** e0ne has joined #openstack-infra | 18:20 | |
| *** emagana has quit IRC | 18:23 | |
| *** emagana has joined #openstack-infra | 18:23 | |
| openstackgerrit | Darragh Bailey proposed openstack-dev/hacking: Add import check for try/except wrapped imports https://review.openstack.org/134288 | 18:24 |
| mtreinish | mordred: https://github.com/emonty/explode/pull/3 | 18:24 |
| jeblair | clarkb, fungi: i have working code that is as fast as the current system. only thing we lose is protection against starvation, but i'm not terribly worried about that | 18:24 |
| SpamapS | ugh networks that resist DHCP are like drivers that won't use blinkers IMO .. wasting everyone's time and usually dangerous | 18:25 |
| jeblair | clarkb, fungi: (we don't have that now) | 18:25 |
| clarkb | jeblair: this is asyncio? | 18:25 |
| jeblair | clarkb: yeah | 18:25 |
| clarkb | SpamapS+ | 18:25 |
| clarkb | er ++ | 18:25 |
| clarkb | jeblair: ya I was fine with it in the original change/patchsets | 18:25 |
| mordred | mtreinish: merged | 18:25 |
| clarkb | jeblair: I think it should be ok | 18:25 |
| fungi | jeblair: awesome. sounds good enough to me | 18:25 |
| jeblair | got some cleanup to do, should be able to push it up soon | 18:25 |
| clarkb | SpamapS: the best part is when you have humans manually configuring nodes and you get conflicting IPs or wrong masks or wrong gateways | 18:26 |
| mtreinish | mordred: you might want to also push a release with it | 18:26 |
| clarkb | SpamapS: debugging that is hard | 18:26 |
| SpamapS | clarkb: yeah it's just ridiculous | 18:26 |
| mordred | mtreinish: that's harder to do from my phone | 18:26 |
| SpamapS | And hell we have this whole service that gives you an API for controlling DHCP we call Neutron. There's just no reason to resist DHCP in this case other than "I had a bad experience one time somewhere" | 18:27 |
| clarkb | mordred: you need ubuntu phone so you can push git tags without crazyness off your phone | 18:27 |
| *** mpavlase has quit IRC | 18:27 | |
| mtreinish | mordred: heh, I have faith :) | 18:27 |
| SpamapS | But yeah, can't have nice things. :-P | 18:27 |
| mordred | clarkb: uhm .... | 18:27 |
| clarkb | mordred: don't take me seriously though | 18:27 |
| *** tonytan4ever has quit IRC | 18:27 | |
| clarkb | however do they have that working on N5s yet? | 18:28 |
| jroll | SpamapS: it's a process, we might get there eventually | 18:28 |
| mtreinish | clarkb: I do things like that with connectbot and logging into one of my servers | 18:28 |
| cody-somerville | Could I bother from fine folks for review on https://review.openstack.org/#/c/137401/ please? | 18:28 |
| *** emagana has quit IRC | 18:28 | |
| jroll | SpamapS: also, dnsmasq explicitly says "for small networks" on their home page... I wouldn't run a whole datacenter on it | 18:28 |
| clarkb | jroll: your neutron networks are typically not large | 18:29 |
| morganfainberg | jroll: where is your sense of adventure? :P | 18:29 |
| clarkb | and you run a dnsmasq for portions of your deployment iirc | 18:29 |
| clarkb | like per hypervisor or some such | 18:29 |
| *** jp_at_hp has quit IRC | 18:29 | |
| jroll | clarkb: I'd imagine it would be dhcp for a whole cell | 18:29 |
| *** e0ne has quit IRC | 18:29 | |
| clarkb | cody-somerville: yolanda specifically said its not ready | 18:29 |
| jroll | and I guess it depends on your definition of large :P | 18:30 |
| clarkb | cody-somerville: has that changed? | 18:30 |
| cody-somerville | yolanda: ^^ | 18:30 |
| yolanda | clarkb, sorry, yes, it's ready now | 18:30 |
| yolanda | i uploaded patchset 2 | 18:30 |
| *** ryanpetrello_ has joined #openstack-infra | 18:30 | |
| SpamapS | jroll: The source of that information _MAY_ have been used to many thousands of nodes as a small network. I'd be interested to hear what is "small" to them. :) | 18:30 |
| *** nadya_ has joined #openstack-infra | 18:30 | |
| jroll | SpamapS: right :) | 18:31 |
| SpamapS | and seriously, this is just push vs. pull all over again. :) | 18:31 |
| jroll | I'm not trying to defend our lack of dhcp, btw :) | 18:31 |
| SpamapS | it's the constant pendulum of distributed computing.. "Push is robust, and decentralized.." "Pull is responsive and self-healing." | 18:31 |
| SpamapS | jroll: yeah I can tell that you're just trying to educate us with the realities that you're faced with, and I appreciate that very much. | 18:32 |
| fungi | yolanda: i don't suppose we have unit test coverage over that operation? | 18:32 |
| clarkb | fungi: we do for everything until where the comment says skip if testing | 18:33 |
| SpamapS | clarkb: btw, what part of cloud-init handles networking? Having trouble finding it. | 18:33 |
| jroll | SpamapS: :) | 18:33 |
| *** ryanpetrello has quit IRC | 18:33 | |
| *** ryanpetrello_ is now known as ryanpetrello | 18:33 | |
| yolanda | fungi, there is a test, but it doesn't arrive to the glance upload, it assumes fake-image and skips | 18:33 |
| SpamapS | oh I think I just did find it actually | 18:33 |
| *** wuhg has quit IRC | 18:33 | |
| SpamapS | and it looks like it only speaks /e/n/i format? | 18:34 |
| *** emagana has joined #openstack-infra | 18:34 | |
| *** pradk has joined #openstack-infra | 18:35 | |
| fungi | yolanda: right, seeing that. i think i'm okay with the change as it seems unlikely to break anything besides dib uploads anyway. i'm assuming you've functionally tested this, and if it turns out to be broken for us we'll spot it pretty quickly | 18:35 |
| yolanda | fungi, yes, i've tested that from my side, it uploaded image as expected | 18:36 |
| fungi | most of it's just cut and paste from the provider manager to the updater | 18:36 |
| fungi | +2 | 18:36 |
| clarkb | jeblair: did you want to review before I approve? | 18:36 |
| yolanda | yes, mostly to move that outer of taskmanager to avoid the blocking | 18:36 |
| clarkb | I am +2 going to give jeblair a chance to look since he suggested the fix | 18:39 |
| clarkb | but can approve it in a bit | 18:39 |
| *** nadya_ has quit IRC | 18:39 | |
| SpamapS | clarkb: ok n/m I found it.. so 'network_config' gets set to /e/n/i format. Hm. | 18:39 |
| *** andreykurilin_ has joined #openstack-infra | 18:39 | |
| clarkb | does that work on rhel? | 18:39 |
| *** e0ne has joined #openstack-infra | 18:40 | |
| jroll | well | 18:40 |
| jroll | network config is parsed from debian-interfaces format | 18:41 |
| *** Masahiro has joined #openstack-infra | 18:41 | |
| jroll | and written as whatever format the OS needs | 18:41 |
| clarkb | oh its the input side that is /e/n/i | 18:41 |
| *** harlowja_ has joined #openstack-infra | 18:41 | |
| jroll | yeah, it's absolutely insane | 18:41 |
| jroll | we have some patches to read from a json format a la https://review.openstack.org/#/c/85673/ | 18:41 |
| *** talluri has quit IRC | 18:42 | |
| *** talluri has joined #openstack-infra | 18:42 | |
| * jroll looks | 18:42 | |
| SpamapS | clarkb: yes it works on rhel | 18:43 |
| SpamapS | Funny I wonder if dprince ever saw this code before he wrote os-net-config. :) | 18:43 |
| *** ivar-lazzaro has joined #openstack-infra | 18:43 | |
| *** changbl has quit IRC | 18:44 | |
| SpamapS | jroll: for your entertainment.. http://git.openstack.org/cgit/openstack/os-net-config | 18:44 |
| jroll | see e.g. https://github.com/jayofdoom/cloud-init-fedora-pkg/blob/master/cloud-init-0.7.5-onmetal-configdrive.patch#L423 | 18:44 |
| *** e0ne has quit IRC | 18:44 | |
| clarkb | wow we have >1 thing to deal with no dhcp | 18:45 |
| *** harlowja has quit IRC | 18:45 | |
| JayF | SpamapS: I mean, I appreciate that tripleo has a custom thing to support that, but cloud-init is what most folks use to do tenant configuration | 18:45 |
| jroll | SpamapS: ha, interesting | 18:45 |
| JayF | and I don't think I'd want to add that as a dep on cloud-init | 18:45 |
| fungi | our project has become so large that we're reinventing the same wheels multiple times | 18:45 |
| *** rushiagr is now known as rushiagr_away | 18:45 | |
| JayF | fungi: I hate reinvented wheels. They never seem to be quite as round. | 18:45 |
| clarkb | this might be a good time to go get breakfast | 18:46 |
| * jroll looks at the clock | 18:46 | |
| *** Masahiro has quit IRC | 18:46 | |
| jroll | sounds like brunch | 18:46 |
| * jroll passes clarkb a bloody mary | 18:46 | |
| *** dannywilson has joined #openstack-infra | 18:48 | |
| *** dannywilson has joined #openstack-infra | 18:48 | |
| *** dkliban_brb is now known as dkliban | 18:50 | |
| *** Ryan_Lane has joined #openstack-infra | 18:50 | |
| *** ZZelle_ has joined #openstack-infra | 18:51 | |
| SpamapS | clarkb: actually os-net-config is not to deal with no dhcp | 18:52 |
| SpamapS | clarkb: it is to deal with complex device configuration. Bridges, bonding, etc. | 18:52 |
| jroll | SpamapS: so wy not fix cloud-init to do that? | 18:52 |
| SpamapS | jroll: GPLv3 and CLA? | 18:52 |
| fungi | SpamapS: clearly you should have expanded the dhcp spec with additional option codes for those ;) | 18:53 |
| SpamapS | jroll: thats certainly why _I_ am prohibited from working on it. :) | 18:53 |
| jroll | SpamapS: I mean, GPL sucks, but is that enough reason to reinvent the wheel? | 18:53 |
| * fungi wonders where the "gpl sucks" is coming from | 18:53 | |
| jroll | at any rate, we did it for you :P | 18:53 |
| SpamapS | GPLv3 is explicitly forbidden for me without high level mgmt approval because of the patent grant clause. | 18:53 |
| jroll | fungi: s/sucks/not great/ | 18:53 |
| jroll | ah | 18:53 |
| jroll | or not preferred, I should say | 18:54 |
| SpamapS | GPL is fine | 18:54 |
| fungi | (i think gplv3 is plenty great for a lot of things, maybe affero clause gpl variants less so) | 18:54 |
| SpamapS | GPLv3 just has thorny bits for large orgs. | 18:54 |
| SpamapS | also | 18:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Use non-blocking IO in server https://review.openstack.org/128754 | 18:55 |
| SpamapS | Heat makes it kind of weird to inject extra cloud-init stuff. | 18:55 |
| SpamapS | not impossible, but kind of weird. | 18:55 |
| jroll | mmm | 18:55 |
| jroll | we're working on that, too :P | 18:55 |
| jroll | well | 18:55 |
| SpamapS | Also cloud-init is hard to update | 18:55 |
| jroll | we're working on making nova do that, based on what neutron tells it about | 18:55 |
| jeblair | SpamapS: i'm going to disagree with you about GPLv3 having thorny bits for large orgs :) | 18:55 |
| jeblair | SpamapS: and i'm not sure how much you want to get into hp politics here, but it's actually really easy for you to work on gplv3 stuff | 18:56 |
| SpamapS | jeblair: large orgs that have a giant trove of patents to cling tightly to? | 18:56 |
| jeblair | SpamapS: yep. send a mail to fontana if you have any questions | 18:56 |
| SpamapS | jeblair: I have already tried once with Ansible and been told explicitly that I am not to patch it. | 18:56 |
| jeblair | SpamapS: i had a long chat with him recently and things are not as bad as we have been told | 18:56 |
| jeblair | SpamapS: when? | 18:57 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add debootstrap to the needed packages for dib https://review.openstack.org/124077 | 18:57 |
| SpamapS | jeblair: thats good to hear. Perhaps OSRB was just rubber stamping my Ansible stuff. :-P | 18:57 |
| SpamapS | jeblair: August? | 18:57 |
| * SlickNik surfaces on account of SpamapS' colorful vocabulary | 18:57 | |
| jeblair | SpamapS: yeah, that's like forever ago. :) an email to fontana on this will be productive. | 18:57 |
| SpamapS | hahaa nice | 18:58 |
| clarkb | jeblair: oh wow the diff there is pretty large | 18:58 |
| clarkb | jeblair: do you want to review https://review.openstack.org/#/c/137401/ before Iapprove it? | 18:58 |
| SpamapS | Ok time to go expose myself to the ball of fire in the sky for a little while. | 18:58 |
| fungi | tbm heads up the hp osrb right? | 18:58 |
| *** yamahata has quit IRC | 18:59 | |
| jeblair | tbm? | 18:59 |
| fungi | martin michlmayr | 18:59 |
| jeblair | clarkb: it's 100 lines less than the last patchset (30% less!) | 18:59 |
| clarkb | jeblair: :) | 18:59 |
| clarkb | I diffed against the old patchset | 19:00 |
| clarkb | and that diff is large maybe I hsould just read it from scratch | 19:00 |
| fungi | he's tbm in #debian-devel | 19:00 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 19:01 | |
| jeblair | clarkb: oh, yeah, so much got removed. if you want the quick walkthrough of the inter-patch diff it's: don't use an instance variable in the read method, git rid of all the threads in the server, handle all server io in the poll loop | 19:01 |
| *** thingee has quit IRC | 19:02 | |
| *** reed has quit IRC | 19:02 | |
| clarkb | jeblair: any interest in throwing that on nodepool-dev.o.o then possibly cutting over to it before tomorrow? | 19:02 |
| clarkb | probably not a great idea considering holiday but maybe no one will be around to notice if it breaks again | 19:02 |
| *** teran has joined #openstack-infra | 19:03 | |
| *** talluri has quit IRC | 19:04 | |
| *** thingee has joined #openstack-infra | 19:04 | |
| jeblair | clarkb: we could do that | 19:05 |
| clarkb | jeblair: looking at the read method you just set a reference to the instance variable and use that? and that helped that much that is interesting | 19:06 |
| *** dmsimard is now known as dmsimard_away | 19:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Use non-blocking IO in server https://review.openstack.org/128754 | 19:06 |
| jeblair | clarkb: 25x improvement. i love it when you have to think like a c programmer to work in python :/ | 19:07 |
| clarkb | ya I guess that removes a couple layers of indirection for access | 19:07 |
| jeblair | (latest ps is pep8 fix) | 19:08 |
| clarkb | jeblair: when you set input_buffer to b'' why not use packet = b'' instead? | 19:08 |
| clarkb | oh wait that doesn't do the right thing derp | 19:08 |
| clarkb | because python | 19:08 |
| jeblair | yeah, think in pointers. :) | 19:08 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/project-config: Create project for puppet-zanata module https://review.openstack.org/137448 | 19:09 |
| clarkb | jeblair: so how does that handle the case where you get the EAGAIN error when reading a normal packet | 19:09 |
| clarkb | jeblair: it appears that it would be dropped on the floor? | 19:09 |
| jeblair | clarkb: it's saved in self.input_buffer | 19:10 |
| clarkb | += does the right thing? | 19:10 |
| * clarkb tests | 19:10 | |
| clarkb | my test says it doesn't | 19:11 |
| clarkb | s1 = 'foo' ; s2 = s1 ; s2 += 'bar' ; print s1 ; print s2 | 19:11 |
| jeblair | grumble | 19:11 |
| clarkb | because strings are immutable you get back a new reference from += | 19:11 |
| openstackgerrit | Merged openstack-infra/storyboard: Add details to event subscription https://review.openstack.org/136364 | 19:12 |
| jeblair | right; this briefly did a stint as a list (where that works) | 19:12 |
| clarkb | jeblair: I think if you do self.input_buffer = packet in a finally block that will cover this | 19:12 |
| jeblair | probably the easiest fix | 19:12 |
| *** TimO has joined #openstack-infra | 19:12 | |
| *** TimO is now known as Guest95734 | 19:13 | |
| clarkb | because the returns are handled | 19:13 |
| *** andreykurilin_ has quit IRC | 19:13 | |
| clarkb | so only exceptions need that | 19:13 |
| jeblair | clarkb: finally runs on returns too, so i'll actually just switch everything to packet | 19:14 |
| jeblair | easier to read that way | 19:14 |
| clarkb | jeblair: ++ | 19:14 |
| sweston | asselin: mmedvede the next version of the script is posted. will be testing it again today. now, all of the subtree splits are done on a single cloned merge repo. this will be much easier for us to track and update the history on the entire repository. | 19:14 |
| mmedvede | sweston: hehe, wrong room | 19:15 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Remove stable compatibility jobs from capped libs https://review.openstack.org/137450 | 19:15 |
| *** bdpayne_ has joined #openstack-infra | 19:15 | |
| *** banix has joined #openstack-infra | 19:16 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Remove stable compatibility jobs from capped libs https://review.openstack.org/137451 | 19:16 |
| mmedvede | sweston: that is cool. Were does it maintain the new repositories? | 19:16 |
| Guest95734 | i was wondering what is the best way to find out/request that my dedicated service account can now be upgraded to be approved for voting in OpenStack? | 19:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Use non-blocking IO in server https://review.openstack.org/128754 | 19:17 |
| sweston | mmedvede: right? embarrassing .. | 19:17 |
| clarkb | Guest95734: you should speak to the project you would like to report voting on. If they are happy to do that then we can update your permissions. there is a while blurb on this at http://ci.openstack.org/third_party.html#permissions-on-your-third-party-system | 19:17 |
| *** rmcall has joined #openstack-infra | 19:18 | |
| jeblair | clarkb: ^ stress tested that; shows no adverse effects | 19:18 |
| *** thingee has quit IRC | 19:18 | |
| *** bdpayne has quit IRC | 19:19 | |
| asselin | sweston, ok cool | 19:19 |
| asselin | sweston, yes, was wonder what meeting I'm missing :) | 19:19 |
| sweston | mmedvede: it's all in github, there's a new merge repository called puppet-modules, you can find it in my organization. | 19:19 |
| *** yfried|afk is now known as yfried | 19:19 | |
| Guest95734 | thanks, and have a great Thanksgiving too! | 19:20 |
| *** Guest95734 has quit IRC | 19:21 | |
| sweston | asselin: hehe, yeah .. am a bit short on sleep today ... | 19:21 |
| *** ddieterly has quit IRC | 19:21 | |
| jeblair | clarkb, fungi: i'll let you mull on that for a bit while I check out that ball of fire SpamapS was on about, and if you like it we can maybe try it out after lunch. or save it for next week. :) | 19:21 |
| clarkb | I am all for trying it today | 19:22 |
| fungi | jeblair: yep, already reading through it | 19:22 |
| clarkb | but fungi is mostly afk iirc | 19:22 |
| fungi | "mostly" | 19:22 |
| fungi | ;) | 19:22 |
| clarkb | but ya reading it now | 19:22 |
| nibalizer | clarkb: fungi jeblair so we don't have to deal with this now | 19:22 |
| nibalizer | but 137233 basically means that the templaing we've got in place for puppet.conf is failsauce | 19:22 |
| jeblair | fungi's 'mostly afk' is more present than some people's active presence :) | 19:22 |
| *** reed has joined #openstack-infra | 19:23 | |
| *** thingee has joined #openstack-infra | 19:23 | |
| nibalizer | so my advice would be to get through thanksgiving weekend by 'not touching it' but then we need to look at the pupeptmaster's puppet.conf and figure out why its working at all | 19:23 |
| * fungi insists he is not a bot | 19:23 | |
| jeblair | nibalizer: sounds like a plan; want to wip that so we don't get too excited about merging it? | 19:24 |
| nibalizer | ya | 19:24 |
| fungi | nibalizer: also thanks for the heads up to defer further jenga moves until after we're well-rested | 19:24 |
| jeblair | (i mean, it should be safe, but "should be safe" + "don't understand why it works" == double fail) | 19:25 |
| clarkb | jeblair: I am now trying to understand admin_request in the read method | 19:25 |
| nibalizer | ya what it means is that currently we're doing if (float(undefined_variable) < 3.6 ): { stuff } | 19:26 |
| *** MaxV has joined #openstack-infra | 19:26 | |
| sweston | mostly afk, that's a good away message ... also like intermittently afk | 19:26 |
| nibalizer | and of course in ruby float(undefined_variable) = 0.0 because ruby | 19:26 |
| clarkb | jeblair: and it appears that we push onto the stack after we pop it? | 19:26 |
| clarkb | jeblair: I am not sure how that is working just yet | 19:26 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/infra-specs: Add Migrate to Zanata spec https://review.openstack.org/133222 | 19:26 |
| mmedvede | sweston: this one? https://github.com/Triniplex/puppet-modules | 19:27 |
| sweston | mmedvede: yup | 19:27 |
| *** thingee has quit IRC | 19:28 | |
| clarkb | jeblair: [].pop(0) is an error | 19:28 |
| jeblair | clarkb: where? | 19:28 |
| *** bitblt has joined #openstack-infra | 19:28 | |
| *** banix has quit IRC | 19:28 | |
| clarkb | jeblair: in readPacket() | 19:29 |
| jeblair | clarkb: as soon as we know we're working on an admin request, we pop; and if we have popped and get EAGAIN, we push | 19:29 |
| clarkb | jeblair: the check is if admin is None: if byte else: admin = True admin_request = self._getAdminRequest() which does a pop | 19:29 |
| clarkb | jeblair: right but what does the initial push? | 19:29 |
| *** yfried is now known as yfried|afk | 19:29 | |
| *** changbl has joined #openstack-infra | 19:29 | |
| clarkb | that list is initialized to [] and I don't see a putAdminRequest happening before the getAdminRequst | 19:30 |
| *** bitblt has left #openstack-infra | 19:30 | |
| jeblair | clarkb: ah -- sending an admin request | 19:30 |
| *** adalbas has quit IRC | 19:30 | |
| *** thingee has joined #openstack-infra | 19:30 | |
| clarkb | oh so that code only runs client side? | 19:31 |
| *** AJaeger_ has joined #openstack-infra | 19:31 | |
| jeblair | clarkb: yeah, it's different on client vs server | 19:31 |
| jeblair | clarkb: on client, this is handling a response to a previously sent admin request | 19:31 |
| jeblair | clarkb: thus it will pop from a previously externally populated list of admin requests that have been sent | 19:31 |
| *** emagana has quit IRC | 19:31 | |
| clarkb | jeblair: but the server won't see that as an admin_request in readPacket()? | 19:31 |
| *** tnovacik has joined #openstack-infra | 19:32 | |
| jeblair | clarkb: server side, _getAdminRequest instantiates a new admin request object | 19:32 |
| *** emagana has joined #openstack-infra | 19:32 | |
| jeblair | clarkb: i think there's a small bug here though | 19:32 |
| jeblair | in that if the server gets eagain while reading it, it will put the admin request on the stack | 19:32 |
| clarkb | I don't see where server side does that | 19:33 |
| jeblair | clarkb: the same read method | 19:33 |
| clarkb | _getAdminRequest just pops off that stack which should be empty | 19:33 |
| clarkb | jeblair: right but something must push onto the stack before we pop | 19:33 |
| jeblair | clarkb: server connections override _getAdminRequest | 19:33 |
| *** AlexF has quit IRC | 19:34 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/jenkins-job-builder: Add parameter to trigger_parameterized_builds https://review.openstack.org/137455 | 19:34 |
| *** ddieterly has joined #openstack-infra | 19:34 | |
| jeblair | clarkb: the server-side bug is a memory leak -- if it gets eagain, it will _putAdminRequest which will push it on a stack that is unused by the server; so i think the fix is a noop _putAdminRequest | 19:34 |
| clarkb | oh I see my grep was failing me | 19:34 |
| clarkb | jeblair: so server side should override putAdminRequest as well | 19:35 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Remove stable compatibility jobs from capped libs https://review.openstack.org/137451 | 19:35 |
| jeblair | clarkb: so in short -- client side behavior should be pop/push from list (where previous submissions of admin requests go in the list); server side should ignore the list completely and only produce new admin request objects to return when they have been fully received | 19:35 |
| clarkb | dhellmann: ^ I think you do, you just want to limit the branches where those compat jobs run to the things they support | 19:36 |
| clarkb | jeblair: yup | 19:36 |
| clarkb | jeblair: are you going to push a new patchset with putServerRequest on server side overridden? | 19:36 |
| dhellmann | clarkb: ? | 19:36 |
| clarkb | dhellmann: you don't want to remove those jobs. you just want ot restrict where they run | 19:36 |
| *** emagana has quit IRC | 19:36 | |
| clarkb | dhellmann: so that you can do bugfixes | 19:36 |
| *** koolhead17 has quit IRC | 19:36 | |
| *** banix has joined #openstack-infra | 19:36 | |
| jeblair | clarkb: yep | 19:36 |
| dhellmann | clarkb: where would I want them to run, though? | 19:37 |
| clarkb | dhellmann: under the version caps | 19:37 |
| *** koolhead17 has joined #openstack-infra | 19:37 | |
| dhellmann | clarkb: those jobs currently test the master version of the library against the stable version of the apps -- that's no longer a supported configuration | 19:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Use non-blocking IO in server https://review.openstack.org/128754 | 19:37 |
| clarkb | dhellmann: no the test the proposed change against that stable version of the apps | 19:38 |
| clarkb | dhellmann: this is still supported for things under the version caps | 19:38 |
| clarkb | say oslo.foo<0.7.0 is your cap | 19:38 |
| clarkb | and you discover 0.6.0 is broken | 19:38 |
| clarkb | securit vulnerability | 19:38 |
| dhellmann | yes | 19:38 |
| clarkb | you then push a patch that will become 0.6.1. you want to test that against say icehouse | 19:38 |
| clarkb | that change doesn't happen on master it happens on another branch | 19:39 |
| dhellmann | yes, that's right | 19:39 |
| clarkb | you want any of those changes to be tested against the server apps stable branches that they support | 19:39 |
| dhellmann | I think I'm content to use unit tests for that | 19:39 |
| dhellmann | sdague: ^^ | 19:40 |
| sdague | yeh, I'm assuming that these sec backports are going to be extremely small | 19:40 |
| dhellmann | and equally infrequent | 19:40 |
| sdague | yeh | 19:40 |
| dhellmann | if that proves not to be the case, we can add some jobs, but I'm not sure we need them at this point | 19:41 |
| sdague | right, agreed | 19:41 |
| clarkb | ok | 19:41 |
| *** ihrachyshka has quit IRC | 19:41 | |
| * clarkb notes that we have broken all the things a couple times in just the last week | 19:41 | |
| *** koolhead17 has quit IRC | 19:41 | |
| *** otter768 has joined #openstack-infra | 19:42 | |
| sdague | so what this really begs for is a thing that has been asked for from time to time | 19:42 |
| *** emagana has joined #openstack-infra | 19:42 | |
| sdague | which is the ability to adhoc a job | 19:42 |
| *** ihrachyshka has joined #openstack-infra | 19:43 | |
| sdague | like when lifeless is getting ready to release new testr, and really wants to build a thing that tests vs. a branch of testr | 19:43 |
| clarkb | you can do that | 19:44 |
| sdague | but it's close to snow blowing time so I'm not sure I have a fully formed thought there | 19:44 |
| *** Ryan_Lane has quit IRC | 19:44 | |
| sdague | clarkb: how? | 19:44 |
| clarkb | its ugly | 19:44 |
| clarkb | you push a devstack/d-g change that does it | 19:44 |
| sdague | that won't let you test against nova unit tests, if you know that's the thing you usually break | 19:44 |
| *** Ryan_Lane has joined #openstack-infra | 19:45 | |
| clarkb | sdague: so its a similar mechanism for nova unittests | 19:45 |
| clarkb | push a change to nova that installs the version ofthe thing you want from git | 19:45 |
| clarkb | or bzr/svn/hg etc | 19:45 |
| sdague | sure, but it's such a system level hack. I wonder if we could make it a simple thing. | 19:46 |
| sdague | anyway, it's a half formed thought | 19:46 |
| *** Sukhdev has joined #openstack-infra | 19:46 | |
| *** otter768 has quit IRC | 19:46 | |
| *** HeOS has joined #openstack-infra | 19:46 | |
| clarkb | I think the trouble with making it a "simple" thing is that you would have to somehow bypass the mechanism I describe which is how we always install software | 19:46 |
| clarkb | IMO you want to test it as normal | 19:47 |
| lifeless | clarkb: or you make the mechanism aware | 19:49 |
| lifeless | clarkb: I think the success criteria needs to be that a single action is taken that runs all jobs that use that upstream thing, with the proposed upstream thing | 19:49 |
| sdague | lifeless: aren't you supposed to be on holiday :) | 19:50 |
| lifeless | I think there are a number of related things needed to do that | 19:50 |
| *** thingee has quit IRC | 19:50 | |
| lifeless | sdague: I am | 19:50 |
| lifeless | sdague: someone said my name :) | 19:50 |
| *** Ryan_Lane has quit IRC | 19:50 | |
| sdague | lifeless: well you have a valid use case here :) | 19:50 |
| lifeless | yup | 19:50 |
| clarkb | right but how do you know that some person over there wherever there is has a thing you should test | 19:51 |
| clarkb | you don't really. So pushing a change to test it is reasonable | 19:51 |
| lifeless | FWIW the proposed? landed? plumbing in toci to make a localpypi index for things from git is one way of avoiding special cases in the code | 19:51 |
| *** ecosta has joined #openstack-infra | 19:51 | |
| lifeless | clarkb: pushing as a mechanism doesn't bother me | 19:51 |
| lifeless | clarkb: pushing 40 patches to test one thing does | 19:51 |
| lifeless | clarkb: because frankly, too much effort. | 19:52 |
| clarkb | lifeless: the toci thing still needs to be told what the composition is | 19:52 |
| clarkb | which is the $patches | 19:52 |
| clarkb | if you want a meta project that has all the tests in the world against it that you can trigger via a single change that is probably doable | 19:52 |
| sdague | clarkb: yeh, I think that would be the shape of it | 19:53 |
| clarkb | but you still need to say here is a ref go test it | 19:53 |
| sdague | yep | 19:53 |
| sdague | but that would be the format of the proposed patch | 19:53 |
| clarkb | jeblair: current gear patchset lgtm | 19:55 |
| *** dims has quit IRC | 19:56 | |
| kevinbenton | in our 3rd party CI system I ran into an issue with a mismatch between oslo.middleware and the global requirements on stable/icehouse tests | 19:57 |
| kevinbenton | and i made this change which seemed to fix it https://review.openstack.org/#/c/136613/ | 19:57 |
| kevinbenton | but it seems like that might not be the real issue | 19:57 |
| *** thingee has joined #openstack-infra | 19:57 | |
| kevinbenton | since the OpenStack CI doesn't seem to be suffering the same issue | 19:57 |
| dhellmann | kevinbenton: see https://review.openstack.org/#/c/137392/2 | 19:57 |
| krotscheck | Any infra cores interested in helping clear out some minor puppet-storyboard patches? https://review.openstack.org/#/q/status:open+project:openstack-infra/puppet-storyboard,n,z | 19:58 |
| clarkb | krotscheck: the rspec job is expected to fail right? | 19:58 |
| kevinbenton | dhellmann: ah, so that should supercede my patch then, right? | 19:58 |
| *** emagana has quit IRC | 19:59 | |
| dhellmann | clarkb: right now I'm trying to figure out how to make check-requirements-integration-dsvm branch-aware, but I think I don't understand enough about the preconditions for that job. | 19:59 |
| krotscheck | clarkb: Yep, nibalizer pointed that out last week. | 19:59 |
| dhellmann | kevinbenton: yeah, we don't want to pin things in devstack itself | 19:59 |
| kevinbenton | dhellmann: sounds good. thanks for the fast response | 19:59 |
| *** emagana has joined #openstack-infra | 19:59 | |
| sdague | ok, need to go deal with snow for a while. Back in a bit. | 20:00 |
| dhellmann | kevinbenton: glad to help! | 20:00 |
| clarkb | dhellmann: it shold already be branch aware | 20:00 |
| *** ryanpetrello has quit IRC | 20:00 | |
| clarkb | dhellmann: maye I don't understand what you mean by branch aware but it should check stable/X against stable/X | 20:00 |
| dhellmann | clarkb: well, it's currently failing in stable/juno because it is checking the master version of requirements in oslo.messaging | 20:00 |
| dhellmann | http://logs.openstack.org/99/137399/2/check/check-requirements-integration-dsvm/bcc2f98/console.html | 20:01 |
| clarkb | dhellmann: what should it be checking in oslo.messaging? | 20:01 |
| *** radez_g0n3 is now known as radez | 20:01 | |
| dhellmann | clarkb: stable/juno, or nothing | 20:01 |
| dhellmann | well, when it's running against the stable/juno version of the global requirements it should check the stable/juno version of messaging | 20:01 |
| clarkb | there is no stable/juno in oslo.messaging | 20:01 |
| clarkb | if there was I think it would work properly | 20:01 |
| dhellmann | that job failed on this patch against https://review.openstack.org/#/c/137399/ | 20:01 |
| dhellmann | ah, ok | 20:02 |
| kevinbenton | clarkb: i thought the branches were explicit in stackrc? | 20:02 |
| clarkb | kevinbenton: thats completely separate | 20:02 |
| *** ryanpetrello has joined #openstack-infra | 20:02 | |
| clarkb | kevinbenton: this is just doing comparison of requirements files | 20:02 |
| clarkb | kevinbenton: and installing things into virtualenvs to ensure they at least install together | 20:03 |
| *** emagana has quit IRC | 20:03 | |
| kevinbenton | clarkb: oh okay. so that process checks branches as well? | 20:03 |
| dhellmann | clarkb: how do I make a new stable/juno branch? is that a gerrit thing? | 20:04 |
| ecosta | hello, i am new here. Is there a reason for jenkins not running for a review, even after all of its dependencies are met? | 20:04 |
| *** ihrachyshka has quit IRC | 20:04 | |
| *** nadya_ has joined #openstack-infra | 20:04 | |
| dhellmann | clarkb: now that I think of it, I think ttx has always done that for me so I might not have permission | 20:04 |
| clarkb | dhellmann: ya in the gerrit gui under the project you can create it | 20:04 |
| clarkb | dhellmann: let me get you a link and ou can see if you have perms. if not I can do it for you | 20:04 |
| dhellmann | k | 20:05 |
| clarkb | dhellmann: https://review.openstack.org/#/admin/projects/openstack/oslo.messaging,branches | 20:05 |
| clarkb | ecosta: can you link to the review? | 20:05 |
| dhellmann | clarkb: yeah, I don't see anything on that page that looks like a "make a branch" button | 20:05 |
| ecosta | https://review.openstack.org/136915 | 20:05 |
| fungi | that change was definitely uploaded while zuul was offline | 20:06 |
| clarkb | dhellmann: there should be a create branch button near the bottom | 20:06 |
| *** e0ne has joined #openstack-infra | 20:06 | |
| fungi | ecosta: a recheck comment should get it going | 20:06 |
| clarkb | dhellmann: if you give me a ref to base stable/juno on I can create that branch | 20:06 |
| dhellmann | clarkb: no, I think I'm not on the approved list | 20:06 |
| dhellmann | clarkb: 6ea3b12492e86f9e8d109fc3490cc4d3a0edd8b6 | 20:06 |
| ecosta | clarkb: fungi: ok, thanks! | 20:07 |
| *** thingee has quit IRC | 20:07 | |
| clarkb | dhellmann: done, you should see it if you refresh that page | 20:07 |
| dhellmann | clarkb: got it, thanks! | 20:07 |
| *** aysyd has quit IRC | 20:07 | |
| *** AJaeger_ has quit IRC | 20:07 | |
| *** thingee has joined #openstack-infra | 20:08 | |
| dhellmann | clarkb: I want to recheck my stable/juno requirements change now. Should I wait some period of time for that branch to be synced somewhere or is it safe to recheck right away? | 20:08 |
| *** ihrachyshka has joined #openstack-infra | 20:08 | |
| *** wenlock1 has joined #openstack-infra | 20:09 | |
| fungi | clarkb: jeblair: 128754 lgtm. not sure what the plan was for landing it in production so didn't approve | 20:09 |
| clarkb | dhellmann: that is a good question let me check the git mirror really quickly | 20:09 |
| dhellmann | clarkb: thanks | 20:09 |
| clarkb | dhellmann: it should sync almost immediately | 20:09 |
| clarkb | dhellmann: http://git.openstack.org/cgit/openstack/oslo.messaging/refs/ has it so you should be fine to recheck | 20:09 |
| dhellmann | clarkb: cool | 20:09 |
| *** ryanpetrello has quit IRC | 20:10 | |
| dhellmann | oh, bother, did the recheck detection get updated to allow extra comments or do I still have to just say "recheck" by itself? | 20:11 |
| *** ldnunes has quit IRC | 20:12 | |
| fungi | dhellmann: as long as the first line starts with the word recheck it should take it | 20:12 |
| *** hashar has joined #openstack-infra | 20:12 | |
| fungi | additional words after recheck are just ignored now | 20:12 |
| dhellmann | fungi: ok, that's what I thought but it didn't seem to work | 20:12 |
| clarkb | fungi: found a bug in the groups vhost stuff. patch incoming shortly | 20:12 |
| dhellmann | fungi: oh, weird, I somehow commented on an old changeset so that's why it didn't work | 20:13 |
| *** thingee has quit IRC | 20:13 | |
| fungi | dhellmann: yep, has to be the most recent | 20:13 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add e-r query for bug 1372670 https://review.openstack.org/136915 | 20:13 |
| uvirtbot | Launchpad bug 1372670 in nova "libvirtError: operation failed: cannot read cputime for domain" [High,Confirmed] https://launchpad.net/bugs/1372670 | 20:13 |
| dhellmann | fungi: yeah, that makes sense; user error | 20:14 |
| fungi | clarkb: d'oh! yep, failed to start apache | 20:14 |
| clarkb | fungi: ya | 20:14 |
| *** thingee has joined #openstack-infra | 20:14 | |
| fungi | clarkb: SSLCertificateChainFile takes one argument, SSL Server CA Certificate Chain file (`/path/to/file' - PEM encoded) | 20:15 |
| clarkb | nibalizer: how do I compare against undef in an erb file? | 20:15 |
| * fungi sighs | 20:15 | |
| *** mpavlase has joined #openstack-infra | 20:16 | |
| clarkb | hrm I think I found a table in puppet docs | 20:16 |
| *** e0ne has quit IRC | 20:17 | |
| *** ecosta has quit IRC | 20:18 | |
| *** denis_makogon has quit IRC | 20:19 | |
| *** dmakogon_ has joined #openstack-infra | 20:19 | |
| *** thingee has quit IRC | 20:19 | |
| *** ryanpetrello has joined #openstack-infra | 20:20 | |
| *** andreykurilin_ has joined #openstack-infra | 20:20 | |
| *** thingee has joined #openstack-infra | 20:21 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Fix the setting of the chain file in drupal vhost https://review.openstack.org/137463 | 20:22 |
| clarkb | fungi: ^ | 20:22 |
| fungi | clarkb: yep, looks right | 20:23 |
| nibalizer | clarkb: gj | 20:23 |
| fungi | i totally missed there was no explicit default being set for it | 20:24 |
| clarkb | ok I am going to eat and stuff then come back ready to do nodepool again | 20:24 |
| fungi | i'm going to beat on the manage-projects fix by approving a few more project creation patches to see if i can get the bug to resurface | 20:25 |
| *** sarob has quit IRC | 20:28 | |
| *** enikanorov_ has quit IRC | 20:28 | |
| *** enikanorov_ has joined #openstack-infra | 20:28 | |
| *** sarob has joined #openstack-infra | 20:29 | |
| *** Masahiro has joined #openstack-infra | 20:30 | |
| *** ociuhandu has quit IRC | 20:31 | |
| mrmartin | fungi: hi. I guess the SSL patch had a little bug that killed apache :) I guess this line is wrong: https://github.com/openstack-infra/system-config/blob/master/modules/drupal/templates/drupal.vhost.erb#L19 because we are not checking the value against undef. | 20:31 |
| fungi | mrmartin: see just moments ago where clarkb proposed a patch to fix that. should be merging shortly | 20:32 |
| mrmartin | oh perfect, I also found that meanwhile | 20:32 |
| openstackgerrit | Merged openstack-infra/system-config: Fix the setting of the chain file in drupal vhost https://review.openstack.org/137463 | 20:33 |
| *** sarob has quit IRC | 20:34 | |
| *** Masahiro has quit IRC | 20:34 | |
| *** AlexF has joined #openstack-infra | 20:34 | |
| fungi | mrmartin: ^ there it is. should be live in a few minutes | 20:35 |
| *** baoli has joined #openstack-infra | 20:35 | |
| mrmartin | fungi: it was super-fast | 20:36 |
| *** rpodolyaka1 has quit IRC | 20:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Create initial ansible deployment repository https://review.openstack.org/132703 | 20:36 |
| *** andreykurilin_ has quit IRC | 20:37 | |
| fungi | okay, hopefully i'll know shortly if that ^ succeeded in manage-projects | 20:37 |
| *** nadya_ has quit IRC | 20:39 | |
| *** Bart has joined #openstack-infra | 20:42 | |
| *** Bart___ has quit IRC | 20:42 | |
| *** belmoreira has joined #openstack-infra | 20:42 | |
| *** lttrl has quit IRC | 20:48 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/infra-specs: Spec to refactor puppet code in openstack_project https://review.openstack.org/137471 | 20:50 |
| *** bswartz has quit IRC | 20:54 | |
| lifeless | clarkb: sdague: meta-repo, aka requirements? | 20:55 |
| sdague | clarkb: so, do you have thoughts on this way of cross testing that the python libs definition in devstack with project-config - https://review.openstack.org/#/c/135948/2 | 20:56 |
| *** nadya_ has joined #openstack-infra | 20:56 | |
| sdague | lifeless: I think more meta than that. A little more like if project-config actually ran the proposed tests in your diff when you proposed to it | 20:57 |
| clarkb | which should be a lot more possible when we ditch jenkins | 20:58 |
| clarkb | since each worker should have a mostly complete view of the world | 20:58 |
| clarkb | sdague: for cross testing like that I think you just run those tess on project-config changes and on devstack changes | 20:58 |
| clarkb | sdague: you can limit it to the layout file on the project-config side | 20:58 |
| sdague | clarkb: so my test assumes that you update project-config first | 20:58 |
| clarkb | sdague: I think thats ok | 20:59 |
| sdague | I'm ok with devstack wedging if someone does a bad in project-config | 20:59 |
| sdague | as it's easy enough to disable (the tests being in tree) | 20:59 |
| clarkb | oh in that case just run it against devtsack | 20:59 |
| lifeless | sdague: if requirements changes trigger everything today (they do, right?) and we want to know when they change that everything will work with the new dep (thats the goal right?) then I don't think we need more meta | 20:59 |
| sdague | mostly the fact that I'm cloning | 20:59 |
| lifeless | OTOH I'm just gasbagging, so whatevs, whomever does it can make it work | 20:59 |
| clarkb | lifeless: no requirements changes trigger very little today | 20:59 |
| sdague | lifeless: they don't trigger everything, they trigger a small static set | 20:59 |
| lifeless | ah | 21:00 |
| * dhellmann wishes this requirements check job had access to wheels for all of these things it is pip installing | 21:00 | |
| lifeless | anyhow, running | 21:00 |
| fungi | clarkb: good news. stackforge/os-ansible-deployment seems to have created successfully, and it's pretty far down the layout.yaml so would have likely run into the timeout i think | 21:00 |
| lifeless | ciao | 21:00 |
| clarkb | dhellmann: it does | 21:00 |
| clarkb | unless we ripped that out | 21:00 |
| clarkb | which we probably did rip out beacuse pypi-mirror is dead | 21:00 |
| dhellmann | clarkb: it's building pyyaml over and over, so I don't know | 21:00 |
| *** AlexF has quit IRC | 21:01 | |
| *** baoli has quit IRC | 21:01 | |
| clarkb | fungi: fyi I manually ran puppet on groups-dev and it seems to work now | 21:01 |
| clarkb | dhellmann: ya I think pypi-mirror getting ripped out meant no more wheels | 21:02 |
| *** baoli has joined #openstack-infra | 21:02 | |
| sdague | clarkb: that's running in a devstack, right? | 21:02 |
| fungi | clarkb: mrmartin: yep, looks like it's working now | 21:03 |
| jeblair | clarkb, fungi: back | 21:03 |
| *** AlexF has joined #openstack-infra | 21:03 | |
| clarkb | sdague: it uses devstack to install the C deps | 21:03 |
| clarkb | sdague: but it shuts down devstack iirc | 21:03 |
| sdague | clarkb: ok | 21:03 |
| fungi | i'm disappearing for a bit, and then testing some more project creation changes when i get back | 21:03 |
| sdague | I was thinking that adding working devpi install into devstack might simplify a bunch of things | 21:04 |
| jeblair | clarkb, fungi: etherpad for testing gear stuff: | 21:04 |
| mrmartin | seems to be: http://puppetboard.openstack.org/report/groups-dev.openstack.org/ee376dc46a10af4c854862a63f26c69c7fb6fda3 | 21:04 |
| clarkb | jeblair: I am ready to try things in just a minute | 21:04 |
| jeblair | clarkb, fungi: https://etherpad.openstack.org/p/Ckic2h1qg2 | 21:04 |
| *** esker has joined #openstack-infra | 21:04 | |
| *** esker has quit IRC | 21:04 | |
| sdague | but, anyway, I need to go run and get beer for tomorrow while we have a break in the snow. | 21:04 |
| clarkb | sdague: I dunno. I still contend devpi exists so that you can have your forks and private code in a pip installable place | 21:04 |
| clarkb | sdague: which really isn't a use case we have anywhere | 21:04 |
| fungi | cool, when i return in a little while i'll catch this program already in progress ;) | 21:04 |
| sdague | clarkb: you can also prepopulate it's cache with wheel builds | 21:04 |
| *** esker has joined #openstack-infra | 21:05 | |
| clarkb | sdague: we can do that on bandersnatch too | 21:05 |
| clarkb | or alongside bandersnatch | 21:05 |
| sdague | ok, so lets do that then :) | 21:05 |
| *** atan8 has quit IRC | 21:05 | |
| dhellmann | clarkb: could you make a stable/juno branch of oslo.vmware on commit 589ba439cf9be506ceafc8ad195fb4260b58a0d8 for me please? | 21:05 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Enable color support based on tty and config https://review.openstack.org/106617 | 21:05 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/nodepool: Add REST API to Nodepool https://review.openstack.org/136884 | 21:05 |
| dhellmann | clarkb: wait, wrong commit | 21:05 |
| dhellmann | clarkb: use 1661a0af8ce14edbfb00fed0a30a3b9135517d93 | 21:06 |
| jeblair | clarkb: oh, the puppet dns change hasn't merged? | 21:06 |
| *** emagana has joined #openstack-infra | 21:06 | |
| clarkb | jeblair: not yet | 21:07 |
| *** pc_m has joined #openstack-infra | 21:07 | |
| jeblair | clarkb: let's merge that now, since the point of that was to make it easy to do this | 21:07 |
| clarkb | jeblair: ok you want to do that? fungi has +2'd it | 21:07 |
| jeblair | have a link? | 21:07 |
| clarkb | its in the etherpad | 21:07 |
| clarkb | https://review.openstack.org/#/c/137167/ | 21:08 |
| *** pc_m has quit IRC | 21:08 | |
| *** pc_m has joined #openstack-infra | 21:09 | |
| *** e0ne has joined #openstack-infra | 21:09 | |
| *** atan8 has joined #openstack-infra | 21:10 | |
| *** Sukhdev has quit IRC | 21:10 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support copyartifact build selector param https://review.openstack.org/133774 | 21:11 |
| jeblair | clarkb: okay, i think that's the preliminaries. let's take a short intermission while that change works its way through and resume in a bit (should be like 20-30 mins) | 21:12 |
| *** weshay has quit IRC | 21:12 | |
| clarkb | ++ | 21:12 |
| *** e0ne has quit IRC | 21:13 | |
| *** achanda has joined #openstack-infra | 21:15 | |
| *** thingee has quit IRC | 21:15 | |
| *** radez is now known as radez_g0n3 | 21:17 | |
| openstackgerrit | Merged openstack-infra/system-config: Allow traffic from nodepool-dev https://review.openstack.org/137167 | 21:17 |
| *** thingee has joined #openstack-infra | 21:17 | |
| *** Sukhdev has joined #openstack-infra | 21:17 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support copyartifact build selector param https://review.openstack.org/133774 | 21:18 |
| *** andreykurilin_ has joined #openstack-infra | 21:19 | |
| *** achanda has quit IRC | 21:20 | |
| *** pradk has quit IRC | 21:23 | |
| jhesketh | Morning | 21:23 |
| dhellmann | clarkb: it looks like you're in the middle of something with jeblair, but when you have a free minute would you let me know? | 21:23 |
| *** baoli has quit IRC | 21:24 | |
| clarkb | dhellmann: I am free for the next 10 minutes or so | 21:24 |
| *** emagana has quit IRC | 21:24 | |
| *** atan8 has quit IRC | 21:24 | |
| *** baoli has joined #openstack-infra | 21:24 | |
| dhellmann | clarkb: cool. I ran into another branch-related issue with that requirements job with oslo.vmware this time. | 21:24 |
| *** emagana has joined #openstack-infra | 21:24 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Basic folder support https://review.openstack.org/134307 | 21:25 |
| clarkb | oh that request ya I haven't created the branch yet | 21:25 |
| clarkb | dhellmann: you want stable/juno based on 1661a0af8ce14edbfb00fed0a30a3b9135517d93 ? | 21:25 |
| dhellmann | clarkb: yeah, I was actually going to ask about what I need to do to get permissions to make branches myself. Is that an infra thing, a release team thing, or something else? | 21:25 |
| dhellmann | clarkb: yes, that's the right tag | 21:25 |
| dhellmann | I suspect there will be a couple more of these, and I'd like to go back and retroactively branch all of the libs so we're consistent | 21:26 |
| clarkb | its a release managers thing | 21:26 |
| dhellmann | ok, I'll talk to ttx about it next week then | 21:26 |
| *** nadya_ has quit IRC | 21:26 | |
| clarkb | so I think adam_g, apevec, sdague, and ttx can do it | 21:26 |
| clarkb | then infra can because hax | 21:26 |
| dhellmann | I need to make sure he's aware of this new process thing for us anyway, since I think we decided to only create branches when we needed them, but now we'll need them at the end of each cycle | 21:26 |
| *** bhunter71 has quit IRC | 21:27 | |
| *** atan8 has joined #openstack-infra | 21:27 | |
| clarkb | branch created | 21:27 |
| dhellmann | ty | 21:27 |
| *** emagana_ has joined #openstack-infra | 21:27 | |
| *** emagana has quit IRC | 21:27 | |
| dhellmann | clarkb: on the wheel thing, it sounds like bandersnatch can do that for us somehow, is that right? | 21:28 |
| clarkb | dhellmann: not really. What I meant to say is we could host wheels like we used to alongside the bandersnatch mirror | 21:29 |
| dhellmann | ah | 21:29 |
| clarkb | dhellmann: so we have pypi.region.openstack.org/simple we could also have pypi.region.openstack.org/wheels/platform/stuff | 21:29 |
| clarkb | then use something to fill that in | 21:29 |
| clarkb | maybe resurrect a subset of pypi-mirror's features | 21:29 |
| dhellmann | a post-merge job on the requirements list, I would think | 21:29 |
| clarkb | I am sure mordred has opinions n this | 21:29 |
| clarkb | dhellmann: ya that would be one way of doing it but it would miss any upstream updates | 21:30 |
| clarkb | more likely a daily cron or something that uses requireemnts as the base and builds wheels on trusty, precise, centos, fedora et al then rsyncs everywhere | 21:30 |
| dhellmann | true | 21:30 |
| *** amitpp has joined #openstack-infra | 21:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Ignore playground directories for translation https://review.openstack.org/133368 | 21:31 |
| *** yfried|afk is now known as yfried | 21:31 | |
| *** emagana has joined #openstack-infra | 21:31 | |
| *** emagana_ has quit IRC | 21:31 | |
| *** weshay has joined #openstack-infra | 21:31 | |
| *** ryanpetrello has quit IRC | 21:32 | |
| *** Bart has quit IRC | 21:33 | |
| *** Bart has joined #openstack-infra | 21:33 | |
| jeblair | clarkb: (afs) | 21:34 |
| jeblair | (which is next on my list once we unblock the dib stuff) | 21:34 |
| *** rfolco has quit IRC | 21:34 | |
| jeblair | clarkb: i don't see the firewall changes having taken effect on jenkins\d+ | 21:36 |
| openstackgerrit | Merged openstack-infra/project-config: Use storyboard for all infra puppet modules https://review.openstack.org/136594 | 21:36 |
| *** JayJ has joined #openstack-infra | 21:36 | |
| krotscheck | woot | 21:36 |
| clarkb | oh ya afs | 21:37 |
| clarkb | jeblair: hrm let me look at puppetboard | 21:37 |
| clarkb | http://puppetboard.openstack.org/node/jenkins01.openstack.org doesn;t look like jenkins01 has reported back yet | 21:37 |
| clarkb | jeblair: so maybe ansible is still running | 21:37 |
| *** thingee has quit IRC | 21:38 | |
| jeblair | clarkb: it is; i was expecting something from the :20-:30 run | 21:38 |
| jhesketh | Hey, if anybody has a chance to create me an initial gerrit group member (https://storyboard.openstack.org/#!/story/2000029) that'd be awesome :-) | 21:38 |
| clarkb | jeblair: it runs every 15 minutes | 21:38 |
| clarkb | so the :17 merge was just too late | 21:39 |
| nibalizer | can i just say that the system-config and project-config split was the best idea ever | 21:39 |
| *** kgiusti has quit IRC | 21:39 | |
| jeblair | clarkb: oh i thought it was 10 | 21:39 |
| nibalizer | every time i mess with our hp version that is pre-split i am so amazed | 21:39 |
| jeblair | jhesketh: will do | 21:39 |
| *** thingee has joined #openstack-infra | 21:39 | |
| clarkb | jeblair: I got it :) | 21:39 |
| jeblair | clarkb, fungi: do we still want people to open bugs for initial group membership? | 21:40 |
| jeblair | (i don't) | 21:40 |
| jhesketh | jeblair: Thanks :-) | 21:40 |
| jeblair | jhesketh: someone already did that | 21:40 |
| jhesketh | you guys are fast! | 21:41 |
| clarkb | jeblair: I personally prefer being pinged and just getting it done, maybe do email for cases when timezones don't overlap | 21:41 |
| clarkb | or ENOIRC | 21:41 |
| *** yfried is now known as yfried|afk | 21:41 | |
| jeblair | clarkb: okay, jenkins02 has the new iptables rule | 21:42 |
| *** kgiusti has joined #openstack-infra | 21:42 | |
| clarkb | hrm the rules say nodepool.o.o twice | 21:43 |
| *** otter768 has joined #openstack-infra | 21:43 | |
| clarkb | is it looking at the reverse record? | 21:43 |
| * clarkb tests from the new trusty node | 21:43 | |
| jeblair | clarkb: yeah, the actual rule is ip based | 21:43 |
| clarkb | I can telnet to jenkins01:8888 so it seems happy | 21:43 |
| jeblair | clarkb: so it does a forward lookup when setting the rule | 21:43 |
| jeblair | clarkb: and if you list rules, it does a reverse lookup to be pretty | 21:44 |
| jeblair | clarkb: sudo iptables -L -n | 21:44 |
| jeblair | clarkb: and you'll get the numeric version | 21:44 |
| clarkb | gotcha | 21:44 |
| clarkb | looks like all of the jenkinses have updated according to puppetboard | 21:44 |
| jeblair | clarkb: shall i do zuul while you do the nodepools? | 21:45 |
| *** wenlock1 has quit IRC | 21:45 | |
| clarkb | sure | 21:45 |
| jeblair | clarkb: ready? | 21:45 |
| clarkb | one sec | 21:46 |
| *** Guest8638 has quit IRC | 21:46 | |
| clarkb | ok I am ready let me know when zuul is down and I will take down old nodepool | 21:46 |
| jeblair | clarkb: zuul is stopped | 21:46 |
| clarkb | old nodepool service is stopped | 21:47 |
| clarkb | if I start new nodepool it will wait until gear is up. will you run an out of band geard? | 21:47 |
| clarkb | (I am ready to start new nodepool service when you are) | 21:47 |
| jeblair | clarkb: geard is running | 21:47 |
| clarkb | ok starting new nodepool now | 21:47 |
| *** wenlock has quit IRC | 21:48 | |
| *** otter768 has quit IRC | 21:48 | |
| *** achanda has joined #openstack-infra | 21:48 | |
| clarkb | it is doing stuff | 21:48 |
| *** dkliban is now known as dkliban_biab | 21:48 | |
| jeblair | clarkb: so actually we want to wait until all old nodepool nodes are deleted, right? | 21:49 |
| jeblair | clarkb: before starting zuul? | 21:49 |
| clarkb | jeblair: I don't think it is required | 21:49 |
| clarkb | it will help with quota limits though | 21:49 |
| clarkb | I am starting the deletions now | 21:49 |
| *** achanda has quit IRC | 21:49 | |
| *** banix has quit IRC | 21:49 | |
| jeblair | clarkb: we might start running jobs on them though | 21:49 |
| *** jungleboyj has quit IRC | 21:49 | |
| clarkb | oh right | 21:50 |
| jeblair | clarkb: ah, so actually it's the ready nodes we care about | 21:50 |
| clarkb | yup | 21:50 |
| jeblair | clarkb: so i guess once all the old ready's are deleted we can start zuul | 21:50 |
| clarkb | the deletion process is running with 10 "workers" | 21:50 |
| *** mattfarina has quit IRC | 21:50 | |
| *** MaxV has quit IRC | 21:51 | |
| *** bhunter71 has joined #openstack-infra | 21:51 | |
| *** kgiusti has quit IRC | 21:51 | |
| *** ihrachyshka has quit IRC | 21:52 | |
| *** fandi has joined #openstack-infra | 21:53 | |
| *** matt__ is now known as mattoliverau | 21:53 | |
| clarkb | jeblair: I spun up a second set of deletions to focus on ready nodes | 21:54 |
| jeblair | k | 21:54 |
| clarkb | so those should go away relatively quickly | 21:54 |
| jeblair | new nodepool got a TimeoutError, but i have geard running in a terminal and it does not seem to be backlogged | 21:54 |
| clarkb | jeblair: iirc old nodepool had one or two right after switching over. Maybe that is a separate issue? | 21:55 |
| *** hdd has joined #openstack-infra | 21:56 | |
| jeblair | clarkb: maybe? the function list is taking only a few seconds, but maybe if a lot of nodes came on line around the same time it could get starved enough... | 21:57 |
| *** dannywilson has quit IRC | 21:58 | |
| *** amitpp has quit IRC | 21:58 | |
| *** dannywilson has joined #openstack-infra | 21:59 | |
| jeblair | clarkb: all the ready nodes are gone ;shall i start zuul? | 22:00 |
| clarkb | yup go for it | 22:00 |
| *** melwitt has joined #openstack-infra | 22:01 | |
| jeblair | clarkb: started and queue resumed | 22:02 |
| *** thingee has quit IRC | 22:02 | |
| jeblair | there are some not_registered jobs | 22:02 |
| *** patrickeast_ has joined #openstack-infra | 22:02 | |
| jeblair | clarkb: so maybe we need to rebuild the devstack-precise image here? | 22:03 |
| clarkb | let me look | 22:03 |
| clarkb | we are building one in iad right now | 22:04 |
| clarkb | a node not image. and if iad on new nodepool has same problem as old nodepool then yes. let me go ahead and delete the existing image in iad and force it to rebuild | 22:04 |
| jeblair | clarkb: before you do that | 22:04 |
| * clarkb waits | 22:04 | |
| jeblair | clarkb: let's make sure that we are currently seeing build failures for it | 22:04 |
| *** patrickeast has quit IRC | 22:04 | |
| *** patrickeast_ is now known as patrickeast | 22:04 | |
| clarkb | ok | 22:05 |
| *** thingee has joined #openstack-infra | 22:05 | |
| *** liam__ has joined #openstack-infra | 22:06 | |
| *** liam__ has quit IRC | 22:06 | |
| clarkb | it is still going | 22:06 |
| jeblair | clarkb: we launched one in dfw before this and it failed with error state | 22:06 |
| jeblair | clarkb: so it's probably worth deleting the dfw image? | 22:06 |
| clarkb | jeblair: sure let me do that | 22:06 |
| jeblair | clarkb: but lets wait and see about iad | 22:06 |
| ayoung | https://review.openstack.org/#/c/129951/ made it through check, but never ended up in a gate queue. We have a current issue with stackforge stuff? | 22:08 |
| *** AlexF has quit IRC | 22:09 | |
| clarkb | ayoung: it got caught in our nodepool service migration (which included a zuul restart) | 22:09 |
| ayoung | clarkb, should I do another reverify? If so, what bug#? | 22:09 |
| clarkb | jeblair: the iad node just switched to delete | 22:10 |
| clarkb | ayoung: no bug number | 22:10 |
| ayoung | reverify no bug | 22:10 |
| clarkb | we stopped requiring bug numbers a while back | 22:10 |
| ayoung | we can do that? | 22:10 |
| clarkb | ayoung: just `recheck` | 22:10 |
| clarkb | the bug numbers individuals supplied were far too inaccurate to be useful data | 22:10 |
| ayoung | its at +2. Will recheck kick it into the merge queue? | 22:10 |
| *** banix has joined #openstack-infra | 22:10 | |
| clarkb | we now focus on elastic recheck instead | 22:10 |
| jeblair | clarkb: it has a +1 from jenkins so i assume we'll need to yank the jenkins vote? | 22:10 |
| clarkb | jeblair: oh right | 22:11 |
| clarkb | or someone can reapprove | 22:11 |
| jeblair | i yanked it | 22:11 |
| clarkb | ayoung: if you recheck it now it will do the correct thing. | 22:11 |
| ayoung | so just "recheck" now? | 22:11 |
| ayoung | or "recheck no bug" | 22:12 |
| ayoung | or does it matter | 22:12 |
| clarkb | ayoung: just `recheck` | 22:12 |
| clarkb | ayoung: the bug stuff is no longer important (it is still accepted but not required) | 22:12 |
| *** banix has quit IRC | 22:12 | |
| jeblair | ayoung: doesn't matter. you could even do "recheck this got caught up in a service restart" too if you want :) | 22:12 |
| ayoung | FIRE IN THE... ahem. Sorry. | 22:12 |
| *** alexpilotti_ has joined #openstack-infra | 22:12 | |
| ayoung | done | 22:12 |
| *** alexpilotti_ has quit IRC | 22:13 | |
| *** thingee has quit IRC | 22:13 | |
| ayoung | were there a couple of those nodepool migration type events? This happened twice to this patch. | 22:13 |
| clarkb | hrm now we are building no devstack-precise nodes | 22:15 |
| clarkb | jeblair: and we just had a timeout | 22:15 |
| *** alexpilotti has quit IRC | 22:15 | |
| jeblair | clarkb: i'll look into the timeout | 22:15 |
| clarkb | ok looks like we just had allocations go through so its not 100% of the time | 22:16 |
| clarkb | gah its building on iad again and I didn't manage to delete images there | 22:16 |
| * clarkb does that now | 22:16 | |
| clarkb | ayoung: yes we ran into trouble on monday | 22:17 |
| ayoung | thanks for the help | 22:17 |
| *** thingee has joined #openstack-infra | 22:17 | |
| *** Masahiro has joined #openstack-infra | 22:18 | |
| *** hdd has quit IRC | 22:21 | |
| jeblair | clarkb: it's currently taking 1-4 seconds to execute the 'status' command from nodepool-dev... i'm going to throw out the idea that maybe the 200 threads it's running may be contributing to starving it on the client side. | 22:21 |
| clarkb | jeblair: that could be | 22:22 |
| clarkb | especially since they all sit on the same cpu | 22:22 |
| *** banix has joined #openstack-infra | 22:22 | |
| jeblair | i've got status running in a loop; if there's a geard performance problem, it should show up there around the same time as a timeouterror | 22:23 |
| *** Masahiro has quit IRC | 22:23 | |
| clarkb | ValueError: invalid literal for int() with base 10: '0build:gate-xstatic-jsencrypt-python33:py3k-precise' | 22:23 |
| clarkb | that looks like a nodepool bug | 22:23 |
| jeblair | clarkb: or a gear bug | 22:24 |
| *** koolhead17 has joined #openstack-infra | 22:24 | |
| jeblair | clarkb: if it reassembled a packet wrong | 22:24 |
| *** hdd has joined #openstack-infra | 22:24 | |
| clarkb | it only happened once | 22:24 |
| *** pc_m has quit IRC | 22:24 | |
| *** rlandy has quit IRC | 22:25 | |
| *** banix has quit IRC | 22:25 | |
| *** esp has joined #openstack-infra | 22:26 | |
| clarkb | other than not getting devstack-precise images to build it seems to be mostly working | 22:27 |
| jeblair | zuul looks pretty happy | 22:27 |
| clarkb | there are allocation grants for devstack-precise but no images building. I think that means we are stuck in a rate limited queue | 22:28 |
| jeblair | another timeouterror | 22:28 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: StoryBoard Streaming API specification https://review.openstack.org/105252 | 22:29 |
| *** mrmartin has quit IRC | 22:29 | |
| jeblair | i just stopped my test program to update it :( oops | 22:29 |
| *** koolhead17 has quit IRC | 22:29 | |
| clarkb | it looked like the admin requset ran right afterwards though | 22:29 |
| clarkb | like it failed then went through | 22:30 |
| *** jungleboyj has joined #openstack-infra | 22:30 | |
| sdague | looks like some unregistered fails in the gate now | 22:34 |
| clarkb | sdague: yup | 22:34 |
| jeblair | i've taken a note of them and will re-enqueue them when the images are available | 22:34 |
| sdague | coolio | 22:34 |
| clarkb | nodepool still refuses to actually build that allocation though | 22:35 |
| jeblair | clarkb: i started a new version of my test script that does the same parsing that nodepool does | 22:35 |
| *** atan8 has quit IRC | 22:35 | |
| jogo | clarkb: yes live migration is working now | 22:35 |
| jeblair | clarkb: yeah, it wants to build it on dfw; i'm not sure why it wants to do that considering there's no current image there | 22:35 |
| jogo | clarkb: (was on a plane for the last 5 hours) | 22:36 |
| clarkb | jeblair: huh I wonder if it sees the building image as valid enough | 22:36 |
| hashar | hey folks! | 22:37 |
| clarkb | that image is almost done building though | 22:37 |
| jeblair | hashar: hi! | 22:37 |
| clarkb | so hopefully we get moving real soon now | 22:37 |
| hashar | is there a doc for storyboard that explains its markup? I would like to paste a stacktrace with a <pre> :] | 22:37 |
| krotscheck | hashar: There is no markup in storyboard. | 22:37 |
| krotscheck | You are welcome to add it :) | 22:38 |
| hashar | hehe | 22:38 |
| hashar | krotscheck: thanks for confirming at least! | 22:38 |
| jeblair | is there a story about that? | 22:38 |
| clarkb | downloading the openvz image | 22:38 |
| clarkb | sdague: what can we do about killing that with fire? | 22:38 |
| clarkb | I don't think openvz is goin to happen. docker wins | 22:38 |
| hashar | zaro: I have filled a story about the Jenkins Gearman plugin maintaining executors in a deadlock https://storyboard.openstack.org/#!/story/2000030 . Was long overdue sorry. | 22:40 |
| jeblair | krotscheck: erm, i just got logged out while doing a search | 22:40 |
| hashar | jeblair: while you are around, I found out zuul-cloner does not support being triggered by a ref-update event ( zuul.model.Ref ). It would complains about missing parameters such as ZUUL_BRANCH | 22:41 |
| jeblair | hashar: ah, we should test/fix that, as well as a timer event | 22:41 |
| hashar | jeblair: would get a patch for it soonish since it is blocking me. Might impact whatever job you have | 22:41 |
| jeblair | i know i'll be running into the timer event issue myself soon (need that to replace devstack-gate) | 22:41 |
| fungi | back and catching up on what i've missed | 22:42 |
| *** weshay has quit IRC | 22:42 | |
| *** banix has joined #openstack-infra | 22:42 | |
| hashar | and I am not sure how to handle the branch fallback for a release tag | 22:43 |
| hashar | i.e. you have branches 1.x and 2.x , a tag for 1.0.1, we would want to fallback to 1.x probably | 22:43 |
| clarkb | hashar: a tag can discover what branches it is on | 22:44 |
| sdague | clarkb: yeh... we can do some image cleanup in devstack post holiday, there are a few things we can clean up now I think | 22:44 |
| clarkb | hashar: maybe do that then pick the first one? | 22:44 |
| jeblair | hashar: in our use case, we have no use for a fallback | 22:44 |
| sdague | did the openvz one fail, or it's just annoying? | 22:44 |
| clarkb | sdague: its just annoying | 22:44 |
| clarkb | sdague: it may eventually fail but I am still witing on it | 22:44 |
| jeblair | clarkb: but in this case we're dealing with a project without the tag | 22:44 |
| jeblair | clarkb, hashar: so i think maybe either fallback to master or explicitly error may be the best options | 22:45 |
| sdague | though with the cirros images, maybe we should cache them in infra | 22:45 |
| clarkb | jeblair: oh right in other projects | 22:45 |
| clarkb | sdague: we do. this is that caching | 22:45 |
| *** ociuhandu has joined #openstack-infra | 22:45 | |
| clarkb | dib makes it much smarter and is why I am working very hard to make dib go | 22:45 |
| *** alexpilotti has joined #openstack-infra | 22:45 | |
| hashar | clarkb: if you have the magic git command to figure out the branch of a tag, I am willing to learn :] | 22:46 |
| *** Ryan_Lane has joined #openstack-infra | 22:46 | |
| fungi | jeblair: clarkb: regarding bugs for group membership, i think i originally suggested it as a signaling mechanism for people who don't irc, but ultimately that may be roughly nobody (within an error margin of we don't care) | 22:47 |
| *** EmilienM has quit IRC | 22:47 | |
| hashar | at worth, we could have a python function that attempts to craft a branch name out of the tag (i.e. tag '1.0.1' -> branch '1.x' ) but that is awkward | 22:47 |
| clarkb | hashar: git branch --contains | 22:47 |
| clarkb | says the google | 22:47 |
| jeblair | hashar: what is your use case for a fallback in ref-updated? | 22:47 |
| *** EmilienM has joined #openstack-infra | 22:47 | |
| *** banix has quit IRC | 22:49 | |
| hashar | jeblair: for mediawiki we maintain old versions. Currently we are preparing 1.24.0 out of branch REL1_24 | 22:50 |
| *** achanda has joined #openstack-infra | 22:50 | |
| hashar | and we back port security patches to old version, for example 1.19.24 . So the tag 1.19.24 would need the repo to fallback to REL1_19 | 22:50 |
| *** stevemar has quit IRC | 22:50 | |
| clarkb | oh hey we are now on to a vmware image | 22:51 |
| *** jamielennox|away is now known as jamielennox | 22:51 | |
| jeblair | hashar: what does your ref-updated job do? | 22:51 |
| clarkb | wow that was fast now on to fedora | 22:51 |
| hashar | clarkb: yeah that works :-] thank you! | 22:52 |
| clarkb | image is snapshotting now in rax-dfw | 22:53 |
| clarkb | I think | 22:53 |
| fungi | clarkb: jeblair: so after the last two nodepool cut-overs we similarly saw it repeatedly fail to build devstack-precise nodes in rax-iad | 22:53 |
| *** dannywilson has quit IRC | 22:53 | |
| fungi | different images each time too | 22:54 |
| clarkb | fungi: Ithink that was related to rax issues that are supposed to be fixed now according to johnthetubaguy | 22:54 |
| fungi | mmm | 22:54 |
| fungi | okay | 22:54 |
| bknudson | I think we're close on https://review.openstack.org/#/c/135903/ -- now getting NOT_REGISTERED | 22:54 |
| clarkb | or at least mitigated | 22:54 |
| hashar | jeblair: they create tarballs for the version released, so we probably don't need multiples repositories there | 22:54 |
| hashar | jeblair: and publish the documentation for the version. Ditto we might not need multiple repos | 22:54 |
| fungi | we weren't getting image-related errors when i was digging into it though, just nova boot ending with the instance in error state | 22:55 |
| *** belmoreira has quit IRC | 22:55 | |
| *** yfried|afk is now known as yfried | 22:55 | |
| *** achanda has quit IRC | 22:55 | |
| fungi | the image issues were different... nova boot would respond with something like "requested image is not active" | 22:55 |
| hashar | jeblair: I am just thinking out loud sorry. Will follow up next week once I have some code / proof of concept | 22:55 |
| fungi | and i think those are the ones johnthetubaguy said should be solved now | 22:56 |
| clarkb | fungi: I got the impression it was all related but I may be wrong | 22:56 |
| fungi | anyway, looks like i missed all the fun? any cleanup i can help with still? | 22:56 |
| clarkb | well it looks like nodepool may not be booting enough nodes right now? | 22:56 |
| fungi | ahh | 22:56 |
| *** Ark has joined #openstack-infra | 22:56 | |
| fungi | having a look to see if anything looks obviously wrong to me | 22:57 |
| *** Ark is now known as Guest51075 | 22:57 | |
| clarkb | jeblair: ^ | 22:57 |
| fungi | 2014-11-26 22:28:46,686 ERROR nodepool.GearmanClient: Exception while listing functions | 22:57 |
| fungi | that was half an hour ago | 22:58 |
| jeblair | fungi: yeah, we've seen a couple TimeoutErrors and one error in parsing the output (!) | 22:58 |
| *** banix has joined #openstack-infra | 22:58 | |
| jeblair | i have a test running in a loop on the same host to try to catch it outside of nodepool | 22:58 |
| fungi | ahh, yeah the misidentifed octal value | 22:58 |
| fungi | that one was pretty amusing | 22:59 |
| *** dannywilson has joined #openstack-infra | 22:59 | |
| clarkb | it seems to have gotten us into a place where we only boot min ready nodes | 22:59 |
| hashar | also I found a rather nasty bug in Zuul that would make it keep changes in its queue indefinitely whenever it lost connection with Gearman | 22:59 |
| fungi | i'm analyzing the demand details | 22:59 |
| *** dannywilson has quit IRC | 22:59 | |
| hashar | https://review.openstack.org/#/c/128921/ <-- working fine for me for a few weeks now:] | 22:59 |
| *** dannywilson has joined #openstack-infra | 23:00 | |
| jeblair | clarkb, fungi: i suspect this may be related: https://review.openstack.org/#/c/136910/ | 23:00 |
| jeblair | clarkb, fungi: probably we got stuck after that parse error | 23:00 |
| *** Guest51075 has quit IRC | 23:01 | |
| fungi | highly likely because i don't see it reporting demand info | 23:01 |
| clarkb | jeblair: I thought that was applied on nodepool-dev? | 23:01 |
| jeblair | hrm lemme check | 23:02 |
| clarkb | rax-dfw image is ready now | 23:02 |
| jeblair | clarkb: oh, so it is. :/ | 23:02 |
| clarkb | ya its installed according to pip freeze | 23:02 |
| fungi | but yeah, the allocation requests show a demand of 0 for all labels | 23:02 |
| jeblair | clarkb: ah, but the exception happened elsewhere | 23:02 |
| *** jungleboyj has quit IRC | 23:03 | |
| clarkb | devstack-precise node building on rax-dfw now | 23:03 |
| *** jungleboyj has joined #openstack-infra | 23:03 | |
| jeblair | clarkb, fungi: i'm going to disable puppet on nodepool and configure some more verbose logging | 23:04 |
| fungi | jeblair: sound like a plan | 23:04 |
| clarkb | wfm | 23:04 |
| *** yfried is now known as yfried|afk | 23:05 | |
| jeblair | restarting nodepool | 23:05 |
| *** ddieterly has quit IRC | 23:05 | |
| fungi | yeah, i find no non-zero deficits after that parse error, but before it there was evidence of non-zero deficit calculations | 23:06 |
| *** ddieterly has joined #openstack-infra | 23:06 | |
| *** MarkAtwood has quit IRC | 23:08 | |
| *** camunoz_ has joined #openstack-infra | 23:09 | |
| *** camunoz_ has quit IRC | 23:09 | |
| *** camunoz has quit IRC | 23:09 | |
| clarkb | the deficits still seem to be 0 | 23:10 |
| *** SumitNaiksatam has quit IRC | 23:10 | |
| fungi | after some more careful parsing of the log, i do see a little (one here, one there) in the log after that traceback, but not the values i would expect | 23:10 |
| *** SumitNaiksatam has joined #openstack-infra | 23:10 | |
| *** ddieterly has quit IRC | 23:11 | |
| *** kgiusti has joined #openstack-infra | 23:11 | |
| jeblair | hrm, there must be something else going on... | 23:11 |
| *** patrickeast has quit IRC | 23:11 | |
| jeblair | the zuul job queue is quite high | 23:11 |
| clarkb | ya | 23:11 |
| clarkb | something server side maybe? | 23:12 |
| jeblair | merger:merge4108 | 23:12 |
| *** ryanpetrello has joined #openstack-infra | 23:12 | |
| clarkb | we have a devstack-precise node now fwiw | 23:14 |
| clarkb | zm01 definitely seems like its stuck | 23:14 |
| fungi | 96 established gearman connections on zuul, according to netstat. that seems relatively high? | 23:14 |
| clarkb | fungi: there is one per jenkins slave | 23:15 |
| jeblair | fungi: there's one for each worker, so that's about right right now | 23:15 |
| fungi | oh, got it | 23:15 |
| *** dannywilson has quit IRC | 23:16 | |
| clarkb | looks like zm01 stalled slightly before we restarted nodepool so I don't think that is related | 23:16 |
| jeblair | clarkb, fungi: okay, i'd like to restart zuul and zm01 with verbose debugging | 23:17 |
| fungi | tcpdump shows no packets on its gearman connection (sniffing at both ends of the socket) | 23:17 |
| *** Sukhdev has quit IRC | 23:17 | |
| jeblair | see what we can see there, and probably shortly thereafter, start the rollback process | 23:17 |
| *** dannywilson has joined #openstack-infra | 23:17 | |
| clarkb | jeblair: rgr | 23:18 |
| fungi | okie dokey | 23:18 |
| fungi | now i see that socket with traffic on it | 23:19 |
| jeblair | just restarted zm01 with debugging | 23:19 |
| *** ryanpetrello has quit IRC | 23:19 | |
| jeblair | stopped zuul | 23:19 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: StoryBoard Permissions Specifications https://review.openstack.org/103667 | 23:19 |
| fungi | yep, different source port for the zm01 end obviously | 23:19 |
| fungi | jeblair: worth noting, right after you restarted zm01, some merges were performed and a couple of jobs started | 23:20 |
| fungi | so that seemed to have dislodged whatever was hung | 23:20 |
| clarkb | we are now building all the nodes | 23:20 |
| jeblair | fungi: yeah, i was hoping that it would just start one and stop again :/ | 23:20 |
| *** JayJ has quit IRC | 23:21 | |
| jeblair | zuul is up; re-enquing | 23:21 |
| fungi | blammo. that's a lot of gearman ;) | 23:21 |
| jeblair | yeah, and i think of it as sparse :) | 23:22 |
| *** andreykurilin_ has quit IRC | 23:22 | |
| fungi | well, this is a relatively slow time | 23:22 |
| *** bswartz has joined #openstack-infra | 23:23 | |
| clarkb | did that unstick the other mergers? /me looks | 23:24 |
| clarkb | yup looks like 06 moved (I had checked it previously) | 23:24 |
| jeblair | hrm, 01 looks idle | 23:25 |
| clarkb | ya 06 also looks like it may have gotten stuck | 23:25 |
| clarkb | so maybe we have the data you wanted? | 23:25 |
| jeblair | oh it just moved again | 23:26 |
| jeblair | i think the mergers may have worked through the queue and are actually just genuinely idle now | 23:26 |
| *** gokrokve has quit IRC | 23:27 | |
| fungi | though it looks like we still haven't gotten any devstack-precise workers to build, hence the not_registered | 23:27 |
| clarkb | fungi: we had one | 23:28 |
| fungi | well, at least no new ones since the last zuul restart | 23:28 |
| clarkb | 36856 | 23:28 |
| clarkb | oh right only unused nodes will register jobs | 23:28 |
| clarkb | well we have some queued | 23:28 |
| fungi | in a variety of providers/regions now that there's a calculated non-minimum deficit | 23:29 |
| fungi | so, yeah, should be solved in short order | 23:29 |
| fungi | no new gearmanclient timeouts in nodepool-dev's nodepoold yet | 23:33 |
| clarkb | I wonder if we are stuck again though | 23:33 |
| clarkb | the status graph isn't a massive spike like I expected | 23:33 |
| clarkb | and no nodes building | 23:33 |
| *** jungleboyj has quit IRC | 23:33 | |
| jeblair | clarkb: i think that just changed | 23:34 |
| clarkb | oh wow yup | 23:34 |
| fungi | the graph may be behind | 23:34 |
| fungi | 822 nodes in the list | 23:34 |
| jeblair | nodepool was behind at the time | 23:34 |
| clarkb | fungi: well I checked manually too | 23:34 |
| clarkb | building list was empty | 23:34 |
| clarkb | then like 20 seconds later it was massive | 23:34 |
| fungi | ahh, i checked right after it skyrocketed i guess | 23:34 |
| jeblair | clarkb: okay now i'm starting to wonder if the mergers are stuck again | 23:35 |
| clarkb | jeblair: zm06 has not moved since ~14 minutes ago | 23:36 |
| *** camunoz has joined #openstack-infra | 23:36 | |
| clarkb | according to zuul staus list we have newer queued changes than that | 23:36 |
| fungi | and the pending count for merger:merge is rising | 23:36 |
| jeblair | 2014-11-26 23:28:26,606 DEBUG gear.ServerConnection: Queuing packet to <gear.ServerConnection 0x1786950 name: Zuul Merger host: 104.130.157.229 port: 44269>: <gear.Packet 0x7fc5b6115950 type: NOOP> | 23:36 |
| jeblair | yep, it wanted to send a noop that zm01 did not receive | 23:37 |
| *** resker has joined #openstack-infra | 23:38 | |
| *** resker has quit IRC | 23:38 | |
| *** fandi has quit IRC | 23:38 | |
| *** camunoz is now known as camunoz_mtg | 23:39 | |
| *** hdd has quit IRC | 23:40 | |
| clarkb | jeblair: should I start rollback prep? | 23:41 |
| *** shashankhegde has joined #openstack-infra | 23:41 | |
| jeblair | clarkb: i'd like another minute to see if i can narrow this down further, but we should do that soon | 23:42 |
| *** banix has quit IRC | 23:42 | |
| *** esker has quit IRC | 23:42 | |
| clarkb | sounds good | 23:42 |
| *** SumitNaiksatam has quit IRC | 23:42 | |
| *** banix has joined #openstack-infra | 23:42 | |
| rediskin | anteaya: hi there | 23:44 |
| *** otter768 has joined #openstack-infra | 23:44 | |
| *** baoli has quit IRC | 23:44 | |
| rediskin | anteaya: i heard you can help with gerrit account for third-party ci | 23:44 |
| *** baoli has joined #openstack-infra | 23:45 | |
| rediskin | anteaya: i sent email a week ago, but still no response T_T | 23:45 |
| jeblair | fungi: any chance you have a tcpdump between zuul and zm01? | 23:45 |
| clarkb | rediskin: we are currently trying to rework the process | 23:45 |
| fungi | jeblair: not one from the time it hung | 23:45 |
| fungi | i had stopped it prior to then, and restarted it after you commented | 23:46 |
| clarkb | rediskin: unfortunately we have been very busy this week dealin with fires and its prompted us to step up work on making this more self service | 23:46 |
| rediskin | clarkb: does this mean i will wait for a month? -_- | 23:46 |
| fungi | jeblair: though if you want to try to retrigger the hang, i can set up to get full-size packets recorded at both ends | 23:46 |
| clarkb | rediskin: hopefully not that long, you could help us review the changes and otherwise participate in the planningaround that | 23:46 |
| rediskin | clarkb: ok, thanks | 23:47 |
| clarkb | rediskin: https://review.openstack.org/#/c/137240/ | 23:47 |
| jeblair | fungi, clarkb: another run would help, but i think we should go ahead and roll back | 23:47 |
| clarkb | ok I can stop new nodepool now | 23:47 |
| clarkb | anything you want to do before I do that? | 23:48 |
| jeblair | hopefully i can stare at that bit of code and guess at what might have gone wrong, maybe reproduce | 23:48 |
| jeblair | clarkb: nope, go ahead | 23:48 |
| fungi | let's do that for now i guess | 23:48 |
| jeblair | i'll stop zuul | 23:48 |
| fungi | i can start deleting nodepool-dev nodes | 23:48 |
| fungi | ready nodes first i guess | 23:49 |
| *** otter768 has quit IRC | 23:49 | |
| clarkb | fungi: I just did it concurrently | 23:49 |
| clarkb | it will fail to delete some nodes because they are already deleted but thats fine | 23:49 |
| fungi | yeah | 23:49 |
| *** SumitNaiksatam has joined #openstack-infra | 23:49 | |
| clarkb | I have the commands all ready though | 23:49 |
| fungi | i'll do it in parallel but have it delete the ready nodes before the others | 23:49 |
| clarkb | kk | 23:50 |
| jeblair | i have enabled puppet and had it reset the config files on nodepool-dev, zuul, and zm01 | 23:50 |
| clarkb | when the ready nodes drops to 0 I will start old nodepool | 23:50 |
| jeblair | i'll have a standalone geard in a minute | 23:51 |
| *** rediskin has left #openstack-infra | 23:51 | |
| jeblair | okay, gear has been downgraded on zuul to the locally patched version (with the 300 sec timeout) and geard is running standalone; ready for old nodepool | 23:53 |
| jeblair | restarted zm01 with the old log config | 23:53 |
| *** hashar has quit IRC | 23:54 | |
| clarkb | we are down to 28 ready nodes | 23:54 |
| clarkb | from ~80 | 23:55 |
| fungi | these last 10 are pesky | 23:56 |
| *** banix has quit IRC | 23:56 | |
| fungi | dfw and ord really don't like to take on api calls | 23:56 |
| clarkb | should we just delete them from the jenkins masters? | 23:57 |
| fungi | they're starting to dwindle now | 23:57 |
| *** marun has quit IRC | 23:57 | |
| fungi | there we are | 23:58 |
| fungi | all gone | 23:59 |
| fungi | working on the non-ready nodes now | 23:59 |
| clarkb | cool I am starting old nodepool now | 23:59 |
| clarkb | jeblair: ready? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!