| *** doug-fish has joined #openstack-infra | 00:01 | |
| jeblair | clarkb, fungi: the voip.ms account for asterisk was set up with my individual professional identity (as i recall, they required PII in order to get a PSTN number). since the attached address no longer works, i don't think it's appropriate for me to renew it anymore. can one of you work on a replacement account and i will close mine? | 00:06 |
|---|---|---|
| *** sflanigan has joined #openstack-infra | 00:09 | |
| *** sflanigan has joined #openstack-infra | 00:09 | |
| clarkb | jeblair: what we currently have is a USA DID number? so would need to sign up a new account and pay for one of those? | 00:12 |
| * clarkb is reading docs to determine what is necessary | 00:14 | |
| openstackgerrit | Merged openstack/requirements: Bump up eventlet to at least 0.18.1 https://review.openstack.org/275452 | 00:15 |
| clarkb | I can probably do that tomorrow morning, though currently distracted by home stuff and image builds | 00:16 |
| *** rfolco has quit IRC | 00:17 | |
| *** daemontool has joined #openstack-infra | 00:17 | |
| *** Swami has quit IRC | 00:18 | |
| *** Swami has joined #openstack-infra | 00:18 | |
| Sam-I-Am | clarkb: oddly enough, something is blocking 9418 all of the sudden :/ | 00:18 |
| Sam-I-Am | intermittently to some extent | 00:18 |
| *** rbrndt_ has quit IRC | 00:18 | |
| *** armax has quit IRC | 00:19 | |
| *** armax_ has joined #openstack-infra | 00:19 | |
| nibalizer | Sam-I-Am: 29418? | 00:19 |
| clarkb | Sam-I-Am: are you in rax-ord by chance? that was where we saw the one job fail | 00:20 |
| openstackgerrit | Merged openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 00:20 |
| *** rfolco has joined #openstack-infra | 00:20 | |
| *** rfolco has quit IRC | 00:20 | |
| Sam-I-Am | nibalizer: isnt git 9418? | 00:20 |
| clarkb | Sam-I-Am: it is, and gerrit is 29418 for its ssh server | 00:20 |
| nibalizer | oh i have no idea | 00:20 |
| nibalizer | i was referncing the gerrit port | 00:20 |
| Sam-I-Am | this is just a stright up git clone | 00:21 |
| Sam-I-Am | git://git.openstack.org/repo | 00:21 |
| clarkb | yes thta is port 9418 | 00:21 |
| nibalizer | TIL | 00:21 |
| fungi | okay, back now | 00:21 |
| Sam-I-Am | clarkb: this host is at ibm | 00:21 |
| *** doug-fish has quit IRC | 00:22 | |
| *** jpr has joined #openstack-infra | 00:22 | |
| *** sabeen2 has quit IRC | 00:23 | |
| *** doug-fish has joined #openstack-infra | 00:24 | |
| *** doug-fish has quit IRC | 00:25 | |
| *** doug-fish has joined #openstack-infra | 00:26 | |
| *** doug-fish has quit IRC | 00:27 | |
| *** eil397 has quit IRC | 00:27 | |
| *** thorst has joined #openstack-infra | 00:29 | |
| *** doug-fish has joined #openstack-infra | 00:30 | |
| fungi | Sam-I-Am: oh, i think i remember some former ibmers talking about being unable to use certain git protocols | 00:30 |
| *** ybathia has quit IRC | 00:31 | |
| *** thorst_ has joined #openstack-infra | 00:31 | |
| Sam-I-Am | it worked fine for a few weeks | 00:31 |
| fungi | jeblair: consensus is that jbryce is the only one who will know what happened around decemberish regarding that mailbox, and he's not around a computer at the moment but i'll follow up as soon as he is | 00:31 |
| openstackgerrit | Merged openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 00:31 |
| anteaya | dims_: thank you | 00:31 |
| Sam-I-Am | today it works too.. sort of, sometimes. | 00:31 |
| fungi | Sam-I-Am: i consider networks which work "sometimes" to really not work at all | 00:32 |
| Sam-I-Am | fungi: lol agree. | 00:32 |
| Sam-I-Am | changing it to http so far works | 00:32 |
| tonyb | Sam-I-Am: that was a common issue there | 00:32 |
| fungi | Sam-I-Am: also i think it was actually git over https which ibm was breaking somehow with their web proxies | 00:32 |
| Sam-I-Am | might just change git_base to use http | 00:32 |
| *** bpokorny_ has quit IRC | 00:33 | |
| fungi | at least back in the timeframe when sdague and mtreinish were still working there | 00:33 |
| *** bpokorny has joined #openstack-infra | 00:33 | |
| *** thorst has quit IRC | 00:34 | |
| anteaya | I share fungi's definition of minimum expectation of networks | 00:34 |
| *** thorst_ has quit IRC | 00:34 | |
| *** thorst has joined #openstack-infra | 00:35 | |
| tonyb | lifeless: I don't want to be a pain but can you look at my comment on https://bugs.launchpad.net/python-fixtures/+bug/1542984 If I *have* to patch 30+ projects I'd like to get started. | 00:35 |
| openstack | Launchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New] | 00:35 |
| mtreinish | fungi: ?? | 00:36 |
| *** tpsilva has quit IRC | 00:36 | |
| mtreinish | oh, ibm networking. Yeah, they break stuff all the time | 00:37 |
| *** sshnaidm has quit IRC | 00:38 | |
| *** BobBall_ has joined #openstack-infra | 00:39 | |
| *** Spyware has joined #openstack-infra | 00:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Experimental jobs for testing against oslo.* from master - Take #2 https://review.openstack.org/277086 | 00:39 |
| *** BobBall_ is now known as Guest15184 | 00:39 | |
| *** thorst has quit IRC | 00:39 | |
| *** doug-fish has quit IRC | 00:40 | |
| anteaya | mtreinish tonyb how was lca? | 00:41 |
| tonyb | anteaya: I didn't go :( | 00:42 |
| *** BobBall has quit IRC | 00:42 | |
| anteaya | oh yeah, you told me and I forgot | 00:42 |
| anteaya | sigh I can't remember anything any more | 00:42 |
| Sam-I-Am | clarkb: i found out you might be able to see this ip in your logs | 00:42 |
| anteaya | :( | 00:42 |
| tonyb | anteaya: sign of a busy mind. | 00:42 |
| anteaya | or something | 00:42 |
| anteaya | sign of something | 00:43 |
| fungi | Sam-I-Am: we have logs. which source address? | 00:43 |
| ianw | mordred: what's happening with a glean release? didn't we do one? | 00:43 |
| Sam-I-Am | fungi: 9.30.183.45 | 00:43 |
| *** mingyu has quit IRC | 00:43 | |
| *** mingyu has joined #openstack-infra | 00:44 | |
| *** ajmiller has quit IRC | 00:44 | |
| fungi | Sam-I-Am: checking now | 00:44 |
| *** Jeffrey4l has joined #openstack-infra | 00:44 | |
| *** doug-fish has joined #openstack-infra | 00:45 | |
| *** ybathia has joined #openstack-infra | 00:47 | |
| fungi | Sam-I-Am: haproxy logs (for at least the last ~20 hours) on git.openstack.org show no connections whatsoever from that client | 00:47 |
| fungi | Sam-I-Am: is it possible you're coming from an ipv6 address? | 00:48 |
| Sam-I-Am | weiiird. what about as http? | 00:48 |
| fungi | Sam-I-Am: otherwise, chances are it's coming from a proxy | 00:48 |
| Sam-I-Am | there's no public v6 on this box | 00:48 |
| fungi | Sam-I-Am: i checked all protocols. nothing from that ip address (git, http or https) | 00:48 |
| Sam-I-Am | ok, back to the drawing board. was told the ip should see the world without anything funky in-between, but i guess thats not the case. | 00:49 |
| Sam-I-Am | my http attempt just died too | 00:49 |
| Sam-I-Am | just in a different way | 00:49 |
| *** sshnaidm has joined #openstack-infra | 00:50 | |
| fungi | Sam-I-Am: i was about to suggest pinging git.o.o so i could spot your address, but... lots of unsolicited continuous pings to that server | 00:51 |
| Sam-I-Am | fungi: i can send you a certain size ping or something | 00:52 |
| *** pahuang has joined #openstack-infra | 00:52 | |
| fungi | Sam-I-Am: tried going to one of those sites like whatsmyipaddress.net? | 00:53 |
| Sam-I-Am | i wonder if that works via curl | 00:53 |
| mgagne | curl slee.ca | 00:53 |
| mgagne | shows your IP | 00:53 |
| mgagne | curl compatible | 00:53 |
| fungi | that's a short one--maybe i'll remember it | 00:53 |
| Sam-I-Am | fungi: 32.97.110.58 | 00:53 |
| Sam-I-Am | mgagne: cool that was nice | 00:53 |
| mgagne | old employee website, used for debug =) | 00:53 |
| fungi | but yeah, in theory if your web access is being proxied then that's the address of the proxy | 00:54 |
| fungi | Sam-I-Am: and indeed, that address has been hitting git.o.o pretty continuously (https that i'm seeing at first glance) so probably a heavily used proxy and not just your traffic | 00:55 |
| Sam-I-Am | fungi: awesome :/ | 00:56 |
| openstackgerrit | Merged openstack/requirements: Only install mock on Python < 3.3 https://review.openstack.org/268945 | 00:56 |
| fungi | Sam-I-Am: slightly deeper analysis says that source ip address has initiated tons of https and also some http to git.o.o | 00:57 |
| fungi | Sam-I-Am: though no git protocol | 00:57 |
| openstackgerrit | Merged openstack-infra/system-config: Collapse groups into one group https://review.openstack.org/277609 | 00:57 |
| *** Jeffrey4l has quit IRC | 00:57 | |
| fungi | i.e. no 9418/tcp | 00:57 |
| Sam-I-Am | fungi: i wonder if its proxying http/https .. 9418 was going direct until someone turned it off. | 00:57 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-statusbot: Allow statusbot to take an array of auth nicks https://review.openstack.org/274252 | 01:00 |
| nibalizer | crinkle: ok it is time | 01:00 |
| nibalizer | well it has to apply to puppetmaster first | 01:00 |
| nibalizer | so more waiting | 01:00 |
| fungi | Sam-I-Am: entirely likely | 01:00 |
| Sam-I-Am | fungi: thx for the help. time to dig in a little further... and hope changing to http/https works. | 01:01 |
| Sam-I-Am | it got further this time, then died in another mysterious way | 01:01 |
| crinkle | nibalizer: eep | 01:01 |
| *** ybathia has quit IRC | 01:01 | |
| nibalizer | crinkle: controller or compute first? | 01:03 |
| crinkle | nibalizer: controller first | 01:03 |
| nibalizer | compute seems like we'd have some ... leeway if it goes sideways | 01:03 |
| nibalizer | like we can oops on a compute and we still have a bunch of computes | 01:03 |
| crinkle | nibalizer: i guess it doesn't really matter for this since everything is already set up, but in general controller would have to go first | 01:04 |
| *** cznewt has quit IRC | 01:04 | |
| nibalizer | ya | 01:04 |
| *** cznewt has joined #openstack-infra | 01:05 | |
| clarkb | fungi: Sam-I-Am connections could be dying between source and proxy too | 01:05 |
| *** Spyware has left #openstack-infra | 01:05 | |
| clarkb | fungi: I didn't see any errors in the grafana graphs so don't expect that proxy -> git.o.o is having lots of trouble | 01:05 |
| *** pvaneck has quit IRC | 01:06 | |
| *** FallenPegasus has joined #openstack-infra | 01:07 | |
| clarkb | greghaynes: http://paste.openstack.org/show/486350/ that upload failed due to an timeout, but database wasn't cleaned up. I used the image-upload command so possible that we don't handle that properly there? I have to go to dinner soon but can dig into that later | 01:08 |
| fungi | jeblair: jbryce says he cleaned up accounts that had no logins in months, but it sounds like if you had it set up as a forward he might have missed that | 01:08 |
| *** hashar has quit IRC | 01:09 | |
| clarkb | greghaynes: that particular image still shows up as building in the db | 01:09 |
| Sam-I-Am | fungi: interesting that you dont see any git protocol... even over the last week? | 01:09 |
| fungi | Sam-I-Am: was just looking at the current log, so last ~20 hours. i can grep rotated logs | 01:09 |
| *** ccrouch has quit IRC | 01:09 | |
| Sam-I-Am | this seems to have started today sometime | 01:09 |
| nibalizer | crinkle: still waiting for the change to have effect on puppetmaster.o.o | 01:10 |
| crinkle | nibalizer: ok | 01:11 |
| fungi | Sam-I-Am: no connections of any sort directly from 9.30.183.45 to git.openstack.org in all of february from what i can tell. should i look back further? | 01:12 |
| Sam-I-Am | fungi: what about from that 32.x ip? | 01:12 |
| *** doug-fish has quit IRC | 01:13 | |
| fungi | Sam-I-Am: tons from 32.97.110.58 but only http and https, so i'm guessing that's a shared web proxy | 01:14 |
| *** weshay_xchat has joined #openstack-infra | 01:14 | |
| *** thorst has joined #openstack-infra | 01:15 | |
| *** Daisy has joined #openstack-infra | 01:16 | |
| Sam-I-Am | bleh. being told there's no proxy, just nat. however you're revealing something different. | 01:16 |
| nibalizer | crinkle: ok we are in business | 01:17 |
| fungi | Sam-I-Am: dunno, former ibmers told me fun stories about web proxies breaking git over https, but maybe the network's been redone in the past couple years | 01:17 |
| *** slogan621 has quit IRC | 01:17 | |
| nibalizer | crinkle: several compute hosts are unreachable: http://paste.openstack.org/show/486351/ | 01:17 |
| nibalizer | known issue? | 01:18 |
| crinkle | nibalizer: possibly | 01:18 |
| crinkle | nibalizer: depends on if those hosts match these uuids https://etherpad.openstack.org/p/infra-cloud-inventory-status | 01:19 |
| Sam-I-Am | fungi: are there any other non-http/https/git ports on g.o.o that i can poke at? | 01:19 |
| clarkb | Sam-I-Am: ssh | 01:20 |
| clarkb | but otherwise those are the only services exposed | 01:20 |
| Sam-I-Am | clarkb: interesting. i cant see ssh either. | 01:20 |
| *** Daisy has quit IRC | 01:20 | |
| fungi | Sam-I-Am: ssh (22/tcp) and snmp (161/udp) should be reachable | 01:20 |
| *** Sukhdev has quit IRC | 01:20 | |
| Sam-I-Am | fungi: yep, 22 just hangs on syn | 01:21 |
| nibalizer | crinkle: okay I have deemed compute021 to be our guinea pig | 01:21 |
| fungi | no, wait, snmp is filtered to only be allowed from our cacti host i think | 01:21 |
| fungi | looking at iptables | 01:21 |
| fungi | icmp ping should be working though for a non-tcp example | 01:22 |
| Sam-I-Am | fungi: i can get to 22 from a non-lab box | 01:22 |
| Sam-I-Am | also noticed that i cant ssh anywhere from my lab box | 01:22 |
| *** slogan621 has joined #openstack-infra | 01:22 | |
| Sam-I-Am | well, anywhere on the internets | 01:22 |
| crinkle | nibalizer: okay | 01:23 |
| Sam-I-Am | fungi: i have this feeling the proxy works, but whatever runs nat broke. | 01:25 |
| Sam-I-Am | which explains http/https | 01:25 |
| nibalizer | crinkle: huh | 01:26 |
| fungi | Sam-I-Am: plausible | 01:26 |
| nibalizer | crinkle: it caused compute021 to submit a cert request | 01:26 |
| Sam-I-Am | i can ping g.o.o though | 01:26 |
| crinkle | nibalizer: huh | 01:26 |
| openstackgerrit | Khai Do proposed openstack-infra/gerritlib: Initialize the logger https://review.openstack.org/277654 | 01:27 |
| crinkle | nibalizer: is it using https://review.openstack.org/#/c/234535 ? that was written back when we were still using puppetmaster | 01:27 |
| nibalizer | no | 01:28 |
| *** amitgandhinz has joined #openstack-infra | 01:28 | |
| nibalizer | i think the issue is we're not sourceing the groupvars | 01:28 |
| nibalizer | so it is falling back to default which is puppetmaster | 01:28 |
| crinkle | nibalizer: so you can fix? | 01:29 |
| nibalizer | i think so | 01:29 |
| nibalizer | i mean i have an educated guess | 01:29 |
| *** EricGonc_ has quit IRC | 01:29 | |
| nibalizer | crinkle: okay i have more ducks in rows | 01:31 |
| nibalizer | trying again | 01:31 |
| clarkb | quack quack | 01:31 |
| nibalizer | tail -f /var/log/syslog on compute021 is probably the best way to keep up | 01:31 |
| nibalizer | or i can screen on the puppetmaster if any rooters wnat to join | 01:31 |
| nibalizer | crinkle: okay new error is it is cowardly refusing to wipe out the current hiera dircetory structure | 01:33 |
| ianw | clarkb: another yak shaving adventure. seems that /var/run isn't symlinked to /run on the f23 images. this puts the system in that awesome state of "mostly working" | 01:33 |
| nibalizer | the stuff on compute 021 differes in unimportant ways from whats expected | 01:33 |
| nibalizer | so im just blowing it away and then it should work | 01:33 |
| nibalizer | trying again | 01:33 |
| nibalizer | its copying over the puppet modules | 01:34 |
| ianw | nibalizer: if you get a chance; quick fix to the nodepool log cleanup; pretty straight-forward : https://review.openstack.org/#/c/277099/ | 01:34 |
| *** mtanino has quit IRC | 01:34 | |
| crinkle | nibalizer: syslog is not telling me about these things so thank you for the live updates | 01:34 |
| nibalizer | oshi its running puppet | 01:34 |
| crinkle | oh now it's gotten to puppet so it's telling me | 01:34 |
| nibalizer | ! | 01:35 |
| *** Sukhdev has joined #openstack-infra | 01:35 | |
| *** gildub has quit IRC | 01:35 | |
| nibalizer | cool | 01:35 |
| nibalizer | okay so its done | 01:35 |
| nibalizer | it failed to submit to puppetdb but thats not that big a deal really | 01:35 |
| nibalizer | can you validate if compute021 is still a happy compute host | 01:36 |
| crinkle | :( | 01:36 |
| *** FallenPegasus has quit IRC | 01:36 | |
| crinkle | nibalizer: seems ok | 01:37 |
| *** tzn has joined #openstack-infra | 01:37 | |
| *** tiswanso has quit IRC | 01:38 | |
| *** c-jm has joined #openstack-infra | 01:41 | |
| nibalizer | woot ok | 01:41 |
| *** tzn has quit IRC | 01:42 | |
| nibalizer | so our next step is to try the controller? | 01:42 |
| crinkle | nibalizer: i think so | 01:42 |
| nibalizer | ok | 01:43 |
| nibalizer | can you take a backup of the hiera data | 01:43 |
| nibalizer | before i blow it away | 01:43 |
| crinkle | nibalizer: have you changed the hiera data? i already have a local source | 01:43 |
| crinkle | nibalizer: also what are we doing about SSL | 01:44 |
| *** c-jm has quit IRC | 01:44 | |
| nibalizer | we're gonna go with the SSL we have, and if we want to redo it we can redo it | 01:44 |
| crinkle | okay | 01:45 |
| *** FallenPegasus has joined #openstack-infra | 01:45 | |
| nibalizer | ok removing local hieradata on controller 00 | 01:46 |
| nibalizer | running ansible | 01:47 |
| nibalizer | copying puppet modules | 01:48 |
| nibalizer | its running puppet | 01:49 |
| nibalizer | done | 01:49 |
| nibalizer | seems to still bee a cloud! | 01:50 |
| crinkle | \o/ | 01:50 |
| *** Sukhdev has quit IRC | 01:51 | |
| nibalizer | booting a node to test it | 01:51 |
| nibalizer | works | 01:53 |
| nibalizer | node is up and sshing and stuff | 01:53 |
| *** apoorvad has quit IRC | 01:54 | |
| nibalizer | ok so I think our next step is to do the manual configuration on every compute host (I can ansible this I think) | 01:55 |
| nibalizer | and at that point we can un-disable the infracloud group | 01:55 |
| crinkle | nibalizer: awesome | 01:55 |
| nibalizer | oh maybe we should try the baremetal host too? | 01:55 |
| crinkle | nibalizer: i think that part is too much in flux to add it right now | 01:56 |
| nibalizer | ok so we would just laeve that disabled? | 01:56 |
| crinkle | nibalizer: i think so | 01:56 |
| nibalizer | ok | 01:56 |
| *** cloudtrainme has joined #openstack-infra | 01:57 | |
| *** sc68cal has quit IRC | 01:57 | |
| *** yamamoto has joined #openstack-infra | 01:59 | |
| *** cloudtrainme has quit IRC | 01:59 | |
| *** cloudtrainme has joined #openstack-infra | 02:01 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add infracloud playbook https://review.openstack.org/234535 | 02:02 |
| *** weshay_xchat has quit IRC | 02:02 | |
| nibalizer | crinkle: ^^ | 02:02 |
| nibalizer | basically your patch updated + the changes we needed in ansible groups | 02:02 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add infracloud playbook https://review.openstack.org/234535 | 02:03 |
| crinkle | nibalizer: cool ty | 02:04 |
| *** gyee has quit IRC | 02:06 | |
| openstackgerrit | Merged openstack/requirements: Bump PyECLib version to 1.0.8 https://review.openstack.org/275181 | 02:07 |
| mgagne | which method do you use to delete branches in Gerrit? REST? CLI? git? | 02:07 |
| openstackgerrit | Colleen Murphy proposed openstack/diskimage-builder: Add --version option to disk-image-create https://review.openstack.org/277658 | 02:07 |
| nibalizer | i just gui | 02:08 |
| nibalizer | I don't have to do it often | 02:08 |
| mgagne | nibalizer do you have to add yourself to Project Bootstrappers group? | 02:08 |
| nibalizer | http://paste.openstack.org/show/486352/ | 02:08 |
| nibalizer | mgagne: I don't remember | 02:09 |
| nibalizer | crinkle: clarkb thoughts on that script ^? | 02:09 |
| crinkle | nibalizer: does ansible not have --no-really-overwrite-this-directory option? | 02:12 |
| nibalizer | it has force but that seems to only apply to symlinks | 02:13 |
| nibalizer | http://docs.ansible.com/ansible/file_module.html | 02:13 |
| crinkle | nibalizer: script seems good to me based on that doc | 02:14 |
| *** tiswanso has joined #openstack-infra | 02:14 | |
| *** doug-fish has joined #openstack-infra | 02:14 | |
| nibalizer | ok i am going to run it | 02:15 |
| nibalizer | then thats probably it for tonight | 02:15 |
| nibalizer | tomorrow we can land 234535 and all the follow on stuff | 02:16 |
| *** Swami has quit IRC | 02:16 | |
| crinkle | sounds good | 02:16 |
| nibalizer | good job making cloud go | 02:17 |
| *** doug-fish has quit IRC | 02:19 | |
| EmilienM | mordred: nothing urgent, but when you have time https://github.com/ansible/ansible-modules-extras/pull/1634 | 02:20 |
| *** cloudtrainme has quit IRC | 02:23 | |
| *** yamamoto has quit IRC | 02:24 | |
| *** doug-fish has joined #openstack-infra | 02:25 | |
| *** xyang1 has quit IRC | 02:26 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/system-config: Remove double slash for individual links https://review.openstack.org/276558 | 02:26 |
| *** erlon has quit IRC | 02:26 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/system-config: Remove double slash for individual links https://review.openstack.org/276558 | 02:29 |
| *** mrunge has quit IRC | 02:29 | |
| nibalizer | so for anyone reading this tomorrow, the current status is that we've done all the testing and code landing to validate that infracloud can be run from the puppetmaster | 02:30 |
| jeblair | clarkb: yes, usa did -- and we selected voip.ms because of ease of getting international dids if we wanted | 02:30 |
| *** bpokorny_ has joined #openstack-infra | 02:30 | |
| nibalizer | all we need to do now is land 234535 which will flip the switch and start the management :) | 02:30 |
| *** bpokorny_ has quit IRC | 02:31 | |
| *** bpokorny has quit IRC | 02:33 | |
| *** mingyu has quit IRC | 02:35 | |
| *** mingyu has joined #openstack-infra | 02:36 | |
| *** mrunge has joined #openstack-infra | 02:36 | |
| EmilienM | nibalizer, crinkle: is it expected to install Kilo ? https://github.com/openstack-infra/puppet-infracloud/blob/master/manifests/controller.pp#L53 | 02:36 |
| crinkle | EmilienM: yes, until https://review.openstack.org/#/c/267817/ is done | 02:37 |
| EmilienM | the puppet code is really clean, nice work crinkle | 02:37 |
| crinkle | haha ty EmilienM | 02:37 |
| EmilienM | no logging at all, you could almost use full hiera :) | 02:37 |
| EmilienM | no logic* | 02:37 |
| *** tzn has joined #openstack-infra | 02:38 | |
| EmilienM | crinkle: where is the code that'll run ansible that runs puppet? I'm working on that too for our integration jobs, so we can do multi node (paul started the work last year) | 02:39 |
| *** tphummel has quit IRC | 02:40 | |
| nibalizer | EmilienM: the entry point is a script called run_all.sh in system-config | 02:40 |
| *** edmondsw has quit IRC | 02:41 | |
| EmilienM | ok | 02:42 |
| *** tzn has quit IRC | 02:42 | |
| *** doug-fish has quit IRC | 02:45 | |
| *** doug-fish has joined #openstack-infra | 02:46 | |
| *** fawadkhaliq has joined #openstack-infra | 02:47 | |
| mgagne | wow, is there only the Jenkins FTP publisher that strips the source path? :-/ | 02:49 |
| *** doug-fish has quit IRC | 02:50 | |
| openstackgerrit | Merged openstack-infra/system-config: Don't write k5start banner to file list https://review.openstack.org/277591 | 02:51 |
| *** unicell1 has joined #openstack-infra | 02:52 | |
| *** rhallisey has quit IRC | 02:52 | |
| *** unicell has quit IRC | 02:53 | |
| lifeless | tonyb: looking | 02:54 |
| *** armax_ is now known as armax | 02:54 | |
| *** sdake has joined #openstack-infra | 02:55 | |
| lifeless | tonyb: replied | 02:58 |
| *** moravec has quit IRC | 02:58 | |
| tonyb | lifeless: Thanks. | 02:59 |
| *** sdake_ has joined #openstack-infra | 03:00 | |
| *** sbalukoff has quit IRC | 03:00 | |
| *** sdake has quit IRC | 03:03 | |
| *** slogan621 has quit IRC | 03:05 | |
| *** moravec has joined #openstack-infra | 03:07 | |
| *** moravec has quit IRC | 03:08 | |
| *** yamamoto has joined #openstack-infra | 03:12 | |
| cody-somerville | Is there any reason we don't use containers for devstack yet? e.g. Does isci linux stuff still not happy with namespaces? | 03:15 |
| mgagne | I'm really not an expert or don't pretend to have the right answer but from what I can remember about it (it was maybe ~2 years ago?), it was a matter of lack of expertise in the support team. and maybe iscsi | 03:17 |
| *** thorst has quit IRC | 03:18 | |
| mgagne | but I'm not really the one that should answer that question =) | 03:18 |
| *** thorst has joined #openstack-infra | 03:18 | |
| *** jpr has quit IRC | 03:20 | |
| *** unicell has joined #openstack-infra | 03:21 | |
| tonyb | cody-somerville: openstack-ansible uses containers for *most* of it services. So it can be done. and that's about all I know about the subject. | 03:21 |
| *** unicell1 has quit IRC | 03:22 | |
| fungi | jeblair: jbryce says he cleaned up accounts that had no logins in months, but it sounds like if you had it set up as a forward he might have missed that | 03:25 |
| *** sridhar_ram has quit IRC | 03:25 | |
| fungi | any recollection if that might have been the case? | 03:25 |
| *** dims has joined #openstack-infra | 03:27 | |
| fungi | i'm assuming that means there's no objection to readding your forward, but i can find out if you want | 03:27 |
| *** thorst has quit IRC | 03:27 | |
| *** dims_ has quit IRC | 03:28 | |
| *** dims has quit IRC | 03:30 | |
| *** FallenPegasus has quit IRC | 03:32 | |
| *** doug-fish has joined #openstack-infra | 03:32 | |
| *** ajmiller has joined #openstack-infra | 03:34 | |
| *** FallenPegasus has joined #openstack-infra | 03:34 | |
| *** ajmiller has quit IRC | 03:37 | |
| *** dingyichen has quit IRC | 03:38 | |
| *** tzn has joined #openstack-infra | 03:39 | |
| *** dingyichen has joined #openstack-infra | 03:39 | |
| *** achanda has joined #openstack-infra | 03:40 | |
| *** moravec has joined #openstack-infra | 03:41 | |
| *** links has joined #openstack-infra | 03:41 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Use deps in testenv:py34 if specified https://review.openstack.org/277676 | 03:42 |
| *** dims has joined #openstack-infra | 03:43 | |
| *** tzn has quit IRC | 03:44 | |
| *** achanda has quit IRC | 03:45 | |
| *** amitgandhinz has quit IRC | 03:46 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Use deps in testenv:py34 if specified https://review.openstack.org/277676 | 03:46 |
| openstackgerrit | Tony Breeds proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 03:46 |
| *** dims has quit IRC | 03:47 | |
| *** gildub has joined #openstack-infra | 03:47 | |
| *** doug-fish has quit IRC | 03:56 | |
| openstackgerrit | Merged openstack/requirements: Octavia requires fixes in taskflow 1.26.0 to function https://review.openstack.org/276549 | 03:56 |
| *** doug-fish has joined #openstack-infra | 03:56 | |
| *** doug-fish has quit IRC | 03:57 | |
| *** dahoo has quit IRC | 04:00 | |
| *** doug-fis_ has joined #openstack-infra | 04:00 | |
| openstackgerrit | Masayuki Igawa proposed openstack-dev/cookiecutter: Add gitignore files https://review.openstack.org/277679 | 04:04 |
| *** doug-fis_ has quit IRC | 04:04 | |
| *** rguillebert has quit IRC | 04:07 | |
| greghaynes | clarkb: jeblair Left a comment on https://review.openstack.org/#/c/273703/9 with what is going on for that test fail - somehow gear client is getting in to a state where calling submitJob blocks forever (traceback included) | 04:09 |
| *** rlandy has quit IRC | 04:09 | |
| *** doug-fish has joined #openstack-infra | 04:09 | |
| *** bpokorny has joined #openstack-infra | 04:09 | |
| greghaynes | it looks like it is blocking on getting a submit completed, and this is right after it has disconnected | 04:10 |
| greghaynes | so my guess is there is a narrow window where a disconnect can happen and gear client does not ever get a work completed nor realize that it was disconnected | 04:11 |
| *** baoli_ has quit IRC | 04:11 | |
| *** apoorvad has joined #openstack-infra | 04:13 | |
| *** ayoung has quit IRC | 04:15 | |
| *** moravec has quit IRC | 04:17 | |
| ianw | mrodden: thanks! looking forward to getting these ones out of my queue | 04:17 |
| *** armax_ has joined #openstack-infra | 04:18 | |
| *** apoorvad has quit IRC | 04:19 | |
| *** armax has quit IRC | 04:21 | |
| *** armax_ is now known as armax | 04:21 | |
| *** thorst has joined #openstack-infra | 04:24 | |
| *** abregman|afk has quit IRC | 04:28 | |
| *** sbalukoff has joined #openstack-infra | 04:30 | |
| *** tiswanso has quit IRC | 04:30 | |
| *** thorst has quit IRC | 04:32 | |
| *** daemontool has quit IRC | 04:32 | |
| *** daemontool has joined #openstack-infra | 04:32 | |
| mrodden | ianw: i'm ok if you want to +W those in, i'm just concerned about the last two since devstack does not currently pass those | 04:36 |
| mrodden | ah nvm just saw your comment | 04:38 |
| *** mrmartin has joined #openstack-infra | 04:38 | |
| *** tzn has joined #openstack-infra | 04:40 | |
| *** salv-orl_ has joined #openstack-infra | 04:41 | |
| ianw | mrodden: yeah, i will cycle back to that and fixup devstack side | 04:42 |
| mrodden | ok | 04:42 |
| *** fawadkhaliq has quit IRC | 04:42 | |
| *** salv-orlando has quit IRC | 04:43 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Use SSL for rabbitmq https://review.openstack.org/277605 | 04:45 |
| *** tzn has quit IRC | 04:45 | |
| openstackgerrit | Merged openstack/requirements: Bump paramiko from 1.13.0 to 1.16.0 https://review.openstack.org/268197 | 04:46 |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Add a use_git flag https://review.openstack.org/273832 | 04:52 |
| *** sridhar_ram has joined #openstack-infra | 04:53 | |
| *** sridhar_ram has quit IRC | 04:53 | |
| *** Sukhdev has joined #openstack-infra | 04:57 | |
| *** dulek has joined #openstack-infra | 05:00 | |
| *** shashank_hegde has quit IRC | 05:02 | |
| *** amotoki has joined #openstack-infra | 05:07 | |
| *** hichihara has joined #openstack-infra | 05:07 | |
| *** jgriffith is now known as jgriffith_away | 05:07 | |
| *** amotoki_ has joined #openstack-infra | 05:09 | |
| *** amotoki has quit IRC | 05:12 | |
| openstackgerrit | Hidekazu Nakamura proposed openstack-infra/infra-manual: Update Accessing Gerrit over HTTPS section https://review.openstack.org/258286 | 05:13 |
| *** fawadkhaliq has joined #openstack-infra | 05:14 | |
| *** fawadkhaliq has quit IRC | 05:14 | |
| *** fawadkhaliq has joined #openstack-infra | 05:14 | |
| *** amotoki_ has quit IRC | 05:15 | |
| *** kdas_ has quit IRC | 05:17 | |
| *** daemontool_ has joined #openstack-infra | 05:21 | |
| *** daemontool has quit IRC | 05:22 | |
| *** thorst has joined #openstack-infra | 05:29 | |
| *** mrmartin has quit IRC | 05:30 | |
| *** Daisy has joined #openstack-infra | 05:36 | |
| *** thorst has quit IRC | 05:37 | |
| openstackgerrit | Merged openstack/requirements: Add os-testr to projects.txt https://review.openstack.org/273312 | 05:37 |
| *** hdd has quit IRC | 05:38 | |
| *** tzn has joined #openstack-infra | 05:41 | |
| *** abregman|afk has joined #openstack-infra | 05:45 | |
| *** tzn has quit IRC | 05:45 | |
| *** doug-fish has quit IRC | 05:48 | |
| *** NobodyCam has quit IRC | 05:48 | |
| *** amotoki has joined #openstack-infra | 05:49 | |
| *** NobodyCam has joined #openstack-infra | 05:50 | |
| *** mrmartin has joined #openstack-infra | 05:55 | |
| *** doug-fish has joined #openstack-infra | 05:55 | |
| *** nadya has joined #openstack-infra | 05:56 | |
| *** nadya is now known as Guest31507 | 05:56 | |
| *** Daisy has quit IRC | 05:58 | |
| *** abregman|afk is now known as abregman | 05:59 | |
| *** Daisy has joined #openstack-infra | 06:00 | |
| *** doug-fish has quit IRC | 06:08 | |
| *** sgordon has quit IRC | 06:09 | |
| *** savihou has joined #openstack-infra | 06:11 | |
| *** sgordon has joined #openstack-infra | 06:11 | |
| *** bpokorny has quit IRC | 06:12 | |
| *** achanda has joined #openstack-infra | 06:18 | |
| *** hdd has joined #openstack-infra | 06:18 | |
| *** nikil89_ has joined #openstack-infra | 06:19 | |
| *** fedexo has joined #openstack-infra | 06:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/277702 | 06:23 |
| *** Guest31507 has quit IRC | 06:26 | |
| *** shashank_hegde has joined #openstack-infra | 06:27 | |
| *** rockyg has quit IRC | 06:29 | |
| openstackgerrit | Fredrik Medley proposed openstack-infra/zuul: Improve job dependencies using graph instead of tree https://review.openstack.org/231124 | 06:30 |
| cody-somerville | that (not so) hilarious moment when you realize you've just spent 30 minutes trying to debug why an image won't boot when you realize you accidentally downloaded the arm64 instead of amd64 image. | 06:33 |
| *** achanda has quit IRC | 06:35 | |
| *** thorst has joined #openstack-infra | 06:35 | |
| *** tzn has joined #openstack-infra | 06:42 | |
| *** mrmartin has quit IRC | 06:42 | |
| *** thorst has quit IRC | 06:42 | |
| *** hdd has quit IRC | 06:42 | |
| *** achanda has joined #openstack-infra | 06:43 | |
| *** flepied has quit IRC | 06:45 | |
| *** tzn has quit IRC | 06:46 | |
| bkero | cody-somerville: just run it through qemu-arm64-static, it'll be fine | 06:47 |
| cody-somerville | that would work if I had configured the image with that arch, aye. | 06:48 |
| *** armax has quit IRC | 06:49 | |
| *** hichihara has quit IRC | 06:49 | |
| *** Daisy has quit IRC | 06:49 | |
| openstackgerrit | Doug Wiegley proposed openstack/requirements: Add neutron-lib to projects.txt https://review.openstack.org/273913 | 06:50 |
| *** Daisy has joined #openstack-infra | 06:53 | |
| *** kushal has joined #openstack-infra | 06:53 | |
| *** scheuran has joined #openstack-infra | 06:55 | |
| *** amotoki has quit IRC | 06:56 | |
| *** amotoki has joined #openstack-infra | 06:58 | |
| *** achanda has quit IRC | 06:58 | |
| *** sdake_ has quit IRC | 07:00 | |
| *** sdake has joined #openstack-infra | 07:01 | |
| *** sdake has quit IRC | 07:03 | |
| *** fedexo_ has joined #openstack-infra | 07:03 | |
| *** fedexo__ has joined #openstack-infra | 07:03 | |
| *** fedexo has quit IRC | 07:04 | |
| AJaeger | just got "Download error on http://pypi.dfw.openstack.org/simple/pbr/: [Errno 113] No route to host -- Some packages may not be found!" | 07:13 |
| AJaeger | this is on the proposal slave | 07:13 |
| AJaeger | jeblair, mordred, krotscheck ^ This happened with many of our periodic translation jobs this morning | 07:15 |
| *** infra-red has joined #openstack-infra | 07:17 | |
| *** chrall_ has joined #openstack-infra | 07:19 | |
| *** Sukhdev has quit IRC | 07:19 | |
| *** esikachev has joined #openstack-infra | 07:19 | |
| *** isaacb has joined #openstack-infra | 07:21 | |
| *** flepied has joined #openstack-infra | 07:22 | |
| *** flepied1 has joined #openstack-infra | 07:23 | |
| *** flepied has quit IRC | 07:27 | |
| *** chrall__ has joined #openstack-infra | 07:28 | |
| *** savihou has quit IRC | 07:28 | |
| AJaeger | pabelanger: https://github.com/openstack/ansible-role-jenkins is still empty - please check later today whether it's still the case and if yes, ask for help to fill it. Same problem with your other repo. | 07:30 |
| *** Daisy has quit IRC | 07:32 | |
| *** mingyu has quit IRC | 07:32 | |
| *** mingyu has joined #openstack-infra | 07:32 | |
| *** chrall_ has quit IRC | 07:32 | |
| *** Daisy has joined #openstack-infra | 07:33 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add unit test for api.update_run https://review.openstack.org/277717 | 07:33 |
| *** chrall__ has quit IRC | 07:36 | |
| *** Daisy has quit IRC | 07:37 | |
| *** jtomasek__ has quit IRC | 07:38 | |
| *** doug-fish has joined #openstack-infra | 07:38 | |
| *** thorst has joined #openstack-infra | 07:40 | |
| *** doug-fis_ has joined #openstack-infra | 07:42 | |
| *** tzn has joined #openstack-infra | 07:43 | |
| *** doug-fish has quit IRC | 07:43 | |
| *** thorst has quit IRC | 07:46 | |
| *** tzn has quit IRC | 07:47 | |
| *** doug-fis_ has quit IRC | 07:47 | |
| *** chrall__ has joined #openstack-infra | 07:49 | |
| *** nmagnezi has joined #openstack-infra | 07:49 | |
| AJaeger | mrodden: thansk for the release of bashate! | 07:55 |
| *** sshnaidm has quit IRC | 07:58 | |
| *** reed_ has quit IRC | 07:59 | |
| *** jcoufal has joined #openstack-infra | 07:59 | |
| *** reed_ has joined #openstack-infra | 08:00 | |
| *** reed_ has quit IRC | 08:01 | |
| vsaienko | please review https://review.openstack.org/#/c/276200/ | 08:01 |
| *** fedexo__ has quit IRC | 08:03 | |
| *** fedexo_ has quit IRC | 08:03 | |
| openstackgerrit | Merged openstack-infra/project-config: run Puppet OpenStack integration jobs on puppet-ceph https://review.openstack.org/277598 | 08:04 |
| AJaeger | vsaienko: it helps a lot if you give some context - your change is for requirements and infra folks do not review that | 08:04 |
| *** mingyu has quit IRC | 08:05 | |
| *** savihou has joined #openstack-infra | 08:05 | |
| *** mingyu has joined #openstack-infra | 08:05 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add unit test for api.update_run https://review.openstack.org/277717 | 08:05 |
| *** arxcruz has quit IRC | 08:06 | |
| *** pblaho has joined #openstack-infra | 08:11 | |
| vsaienko | AJaeger, i thought that global requirements is managed by infra folks. | 08:13 |
| AJaeger | vsaienko: infra is only fallback, there's a separate core team, check the access list on review.openstack.org for it | 08:15 |
| vsaienko | AJaeger, ok thank you understood | 08:16 |
| *** arxcruz has joined #openstack-infra | 08:19 | |
| *** mrmartin has joined #openstack-infra | 08:21 | |
| *** Jeffrey4l has joined #openstack-infra | 08:23 | |
| *** jlanoux has joined #openstack-infra | 08:27 | |
| *** oomichi has quit IRC | 08:28 | |
| *** shardy has joined #openstack-infra | 08:30 | |
| *** mikelk has joined #openstack-infra | 08:30 | |
| *** esikachev has quit IRC | 08:31 | |
| *** matrohon has joined #openstack-infra | 08:31 | |
| *** MCoLo has joined #openstack-infra | 08:33 | |
| *** deepakcs has joined #openstack-infra | 08:33 | |
| deepakcs | AJaeger: ping | 08:34 |
| *** dingyichen has quit IRC | 08:37 | |
| *** zz_dimtruck is now known as dimtruck | 08:37 | |
| *** jaosorior has joined #openstack-infra | 08:37 | |
| *** esikachev has joined #openstack-infra | 08:38 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add unit test for api.update_run https://review.openstack.org/277717 | 08:39 |
| *** yaume has joined #openstack-infra | 08:39 | |
| *** tzn has joined #openstack-infra | 08:44 | |
| *** thorst has joined #openstack-infra | 08:45 | |
| *** shashank_hegde has quit IRC | 08:45 | |
| *** tzn has quit IRC | 08:48 | |
| *** e0ne has joined #openstack-infra | 08:49 | |
| *** dimtruck is now known as zz_dimtruck | 08:51 | |
| *** [1]Thelo has joined #openstack-infra | 08:52 | |
| *** thorst has quit IRC | 08:53 | |
| *** nadya has joined #openstack-infra | 08:53 | |
| *** nadya is now known as Guest22569 | 08:54 | |
| *** sshnaidm has joined #openstack-infra | 08:54 | |
| *** Thelo has quit IRC | 08:54 | |
| *** [1]Thelo is now known as Thelo | 08:54 | |
| *** fhubik has joined #openstack-infra | 08:56 | |
| *** mingyu has quit IRC | 09:00 | |
| *** mingyu has joined #openstack-infra | 09:01 | |
| *** samuelBartel has joined #openstack-infra | 09:01 | |
| *** jaosorior has quit IRC | 09:03 | |
| *** fhubik is now known as fhubik_brb | 09:04 | |
| *** claudiub has joined #openstack-infra | 09:06 | |
| *** placko has joined #openstack-infra | 09:07 | |
| *** jaosorior has joined #openstack-infra | 09:08 | |
| *** jaosorior has quit IRC | 09:08 | |
| *** jaosorior has joined #openstack-infra | 09:09 | |
| *** lucas-dinner is now known as lucasagomes | 09:10 | |
| *** fhubik_brb is now known as fhubik | 09:15 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/project-config: Make TripleO puppet check jobs voting https://review.openstack.org/270165 | 09:17 |
| *** ifarkas has joined #openstack-infra | 09:19 | |
| *** e0ne has quit IRC | 09:19 | |
| *** ihrachys has joined #openstack-infra | 09:19 | |
| *** dizquierdo has joined #openstack-infra | 09:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:31 | |
| *** chrall__ has quit IRC | 09:35 | |
| *** rossella_s has joined #openstack-infra | 09:35 | |
| *** ociuhandu has quit IRC | 09:38 | |
| *** daemontool has joined #openstack-infra | 09:39 | |
| *** derekh has joined #openstack-infra | 09:39 | |
| *** daemontool_ has quit IRC | 09:40 | |
| *** Daisy has joined #openstack-infra | 09:41 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** tzn has joined #openstack-infra | 09:44 | |
| AJaeger | deepakcs: https://blogs.gnome.org/markmc/2014/02/20/naked-pings/ | 09:45 |
| deepakcs | AJaeger: sorry :) quick q on https://review.openstack.org/#/c/273326/ | 09:45 |
| deepakcs | AJaeger: is there a way to know which patches were part of the data set using which sdague created that graph (which shows that new plugin is failing more often than old one) ? | 09:46 |
| AJaeger | deepakcs: shoot - but best wait two hours and discuss with sdague | 09:46 |
| deepakcs | AJaeger: sure, i thought u might know hence wanted to cashin :) | 09:46 |
| openstackgerrit | Merged openstack-infra/zuul: Don't require 'commit' attribute in merge event https://review.openstack.org/272123 | 09:46 |
| AJaeger | deepakcs: look at the URL of the graph ;) | 09:47 |
| *** fawadkhaliq has quit IRC | 09:47 | |
| AJaeger | open it in the browser | 09:47 |
| *** fawadkhaliq has joined #openstack-infra | 09:47 | |
| *** chrall__ has joined #openstack-infra | 09:47 | |
| AJaeger | deepakcs: the graph just shows those jobs that were run during that time, so it's the same patches for both | 09:48 |
| deepakcs | AJaeger: yeah, i wanted to get into those patches' CI logs for these jobs to figure why one is failing more than the other | 09:48 |
| AJaeger | I guess you need to look at logstash to see which jobs were run during this time. Sorry, can't help with these further | 09:49 |
| *** tzn has quit IRC | 09:49 | |
| deepakcs | AJaeger: np, thanks for ur time. | 09:50 |
| *** thorst has joined #openstack-infra | 09:50 | |
| *** fhubik is now known as fhubik_brb | 09:52 | |
| *** fhubik_brb is now known as fhubik | 09:52 | |
| *** NobodyCam has quit IRC | 09:52 | |
| *** kzaitsev_mb has quit IRC | 09:53 | |
| *** thorst has quit IRC | 09:57 | |
| *** NobodyCam has joined #openstack-infra | 09:59 | |
| *** fawadkhaliq has quit IRC | 10:01 | |
| *** Jeffrey4l has quit IRC | 10:07 | |
| *** Guest15184 is now known as BobBall | 10:07 | |
| *** rguillebert has joined #openstack-infra | 10:10 | |
| *** jtomasek has joined #openstack-infra | 10:11 | |
| *** nilasae has joined #openstack-infra | 10:13 | |
| vponomaryov | Looking for help of core team in merging two simple commits that add CI jobs for new Manila project drivers that should be tested in CI before FF: https://review.openstack.org/#/c/275823/ and https://review.openstack.org/#/c/277446/ | 10:17 |
| *** electrofelix has joined #openstack-infra | 10:18 | |
| *** e0ne has joined #openstack-infra | 10:21 | |
| *** FallenPegasus has quit IRC | 10:22 | |
| *** hashar has joined #openstack-infra | 10:25 | |
| *** Roamer`_ is now known as Roamer` | 10:28 | |
| *** FallenPegasus has joined #openstack-infra | 10:28 | |
| *** sfinucan has joined #openstack-infra | 10:28 | |
| *** jaosorior has quit IRC | 10:29 | |
| *** deva_ has joined #openstack-infra | 10:29 | |
| *** jaosorior has joined #openstack-infra | 10:29 | |
| *** jesusaurus has joined #openstack-infra | 10:30 | |
| *** esp has joined #openstack-infra | 10:30 | |
| *** tcammann_ has joined #openstack-infra | 10:30 | |
| *** arxcruz has quit IRC | 10:30 | |
| *** samueldmq1 has joined #openstack-infra | 10:33 | |
| *** NobodyCa1 has joined #openstack-infra | 10:33 | |
| openstackgerrit | Alexander Charykov proposed openstack-infra/gerritlib: Add limit for bulk_query request https://review.openstack.org/277773 | 10:33 |
| *** amotoki has quit IRC | 10:34 | |
| openstackgerrit | Alexander Charykov proposed openstack-infra/gerritlib: Add limit for bulk_query request https://review.openstack.org/277773 | 10:34 |
| *** tcammann_ has quit IRC | 10:35 | |
| *** jesusaurus has quit IRC | 10:37 | |
| *** esp has quit IRC | 10:37 | |
| *** samueldmq1 has quit IRC | 10:38 | |
| *** NobodyCa1 has quit IRC | 10:38 | |
| *** deva_ has quit IRC | 10:38 | |
| *** notnownikki has joined #openstack-infra | 10:40 | |
| *** salv-orlando has joined #openstack-infra | 10:40 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-lodgeit: fix puppet-lodgeit to support latest mysql https://review.openstack.org/275244 | 10:41 |
| *** salv-orl_ has quit IRC | 10:43 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/project-config: Make TripleO puppet check jobs voting https://review.openstack.org/270165 | 10:44 |
| *** rfolco has joined #openstack-infra | 10:44 | |
| *** dims has joined #openstack-infra | 10:45 | |
| *** tzn has joined #openstack-infra | 10:45 | |
| *** dims has quit IRC | 10:49 | |
| *** amotoki has joined #openstack-infra | 10:49 | |
| *** Daisy has quit IRC | 10:49 | |
| *** dims has joined #openstack-infra | 10:50 | |
| *** Daisy has joined #openstack-infra | 10:50 | |
| *** arxcruz has joined #openstack-infra | 10:50 | |
| *** fawadkhaliq has joined #openstack-infra | 10:50 | |
| *** tzn has quit IRC | 10:50 | |
| *** deepakcs has quit IRC | 10:52 | |
| *** dprince has joined #openstack-infra | 10:52 | |
| *** tzn has joined #openstack-infra | 10:53 | |
| *** tzn has quit IRC | 10:54 | |
| *** thorst has joined #openstack-infra | 10:54 | |
| *** weshay_xchat has joined #openstack-infra | 10:55 | |
| *** dizquierdo has quit IRC | 10:55 | |
| *** amotoki has quit IRC | 10:55 | |
| *** tpsilva has joined #openstack-infra | 10:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:56 | |
| *** erlon has joined #openstack-infra | 10:56 | |
| *** jpr has joined #openstack-infra | 10:56 | |
| *** tzn has joined #openstack-infra | 10:58 | |
| *** flepied1 has quit IRC | 11:00 | |
| *** flepied has joined #openstack-infra | 11:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Add ZFSonLinux experimental Tempest job for Manila https://review.openstack.org/277446 | 11:01 |
| *** thorst has quit IRC | 11:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Add LXD experimental Tempest job for Manila https://review.openstack.org/275823 | 11:02 |
| openstackgerrit | Alexander Charykov proposed openstack-infra/gerritlib: Add limit for bulk_query request https://review.openstack.org/277773 | 11:05 |
| *** yamamoto has quit IRC | 11:05 | |
| *** dizquierdo has joined #openstack-infra | 11:06 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Fix periodic log cleanup https://review.openstack.org/277099 | 11:10 |
| openstackgerrit | Merged openstack-infra/project-config: Support stable requirements https://review.openstack.org/277550 | 11:11 |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config: Enables Congress to launch with dist_arch flag as True https://review.openstack.org/277787 | 11:11 |
| openstackgerrit | Merged openstack-infra/project-config: Remove tooz testing of etcd/consul on liberty or kilo https://review.openstack.org/277526 | 11:16 |
| *** fawadkhaliq has quit IRC | 11:19 | |
| *** fawadkhaliq has joined #openstack-infra | 11:19 | |
| *** fawadkhaliq has quit IRC | 11:19 | |
| *** fawadkhaliq has joined #openstack-infra | 11:20 | |
| *** ihrachys has quit IRC | 11:20 | |
| openstackgerrit | Alexey Ovchinnikov proposed openstack/requirements: Add pylxd library https://review.openstack.org/277789 | 11:20 |
| *** FallenPegasus has quit IRC | 11:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch non-infra Ansible roles to using 'linters' tox target https://review.openstack.org/277479 | 11:21 |
| *** fhubik has quit IRC | 11:33 | |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Rewrite use of ::mysql::server in zanata::mysql https://review.openstack.org/273339 | 11:34 |
| *** shardy has quit IRC | 11:34 | |
| *** egonzalez has joined #openstack-infra | 11:34 | |
| *** baoli has joined #openstack-infra | 11:36 | |
| *** shardy has joined #openstack-infra | 11:36 | |
| *** baoli has quit IRC | 11:37 | |
| *** nikil89_ has quit IRC | 11:37 | |
| *** jcoufal has quit IRC | 11:38 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic https://review.openstack.org/269684 | 11:38 |
| *** dims_ has joined #openstack-infra | 11:40 | |
| *** dims has quit IRC | 11:40 | |
| *** FallenPegasus has joined #openstack-infra | 11:42 | |
| *** mingyu_ has joined #openstack-infra | 11:42 | |
| *** FallenPegasus has quit IRC | 11:43 | |
| *** boris-42 has quit IRC | 11:43 | |
| *** mingyu has quit IRC | 11:45 | |
| *** yamamoto has joined #openstack-infra | 11:51 | |
| *** xek_ is now known as xek | 11:51 | |
| *** kzaitsev_mb has quit IRC | 11:55 | |
| *** thorst has joined #openstack-infra | 12:00 | |
| BobBall | mordred / fungi: To stop caching images not used in the gate, I suggest that http://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/image_list.sh#n16 should take a configurable list of drivers and https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#L118 pass just the drivers that are tested in the gate (ironic | 12:01 |
| BobBall | libvirt dummy) - thoughts? | 12:02 |
| *** FallenPegasus has joined #openstack-infra | 12:02 | |
| *** thorst has quit IRC | 12:07 | |
| *** FallenPegasus has quit IRC | 12:07 | |
| *** ihrachys has joined #openstack-infra | 12:10 | |
| *** alivigni has joined #openstack-infra | 12:10 | |
| *** dims_ has quit IRC | 12:11 | |
| *** lucasagomes is now known as lucas-hungry | 12:13 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Add pass through for swift_account_temp_key https://review.openstack.org/276902 | 12:14 |
| *** dizquierdo is now known as dizquierdo_afk | 12:17 | |
| *** dims_ has joined #openstack-infra | 12:19 | |
| dims_ | AJaeger : trading reviews :) here's a small thing i needed to get keystone with oslo master to work - https://review.openstack.org/277676 | 12:22 |
| *** mrmartin has quit IRC | 12:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:24 | |
| *** gildub has quit IRC | 12:24 | |
| AJaeger | ;) | 12:26 |
| *** mtanino has joined #openstack-infra | 12:28 | |
| *** moravec has joined #openstack-infra | 12:34 | |
| *** gordc has joined #openstack-infra | 12:36 | |
| openstackgerrit | Alexey Ovchinnikov proposed openstack/requirements: Add pylxd library https://review.openstack.org/277789 | 12:36 |
| *** openstackgerrit_ has joined #openstack-infra | 12:38 | |
| *** tdurakov_ has joined #openstack-infra | 12:39 | |
| *** tdurakov_ has quit IRC | 12:39 | |
| *** fawadkhaliq has quit IRC | 12:43 | |
| *** jcoufal has joined #openstack-infra | 12:44 | |
| *** marcusvrn_ has joined #openstack-infra | 12:44 | |
| *** doug-fish has joined #openstack-infra | 12:45 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** thorst has joined #openstack-infra | 12:46 | |
| *** thorst_ has joined #openstack-infra | 12:47 | |
| *** doug-fish has quit IRC | 12:47 | |
| *** doug-fish has joined #openstack-infra | 12:48 | |
| *** dizquierdo_afk is now known as dizquierdo | 12:50 | |
| *** thorst has quit IRC | 12:51 | |
| *** rlandy has joined #openstack-infra | 12:54 | |
| dims_ | sdague : if you get a chance, please peek at a couple of quick reviews to get py27/34 running against oslo-master working properly - https://review.openstack.org/#/c/277421/ https://review.openstack.org/#/c/277676/ | 13:00 |
| sdague | dims_: I don't really understand why for - https://review.openstack.org/#/c/277676/2 | 13:02 |
| sdague | like what is failing that we need to change for that | 13:02 |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| dims_ | sdague : one sec | 13:03 |
| *** ildikov has quit IRC | 13:06 | |
| dims_ | sdague : keystone's tox.ini has a "[testenv:py34]" section which adds nose to the deps. if i don't do this, the new section "[testenv:py34-oslo-master]" dynamically generated by run-tox-with-oslo-master.sh fails as nose is not available | 13:06 |
| *** woodster_ has quit IRC | 13:06 | |
| dims_ | sdague : as [testenv:py34]/commands uses nosetests to run the py34 tests | 13:06 |
| *** lucas-hungry is now known as lucasagomes | 13:07 | |
| *** jpr has quit IRC | 13:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Drop branch selection on oslo-master periodic job https://review.openstack.org/277421 | 13:11 |
| sdague | why does keystone add nose to it's deps? | 13:12 |
| sdague | honestly, that should be fixed, right? | 13:13 |
| *** dizquierdo has quit IRC | 13:14 | |
| sdague | I really don't like the layers of work arounds | 13:16 |
| *** yamamoto has quit IRC | 13:16 | |
| *** links has quit IRC | 13:17 | |
| *** sdake has joined #openstack-infra | 13:19 | |
| *** annegentle has joined #openstack-infra | 13:20 | |
| *** AJaeger has quit IRC | 13:20 | |
| *** sdake_ has joined #openstack-infra | 13:21 | |
| *** AJaeger has joined #openstack-infra | 13:21 | |
| *** fawadkhaliq has joined #openstack-infra | 13:22 | |
| *** yamamoto has joined #openstack-infra | 13:23 | |
| *** sdake has quit IRC | 13:24 | |
| *** sdake_ is now known as sdake | 13:24 | |
| *** jlanoux has quit IRC | 13:24 | |
| *** jlanoux has joined #openstack-infra | 13:25 | |
| *** deepakcs has joined #openstack-infra | 13:25 | |
| deepakcs | sdague: Hi, regards to https://review.openstack.org/#/c/273326/ | 13:25 |
| deepakcs | sdague: I am wondering how to figure why the rate for plugin job is higher, it configures ceph the same way as hook job and for the same projects as the hook job | 13:26 |
| deepakcs | sdague: is there a way to know which patches were part of the data set using which you created that graph (which shows that new plugin is failing more often than old one) ? | 13:26 |
| sdague | deepakcs: it's just all the upstream test results | 13:26 |
| sdague | you need to go look at the fails and figure out why | 13:26 |
| deepakcs | sdague: ok, in the past i had done some random picking on patches and most of the time i either saw both jobs pass or fail, rarely saw 1 pass, 1 fail, but ur data provides a diff perspective | 13:27 |
| deepakcs | sdague: so i guess the only way is to start looking at individual patches where old job passed but new failed and figure out / debug from there ? | 13:28 |
| sdague | yes | 13:28 |
| deepakcs | sdague: ok, thanks | 13:28 |
| sdague | that's the responsibility for a new job, to understand every failure and why it's ok that people will be hit by it | 13:28 |
| deepakcs | sdague: the data set usign which you created that graph, is it possible to dump the associated patch IDs too ? | 13:29 |
| *** mase_x200 has joined #openstack-infra | 13:29 | |
| deepakcs | sdague: i mean dump them in the graph, so that its makes the job a bit easier | 13:30 |
| *** regXboi has joined #openstack-infra | 13:31 | |
| *** andrearosa has left #openstack-infra | 13:32 | |
| sdague | deepakcs: no, that's just graphite | 13:32 |
| deepakcs | sdague: ok, thanks again, good day | 13:32 |
| mordred | sdague: keystone adds nodes to its deps? | 13:34 |
| sdague | mordred: for python 3 testing | 13:34 |
| mordred | sdague: tat makes no sense | 13:34 |
| sdague | https://github.com/openstack/keystone/blob/f473352e1a751544d7a7e4fc1cae8c6cfaf3ce98/tox.ini#L16 | 13:34 |
| sdague | blame fungi, it was his patch :) | 13:34 |
| mordred | fungi: I blame you | 13:35 |
| sdague | at the time, it was kind of the only sane approach, given the initial test had an exclude and testr doesn't do that | 13:36 |
| sdague | but we've got an ostestr blacklist facility now | 13:36 |
| mordred | ah - ok | 13:36 |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Adjust API Endpoints for Task Links https://review.openstack.org/277831 | 13:36 |
| sdague | https://github.com/openstack/nova/blob/fec6f9dae23968bf0944c872cf66f411efaf380c/tox.ini#L52 | 13:36 |
| sdague | which should be done here instead of this I think | 13:37 |
| sdague | but yes, we're explicitly running nose on all keystone jobs right now | 13:37 |
| sdague | sorry, all keystone patches | 13:37 |
| sdague | in the 34 job | 13:37 |
| *** nelsnels_ has joined #openstack-infra | 13:37 | |
| *** nelsnelson has quit IRC | 13:37 | |
| *** ihrachys has quit IRC | 13:40 | |
| *** sdake has quit IRC | 13:40 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Revert "Set gate-grenade-dsvm-multinode to non-voting" https://review.openstack.org/277833 | 13:41 |
| *** mriedem has joined #openstack-infra | 13:41 | |
| *** jordanP has joined #openstack-infra | 13:41 | |
| *** sdake has joined #openstack-infra | 13:41 | |
| *** daemontool has quit IRC | 13:42 | |
| sdague | ^^^ we should get that voting again now that the provider regressions were addressed | 13:42 |
| *** jlanoux has quit IRC | 13:43 | |
| AJaeger | sdague: could you enhance the commit message, please? ^ | 13:43 |
| *** jlanoux has joined #openstack-infra | 13:43 | |
| sdague | AJaeger: it's in the comments | 13:44 |
| *** eharney has joined #openstack-infra | 13:45 | |
| *** mase_x200 has quit IRC | 13:45 | |
| dims_ | sdague : they need deps at least for the ldap stuff - http://paste.openstack.org/show/486420/ | 13:45 |
| dims_ | sdague : i can work with them to switch to os-testr etc and try other possibilities. but right now i am trying get rid of my travis setup | 13:46 |
| sdague | dims_: ldap is 2.7 only | 13:47 |
| dims_ | sdague : yes | 13:47 |
| sdague | https://github.com/openstack/requirements/blob/b7c385b3ae972c5efd3df6ae8451f4e378564acc/global-requirements.txt#L285 | 13:47 |
| sdague | oh, I get what you are saying | 13:47 |
| dims_ | yes, we have been talking about ldap support for py34 | 13:47 |
| sdague | so, honestly, it seems kind of pointless to bother with keystone py34 at all until there is an ldap story | 13:48 |
| *** sdake has quit IRC | 13:48 | |
| dims_ | sdague : i know, but a change in oslo master if it ends up breaking keystone 34 unit tests will be a problem | 13:48 |
| dims_ | since it's voting | 13:48 |
| sdague | right, but what are the odds of that? | 13:49 |
| *** sdake has joined #openstack-infra | 13:49 | |
| dims_ | sdague : we broke heat py34 last weekend :) | 13:49 |
| sdague | you can't be breaking any keystone tests, because you ran them on py27 | 13:49 |
| sdague | dims_: how? | 13:49 |
| sdague | and you are running py34 with a lot of projects, so you have some general coverage | 13:49 |
| dims_ | sdague : i have 2 periodic jobs, one for py27 and one for py34 | 13:49 |
| sdague | dims_: sure | 13:50 |
| sdague | anyway, I don't think it's worth the work around layer personally | 13:50 |
| dims_ | one sec let me show you the example from oslo.db and heat | 13:50 |
| sdague | if the keystone team wants protection, they should get their python 34 job aligned with their 27 one | 13:50 |
| dims_ | sdague : https://review.openstack.org/#/c/277328/ | 13:50 |
| dims_ | sdague : yes, i'll work with bknudson_ and stevemar on that :) | 13:51 |
| sdague | dims_: and that *only* broke on 34? | 13:52 |
| dims_ | sdague : given the time of the schedule right now, it may be a bit hard to do it right away | 13:52 |
| sdague | how did it pass on 27 | 13:52 |
| *** cbader has joined #openstack-infra | 13:55 | |
| pabelanger | AJaeger: Ya, I noticed that too. git.openstack.org is fine, so I think it was a replicate issue. Thanks for the heads up | 13:55 |
| AJaeger | pabelanger: is it up now? | 13:55 |
| AJaeger | I mean synced... | 13:56 |
| pabelanger | AJaeger: yolanda: do you mind adding me to https://review.openstack.org/#/admin/groups/1280,members https://review.openstack.org/#/admin/groups/1281,members https://review.openstack.org/#/admin/groups/1282,members and https://review.openstack.org/#/admin/groups/1283,members ? | 13:56 |
| yolanda | hi pabelanger | 13:56 |
| yolanda | sure | 13:56 |
| *** edmondsw has joined #openstack-infra | 13:56 | |
| *** amitgandhinz has joined #openstack-infra | 13:56 | |
| *** openstackgerrit_ has quit IRC | 13:56 | |
| yolanda | done | 13:57 |
| dims_ | sdague : here's how - http://paste.openstack.org/show/486424/ | 13:57 |
| pabelanger | yolanda: danke | 13:58 |
| dims_ | sdague : py2.7 base exception has a "message" and py34 does not | 13:58 |
| *** jsavak has joined #openstack-infra | 13:58 | |
| sdague | dims_: yeh, I see. The heat code shouldn't have been in the 34 run, it wasn't valid 34 code | 13:58 |
| sdague | and oslo was masking it | 13:58 |
| *** daemontool has joined #openstack-infra | 13:59 | |
| sdague | anyway, I think it's a bad work around. | 13:59 |
| *** vgridnev has joined #openstack-infra | 14:00 | |
| *** annegentle has quit IRC | 14:01 | |
| openstackgerrit | Anton Arefiev proposed openstack-infra/project-config: Add auto-discovery test job to ironic-inspector https://review.openstack.org/277843 | 14:02 |
| *** ildikov has joined #openstack-infra | 14:02 | |
| *** xyang1 has joined #openstack-infra | 14:02 | |
| *** Daisy has quit IRC | 14:02 | |
| *** rossella_s has quit IRC | 14:02 | |
| *** dims_ has quit IRC | 14:03 | |
| *** rossella_s has joined #openstack-infra | 14:03 | |
| mordred | AJaeger: https://review.openstack.org/#/c/277114 is ready I believe | 14:07 |
| AJaeger | mordred: what was going on with the proposal slaves this morning? Did you see my comment in the backscroll? | 14:07 |
| *** deepakcs has quit IRC | 14:07 | |
| *** mtanino has quit IRC | 14:07 | |
| mordred | so - I'm not 100% what's going on with the transition there - but the pypi mirrors are the old ones | 14:09 |
| mordred | it looks like the proposal slave is still using the old mirrors | 14:09 |
| AJaeger | interesting that only a part of the jobs failed... | 14:09 |
| mordred | yah | 14:09 |
| *** sdake has quit IRC | 14:10 | |
| *** dims has joined #openstack-infra | 14:10 | |
| mordred | AJaeger: hrm. I see no pip config pointing at mirrors on the proposal slave | 14:10 |
| *** jpeeler has joined #openstack-infra | 14:11 | |
| *** jpeeler has quit IRC | 14:11 | |
| *** jpeeler has joined #openstack-infra | 14:11 | |
| AJaeger | mordred: example log https://jenkins.openstack.org/job/designate-dashboard-propose-translation-update/141/console | 14:12 |
| EmilienM | pabelanger: when you have time this week, I might need your help at getting logs when running multi node jobs | 14:13 |
| *** baoli has joined #openstack-infra | 14:14 | |
| dims | sdague : my connection dropped. so repeating last few lines...heat was doing the right thing because oslo.db exception class had a "message" field in it. when we yanked it out heat py34 unit tests broke. so the fault was on oslo.db side too not just heat | 14:14 |
| pabelanger | EmilienM: sure. Have you looked into devstack multinode yet? | 14:15 |
| EmilienM | pabelanger: a little but i'll look more and ask you questions if I have | 14:15 |
| pabelanger | EmilienM: sure | 14:16 |
| *** daemontool_ has joined #openstack-infra | 14:16 | |
| *** dkranz has joined #openstack-infra | 14:16 | |
| *** links has joined #openstack-infra | 14:16 | |
| sdague | dims: right, because the heat code wan't py34 | 14:16 |
| sdague | and oslo was masking it | 14:16 |
| sdague | anyway, I think it's a bad work around. | 14:17 |
| *** baoli_ has joined #openstack-infra | 14:17 | |
| *** daemontool has quit IRC | 14:17 | |
| EmilienM | mordred: I'm going to push more in ansible/puppet like I did last night, is it ok if I ping your nick on github for the PRs ? | 14:18 |
| EmilienM | I see lot of missing features | 14:18 |
| dims | sdague : in the end we had a review in heat to cleanup and oslo.db has log/warnings to not use message. so it's all good :) | 14:18 |
| dims | sdague : these tests still help is my point :) | 14:18 |
| sdague | dims: I'm not saying they don't help, I'm saying that keystone can be skipped until they get a system together to work here with normal flow | 14:19 |
| sdague | and put it back on them | 14:19 |
| *** eharney has quit IRC | 14:19 | |
| openstackgerrit | George Shuklin proposed openstack/diskimage-builder: Add DIB_APT_SOURCES_INLINE to apt-sources element. https://review.openstack.org/277850 | 14:20 |
| *** baoli has quit IRC | 14:20 | |
| *** jgriffith_away is now known as jgriffith | 14:21 | |
| *** yamahata has joined #openstack-infra | 14:21 | |
| *** peristeri has joined #openstack-infra | 14:21 | |
| *** jsavak has quit IRC | 14:21 | |
| *** peristeri has quit IRC | 14:21 | |
| *** tzn has quit IRC | 14:21 | |
| *** esp has joined #openstack-infra | 14:21 | |
| *** tcammann has joined #openstack-infra | 14:21 | |
| *** amotoki has joined #openstack-infra | 14:21 | |
| dims | sdague : understood | 14:21 |
| *** _amrith_ is now known as amrith | 14:21 | |
| *** sfinucan has quit IRC | 14:22 | |
| *** NobodyCa1 has joined #openstack-infra | 14:22 | |
| *** jsavak has joined #openstack-infra | 14:22 | |
| *** sfinucan has joined #openstack-infra | 14:22 | |
| *** sabeen has joined #openstack-infra | 14:23 | |
| *** tiswanso has joined #openstack-infra | 14:24 | |
| *** sshnaidm has quit IRC | 14:25 | |
| dims | AJaeger : please remind me as my memory slips, what time do the images get built? | 14:25 |
| *** rhallisey has joined #openstack-infra | 14:26 | |
| AJaeger | dims: start time is 14 or 15:00 UTC - so right now.... | 14:26 |
| dims | AJaeger : ah cool thanks | 14:26 |
| AJaeger | dims: check log files at http://nodepool.openstack.org/ | 14:26 |
| dims | thanks AJaeger | 14:27 |
| openstackgerrit | yolanda.robla proposed openstack-infra/jeepyb: Allow create/update projects with description into Gerrit https://review.openstack.org/110009 | 14:27 |
| *** sabeen3 has joined #openstack-infra | 14:28 | |
| *** tcammann has quit IRC | 14:28 | |
| *** esp has quit IRC | 14:29 | |
| *** NobodyCa1 has quit IRC | 14:29 | |
| *** ccrouch has joined #openstack-infra | 14:29 | |
| *** tiswanso has quit IRC | 14:30 | |
| *** akshai has joined #openstack-infra | 14:30 | |
| *** sabeen has quit IRC | 14:30 | |
| mordred | EmilienM: yes please | 14:30 |
| *** tiswanso has joined #openstack-infra | 14:31 | |
| *** Guest48074 has quit IRC | 14:31 | |
| mordred | EmilienM: I'm glad the module is useful to you! | 14:31 |
| *** amotoki has quit IRC | 14:31 | |
| EmilienM | mordred: our puppet CI will use it for multi node testing | 14:32 |
| *** aovchinnikov has joined #openstack-infra | 14:32 | |
| *** yamamoto has quit IRC | 14:32 | |
| *** yamamoto has joined #openstack-infra | 14:33 | |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/nodepool: Remove failed job from running list https://review.openstack.org/277854 | 14:34 |
| *** jgriffith is now known as jgriffith_away | 14:35 | |
| *** amotoki has joined #openstack-infra | 14:35 | |
| mordred | EmilienM: woot | 14:35 |
| mordred | EmilienM: did you see the craziness we're doing with ansible/puppet and copying hiera files around for infra? :) | 14:35 |
| EmilienM | yes | 14:36 |
| EmilienM | I need to convince crinkle to use Hiera in our CI | 14:36 |
| EmilienM | convaince* | 14:36 |
| mordred | no, you had it right the first time - convince | 14:37 |
| EmilienM | she's right by saying it adds more complexity for our use case | 14:37 |
| *** amotoki has quit IRC | 14:37 | |
| EmilienM | but by doing multi node deployment, I see a lot of interest of using Hiera | 14:37 |
| *** yamamoto has quit IRC | 14:37 | |
| mordred | that reminds me - I need to sync up with nibalizer on why he doesn't want to use automatic parameter lookup | 14:38 |
| *** sdake has joined #openstack-infra | 14:38 | |
| *** jordanP has quit IRC | 14:39 | |
| *** tzn has joined #openstack-infra | 14:39 | |
| yolanda | hi mordred, i remember seeing some comments from crinkle in that review, complaining about bad performance of automatic parameter lookup | 14:39 |
| *** tongli has joined #openstack-infra | 14:40 | |
| mordred | oh - good - my favorite | 14:40 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for test_rebuild_instance_with_volume bug 1543607 https://review.openstack.org/277859 | 14:40 |
| openstack | bug 1543607 in tempest "test_rebuild_instance_with_volume fails on shared storage backends" [Critical,Fix released] https://launchpad.net/bugs/1543607 - Assigned to Davanum Srinivas (DIMS) (dims-v) | 14:40 |
| *** amotoki has joined #openstack-infra | 14:41 | |
| *** amotoki has quit IRC | 14:41 | |
| *** Hal has joined #openstack-infra | 14:42 | |
| *** Hal is now known as Guest88685 | 14:42 | |
| *** jgriffith_away is now known as jgriffith | 14:43 | |
| *** ociuhandu has joined #openstack-infra | 14:43 | |
| fungi | sdague: mordred: to be fair, all i did in https://review.openstack.org/118778 was copy the py33 section in that tox.ini and edit it to be py34. it's not so obvious now because since then the py33 section was removed ;) | 14:44 |
| mordred | fungi: I blame you anyway | 14:44 |
| mordred | fungi: also, good morning | 14:44 |
| fungi | i accept your blame | 14:44 |
| mordred | fungi: also, https://review.openstack.org/#/c/277114 is ready! | 14:44 |
| *** zz_dimtruck is now known as dimtruck | 14:44 | |
| mordred | all hail the new AFS-based per-region APT mirrors | 14:44 |
| openstackgerrit | Merged openstack-infra/gerritlib: Bump pbr and hacking version https://review.openstack.org/199568 | 14:44 |
| openstackgerrit | Merged openstack-infra/gerritlib: Stop specifying requirements.txt in tox.ini https://review.openstack.org/211347 | 14:44 |
| *** ociuhandu has quit IRC | 14:46 | |
| *** EricGonczer_ has joined #openstack-infra | 14:48 | |
| fungi | mordred: out of curiosity, why dd instead of cat with the heredocs? | 14:48 |
| *** julim_ has quit IRC | 14:49 | |
| *** nmagnezi has quit IRC | 14:49 | |
| fungi | also some nodepool images have already started building, so even if we approve it right away we're guaranteed to have some nodes built without it until image updates kick off again tomorrow | 14:51 |
| *** amit213 has quit IRC | 14:51 | |
| *** fawadkhaliq has quit IRC | 14:52 | |
| *** pradk has joined #openstack-infra | 14:52 | |
| dims | mordred : fungi : LOL | 14:53 |
| *** mtanino has joined #openstack-infra | 14:54 | |
| *** links has quit IRC | 14:54 | |
| *** mkoderer has quit IRC | 14:55 | |
| sdague | fungi: shell redirection, right? | 14:55 |
| *** cgross has quit IRC | 14:56 | |
| *** cgross has joined #openstack-infra | 14:56 | |
| *** mkoderer has joined #openstack-infra | 14:56 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 14:57 |
| *** ociuhandu has joined #openstack-infra | 14:58 | |
| *** dizquierdo has joined #openstack-infra | 14:58 | |
| fungi | sdague: i'm not following | 14:58 |
| fungi | there is shell redirection, but you can redirect cat in a shell | 14:59 |
| sdague | right, but the redirect happens outside the sudo | 14:59 |
| sdague | sudo cat foo << > blah | 14:59 |
| sdague | blah is written as normal user | 14:59 |
| fungi | oh! yes i totally missed the sudo | 15:00 |
| sdague | you'd have to nest another shell | 15:00 |
| fungi | so this avoids doing a nested... right | 15:00 |
| *** vmbrasseur has quit IRC | 15:00 | |
| sdague | sudo bash -c is just ugly as sin, so I approve of this dd use | 15:01 |
| *** vmbrasseur has joined #openstack-infra | 15:01 | |
| fungi | yep, completely agree | 15:01 |
| sdague | fyi - https://review.openstack.org/#/c/271434/ - we played around with various PS4 values for devstack and landed on something, that proposes it for devstack gate | 15:02 |
| *** dimtruck is now known as zz_dimtruck | 15:02 | |
| fungi | Clint: the reason for not signing the mirrors (yet) is added complexity of key management for them given they're only being used on untrusted/disposable systems for now (and it discourages other people from trying to use the mirrors in our ci) | 15:02 |
| * Clint nods. | 15:02 | |
| sdague | I'm overall mixed on the fact that these make things more ugly, however, they do make it a ton easier to figure out where things went wrong | 15:02 |
| sdague | http://logs.openstack.org/34/271434/2/check/gate-tempest-dsvm-full/97c1bb9/logs/devstacklog.txt.gz - to see what it looks like in devstack | 15:03 |
| *** Daisy has joined #openstack-infra | 15:03 | |
| fungi | i'm slightly conflicted on the lack of signing, mostly because these machines are also building the sdist tarballs and wheels which we eventually upload to pypi | 15:03 |
| *** EricGonc_ has joined #openstack-infra | 15:03 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for test_rebuild_instance_with_volume bug 1543607 https://review.openstack.org/277859 | 15:03 |
| *** dtantsur is now known as dtantsur|brb | 15:03 | |
| openstack | bug 1543607 in tempest "test_rebuild_instance_with_volume fails on shared storage backends" [Critical,Fix released] https://launchpad.net/bugs/1543607 - Assigned to Davanum Srinivas (DIMS) (dims-v) | 15:03 |
| mordred | fungi: yah - I think we should solve signing next | 15:04 |
| mordred | fungi: it's not THAT hard - we do have the ability to manage secrets and stuff | 15:04 |
| *** Daisy has quit IRC | 15:04 | |
| fungi | yep | 15:04 |
| *** Daisy has joined #openstack-infra | 15:04 | |
| mordred | fungi: I was thinking that we could make a keyring for signing to put into hiera, and then infra-root individuals could fetch it via ssh, sign it and add the signature to the key - since we already have a de-facto trust relationship with infra root humans via ssh to the boxes | 15:05 |
| fungi | sdague: 271434 lgtm | 15:06 |
| *** tiswanso has quit IRC | 15:06 | |
| fungi | i actually like that output, after browsing some of the logs from it | 15:06 |
| *** yamamoto has joined #openstack-infra | 15:07 | |
| *** EricGonczer_ has quit IRC | 15:07 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:07 | |
| fungi | mordred: agreed, that's basically what i was thinking with the first few steps at http://specs.openstack.org/openstack-infra/infra-specs/specs/artifact-signing.html#work-items | 15:08 |
| fungi | expiration was a straw man, to provoke further discussion of key rotation process | 15:08 |
| fungi | i'm not particularly vested in doing it every release cycle, it just was a convenient option | 15:09 |
| *** tiswanso has joined #openstack-infra | 15:09 | |
| aovchinnikov | Hi everyone. Could you please have a quick look at https://review.openstack.org/#/c/277789? It is a change to requirments and now it is blocking a crucial change in manila which came up a bit late before feature freeze. | 15:10 |
| mordred | fungi: nod. so I guess the addition here would be to also add the key to mirror_update.openstack.org so that it could sign the repos | 15:11 |
| mordred | fungi: and that we'd want to make sure to add the key sigs to apt-key on our build slaves when we create them | 15:11 |
| fungi | mordred: i suspect we want more than one key, so that we don't dilute the security of one key by exposing it to more than one worker | 15:14 |
| jeblair | mordred: should i +3 the configure mirror patch? | 15:14 |
| fungi | or at least a different key for signing the package mirrors than for signing tarballs/wheels | 15:14 |
| sdague | https://review.openstack.org/#/c/277833/ - why did that not get tests run on it? | 15:15 |
| fungi | sdague: usual explanation is a circular dependency somewhere. is something depending on that change as far as you're aware? | 15:15 |
| fungi | i'll check zuul logs | 15:15 |
| sdague | fungi: not as far as I know | 15:16 |
| sdague | I just hit revert button in gerrit | 15:16 |
| sdague | so changeid should be new | 15:16 |
| *** EricGonc_ has quit IRC | 15:16 | |
| fungi | oh. i wonder whether that's a new event type in 2.11 | 15:16 |
| jeblair | mtreinish: hi, can we have a conversation about "the direction i want to force on users" ? | 15:16 |
| sdague | fungi: maybe? | 15:16 |
| sdague | it just seemed odd to em | 15:17 |
| sdague | me | 15:17 |
| sdague | I can recheck if needed, but figured it was worth asking | 15:17 |
| mtreinish | jeblair: ? | 15:17 |
| *** ajmiller has joined #openstack-infra | 15:17 | |
| fungi | nope, not a new event type. zuul saw a patchset-created for it and should enqueue. still reading logs | 15:17 |
| jeblair | mtreinish: cause i think i proposed a pretty complete solution in https://review.openstack.org/276318 that requires very little work on your part and "direction you force on users" seems to have a bit of tone to it. | 15:17 |
| mordred | jeblair: yes! | 15:18 |
| jeblair | mordred: done | 15:18 |
| *** sshnaidm has joined #openstack-infra | 15:19 | |
| *** ildikov has quit IRC | 15:19 | |
| *** ildikov has joined #openstack-infra | 15:19 | |
| fungi | zuul enqueued 277833,1 into check at 13:43:16 according to its logs | 15:19 |
| mtreinish | jeblair: there wasn't any tone implied in it, maybe just not thinking clearly because of the jet lag. I was not ever -1 on that patch | 15:19 |
| *** EricGonczer_ has joined #openstack-infra | 15:20 | |
| *** Daisy has quit IRC | 15:20 | |
| mtreinish | I just think changing the status quo makes things worse in the short term even if we have a proposed way out | 15:20 |
| *** Daisy has joined #openstack-infra | 15:20 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP Task Links https://review.openstack.org/277882 | 15:20 |
| jeblair | mtreinish: okay, then let's decide on a way out quickly, because we're about to dig ourselves into a hole that's only going to get harder to get out of later if everyone and their mother wants their own zuul pipeline :) | 15:21 |
| *** piet has joined #openstack-infra | 15:22 | |
| jeblair | mtreinish: so it sounds like investing in the pipeline+project solution isn't a good idea because you immediately see the need for overlapping areas of interest | 15:22 |
| sdague | mtreinish: where should one file UX bugs/rfe for openstack health? | 15:23 |
| jeblair | mtreinish: does the 'zuul build tag' solution work? should we attach tags to jobs, projects, or both? | 15:23 |
| mtreinish | sdague: https://bugs.launchpad.net/openstack-health | 15:23 |
| mtreinish | jeblair: yeah build tags should work fine | 15:23 |
| mtreinish | as long as it ends up in the metadata the gearman log client gets that'll end up as a selectable in the ui | 15:23 |
| mtreinish | it'd probably attach to jobs I think | 15:24 |
| jeblair | mtreinish: yeah, any parameters we pass from zuul to jenkins are accessible by that, so no prob. | 15:24 |
| *** Daisy has quit IRC | 15:25 | |
| jeblair | mtreinish: okay, i'll put them there first; if we want to also attach to projects later, shouldn't be a huge change. | 15:25 |
| openstackgerrit | Merged openstack-infra/project-config: Update configure_mirror to use per-region APT mirrors https://review.openstack.org/277114 | 15:25 |
| mordred | woot | 15:25 |
| mordred | jeblair: so, now we need to kick new images I think yeah? | 15:25 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP Task Links https://review.openstack.org/277882 | 15:26 |
| jeblair | mordred: yeah, though builds are already in progress :( | 15:26 |
| jeblair | or should be at least | 15:26 |
| jeblair | depending on whether things are still broken | 15:26 |
| fungi | sdague: for some reason zuul ended up raising an exception when trying to report that change's results in gerrit http://paste.openstack.org/show/486448/ | 15:26 |
| sdague | mtreinish: https://bugs.launchpad.net/openstack-health/+bug/1543635 | 15:27 |
| openstack | Launchpad bug 1543635 in OpenStack Health "In efficient project list on main page" [Undecided,New] | 15:27 |
| *** hashar has quit IRC | 15:28 | |
| *** rbrndt has joined #openstack-infra | 15:28 | |
| *** EricGonczer_ has quit IRC | 15:28 | |
| *** shardy has quit IRC | 15:28 | |
| *** jlanoux has quit IRC | 15:29 | |
| mtreinish | sdague: that's a good idea | 15:30 |
| mordred | jeblair: well, I suppose we cna also just wait until new images tomorrow | 15:30 |
| jeblair | fungi: that should only happen if some jobs weren't launched.. | 15:30 |
| mordred | jeblair: I have released the locks, btw, so cron should be doing its thing | 15:30 |
| *** zz_dimtruck is now known as dimtruck | 15:31 | |
| mordred | jeblair, fungi: have we done any announcements about new mirrors? | 15:32 |
| fungi | mordred: not to my knowledge | 15:33 |
| mordred | k. I've got a first draft - I'll etherpad it up in a sec | 15:33 |
| *** infra-red has quit IRC | 15:33 | |
| AJaeger | it's worth a #success as well ;) | 15:34 |
| fungi | jeblair: at 13:43:43 in the log i see it submitting a merger:merge job into gearman for that change and then the very next log lines are reporting that change, no jobs for that change, removing it from the pipeline | 15:34 |
| AJaeger | jeblair: thanks for following up on the pipelines. Ok to +A these now? | 15:34 |
| *** annegentle has joined #openstack-infra | 15:35 | |
| jeblair | AJaeger: i think so | 15:35 |
| *** Guest22569 has quit IRC | 15:35 | |
| *** jpr has joined #openstack-infra | 15:35 | |
| AJaeger | jeblair: ok, will do soon. | 15:35 |
| fungi | but then down at 14:31:45 it has a merge complete for that, and then just below there says it's unable to merge | 15:36 |
| fungi | oh, i may be conflating log entries for check and merge-check pipelines. i need to look a little closer | 15:37 |
| jasondotstar | question guys | 15:38 |
| jasondotstar | GM btw. | 15:38 |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events https://review.openstack.org/277386 | 15:38 |
| jasondotstar | does the -infra team 'sponsor' dev environments (and I use that term loosely) | 15:38 |
| anteaya | jasondotstar: women work in this channel too | 15:39 |
| jasondotstar | working with the openstack-salt team, and we are looking for a cloud based dev environment to test | 15:39 |
| anteaya | unless you are only addressing the men | 15:39 |
| openstackgerrit | Stephen Gordon proposed openstack-infra/project-config: Skip magnum functional test jobs on docs changes https://review.openstack.org/277892 | 15:39 |
| monester | Hi all! please review my patch to python-gerritlib, https://review.openstack.org/#/c/277773/ | 15:39 |
| monester | it really can reduce code in some applications | 15:40 |
| jasondotstar | anteaya: no malice intended. i use the term 'guys' loosely | 15:40 |
| openstackgerrit | Omer Anson proposed openstack/requirements: Add scapy and python-pytun https://review.openstack.org/277893 | 15:41 |
| *** krtaylor has quit IRC | 15:42 | |
| anteaya | yeah well some terms that are gender inclusive go over far better herer | 15:42 |
| anteaya | here | 15:42 |
| anteaya | including team, group, folks, people, infra | 15:43 |
| jasondotstar | thanks anteaya, I'll keep that in mind. | 15:43 |
| anteaya | thank you | 15:43 |
| mordred | jasondotstar: when you say "cloud based dev environment" - what do you mean? | 15:43 |
| mordred | jasondotstar: (I have some suggestions, depending on what you're wanting to accomplish) | 15:44 |
| jasondotstar | well, we're looking to test our salt formulas. | 15:45 |
| *** julim has joined #openstack-infra | 15:45 | |
| *** woodster_ has joined #openstack-infra | 15:45 | |
| jasondotstar | those salt formulas are being used to stand up openstack services (obviously) | 15:45 |
| dprince | Hi infra. We are seeing continued FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel, errors in the TripleO jobs today | 15:46 |
| jasondotstar | so the ability to standing up the server side, and minions | 15:46 |
| sigmavirus24 | lifeless: when you pop back around, can you take a look at https://bugs.launchpad.net/pbr/+bug/1542383 ? | 15:46 |
| openstack | Launchpad bug 1542383 in PBR "wsgi_scripts and python wheels" [Undecided,New] | 15:46 |
| dprince | This is using Fedora jenkins slaves. Anyone else seeing similar issues? | 15:46 |
| clarkb | dprince: magnum is/was, the slave(s) were completely unreachable | 15:46 |
| * clarkb wonders if tests are doing funny things with networking/containers | 15:47 | |
| jasondotstar | from VMs as opposed to a local dev enviornment (vagrant) | 15:47 |
| dprince | clarkb: ah, good to know | 15:47 |
| *** marcusvrn_ has quit IRC | 15:47 | |
| dprince | clarkb: example for us here http://logs.openstack.org/42/277742/1/check-tripleo/gate-tripleo-ci-f22-nonha/439c77d/console.html | 15:47 |
| *** yamamoto has quit IRC | 15:48 | |
| jasondotstar | mordred: ^^^ | 15:48 |
| clarkb | dprince: when I investigated the magnum problems nova console-log showed a dozen or so messages about processes not responding for >120 seconds | 15:50 |
| *** shardy has joined #openstack-infra | 15:51 | |
| clarkb | dprince: and that VM was not accepting ssh connections (and when left overnight didn't accept them in the morning so wasn't just a temporary situation) | 15:51 |
| AJaeger | team, yesterday two new repos were setup for pabelanger : ansible-role-jenkins and ansible-role-jenkins-plugins. The import worked fine, see http://git.openstack.org/cgit/openstack/ansible-role-jenkins but github has no content: https://github.com/openstack/ansible-role-jenkins (same for the other repo). | 15:51 |
| *** angdraug has joined #openstack-infra | 15:51 | |
| jeblair | fungi: i believe there is a recently introduced zuul bug | 15:51 |
| clarkb | dprince: our theory for them was that they were digging into swap deep enough that oomkiller may have been invoked but I half expected to see evidence of that in the nova console-log output | 15:51 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Do not mention "independent" in announcements https://review.openstack.org/277902 | 15:51 |
| dprince | clarkb: okay, so what I read here is it could potentially be an issue on our side | 15:52 |
| clarkb | dprince: yes, at least for the magnum tests we know that other tests are functioning on those images and on those cloud regions | 15:52 |
| *** EricGonczer_ has joined #openstack-infra | 15:52 | |
| clarkb | dprince: as a result we expect the oddity is a result of the specific job run | 15:52 |
| pabelanger | AJaeger: Ya, not sure why it is not replicating atm | 15:53 |
| *** EricGonc_ has joined #openstack-infra | 15:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove periodic-neutron pipeline in favor of periodic https://review.openstack.org/276317 | 15:53 |
| openstackgerrit | Merged openstack-infra/project-config: Remove periodic-qa pipeline in favor of periodic https://review.openstack.org/276318 | 15:54 |
| fungi | jeblair: yep, had to step away for a moment but was wondering if we've somehow broken merge failure reporting | 15:54 |
| fungi | was about to test that theory | 15:54 |
| *** isaacb has quit IRC | 15:54 | |
| fungi | it clicked that several third-party ci operators had wandered in over the past week or so complaining about KeyError: 'LOG_PATH' | 15:55 |
| mordred | jasondotstar: but you want to be able to do that as devs, not necessarily talking about doing that in the gate at th emoment, yeah? | 15:55 |
| *** nikhil is now known as nikhil_k | 15:56 | |
| jasondotstar | mordred: right. eventually a beaker-esque gate would be necessary, but for now, just something for us to test our formulas with as the devs | 15:56 |
| *** EricGonczer_ has quit IRC | 15:56 | |
| mordred | yah - so... | 15:56 |
| mordred | we don't really have anything like that we can give people from infra | 15:57 |
| mordred | jasondotstar: first suggestion is trystack.org - which will let you spin up nodes and do things with them but your nodes go away from time to time | 15:57 |
| mordred | jasondotstar: what I do is just get a cloud account on one of the public openstack clouds and make my employer pay for the amex bill | 15:57 |
| mordred | jasondotstar: or, if you do have local machines | 15:57 |
| mordred | jasondotstar: just running devstack on one of them and then using that as a cloud to get vms from is not a terrible way to go | 15:58 |
| jasondotstar | mordred: trystack is an option - the 24 hour termination thing might be an issue | 15:58 |
| jasondotstar | mordred: seeing what my employer is willing to fund is another | 15:58 |
| openstackgerrit | Dina Belova proposed openstack/requirements: Make osprofiler requirements >1.0.0 https://review.openstack.org/277583 | 16:00 |
| cznewt | mordred: we will be hosting lab environments for openstack-salt deployments, you can try that | 16:00 |
| *** esikachev has quit IRC | 16:00 | |
| *** annegentle has quit IRC | 16:01 | |
| *** esker has joined #openstack-infra | 16:01 | |
| *** annegentle has joined #openstack-infra | 16:02 | |
| mordred | jasondotstar: ^^ | 16:02 |
| *** eharney has joined #openstack-infra | 16:02 | |
| jasondotstar | cznewt: ack. | 16:02 |
| *** EricGonc_ has quit IRC | 16:02 | |
| openstackgerrit | Tom Cocozzello proposed openstack/requirements: Bump upper constraint on eventlet to 0.18.2 https://review.openstack.org/277911 | 16:03 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack/requirements: bump eventlet version from 0.18.1 to 0.18.2 https://review.openstack.org/277912 | 16:03 |
| markus_z | dims: ^ | 16:03 |
| *** gokrokve has joined #openstack-infra | 16:04 | |
| markus_z | dims: Whom do I notify about this patch too? | 16:04 |
| jeblair | fungi: i think i see it; i'll link you to it after breakfast. | 16:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename periodic networking jobs https://review.openstack.org/276367 | 16:05 |
| *** jlanoux has joined #openstack-infra | 16:05 | |
| AJaeger | dougwig, anteaya: Could you review the above change, please? | 16:05 |
| *** ddecapit has joined #openstack-infra | 16:05 | |
| *** ddecapit is now known as DuaneDeC7 | 16:06 | |
| *** kzaitsev_mb has quit IRC | 16:06 | |
| markus_z | dims: There is another patch too, just seconds older: https://review.openstack.org/#/c/277911/ | 16:07 |
| *** armax has joined #openstack-infra | 16:08 | |
| *** fawadkhaliq has joined #openstack-infra | 16:08 | |
| anteaya | AJaeger: just finishing up reading scrollback | 16:08 |
| *** sc68cal has joined #openstack-infra | 16:08 | |
| anteaya | AJaeger: then I will review, thank you for writing the patch | 16:08 |
| AJaeger | thanks, anteaya | 16:09 |
| anteaya | oh I thought we had renamed them | 16:09 |
| fungi | jeblair: thanks, got sidetracked with a meeting | 16:09 |
| anteaya | unless that is not a new patch | 16:09 |
| *** DuaneDeC7 has quit IRC | 16:09 | |
| AJaeger | anteaya: it's a rebase plus a change to add -master | 16:09 |
| AJaeger | let me comment | 16:09 |
| anteaya | ah okay yes, will review, almost done backscroll | 16:09 |
| *** EricGonczer_ has joined #openstack-infra | 16:10 | |
| AJaeger | anteaya: you reviewed the very first iteration of it - now we're at the third... | 16:10 |
| *** burgerk has joined #openstack-infra | 16:10 | |
| *** EricGonc_ has joined #openstack-infra | 16:10 | |
| clarkb | greghaynes: figured out that exception not being handled quite right from yesterday. the image upload worker needs an onFailed() method to cleanup the snapshot table | 16:11 |
| *** DuaneDeC7 has joined #openstack-infra | 16:11 | |
| *** boris-42 has joined #openstack-infra | 16:11 | |
| AJaeger | anteaya: and please review 274930 again and consider removing your -2. | 16:11 |
| greghaynes | clarkb: ah | 16:11 |
| anteaya | AJaeger: I knew I remembered it | 16:11 |
| clarkb | greghaynes: I will probably make a stab at that later today | 16:11 |
| anteaya | AJaeger: if that is the pipeline thing then yes, I'm fine to remove -2 if it points to periodic | 16:11 |
| greghaynes | clarkb: did you see the gear client blocking issue I mentioned? | 16:11 |
| * AJaeger just checking whether we have all pieces together | 16:11 | |
| anteaya | AJaeger: thanks for the juggling | 16:12 |
| clarkb | greghaynes: I saw that you mentioned it but haven't had a chance to look into it yet | 16:12 |
| *** daemontool__ has joined #openstack-infra | 16:12 | |
| *** daemontool__ is now known as daemontool | 16:13 | |
| greghaynes | clarkb: I commented on the review for handling failed delete with a traceback | 16:13 |
| jeblair | fungi, clarkb, jhesketh: see comments on https://review.openstack.org/223298 | 16:13 |
| *** krtaylor has joined #openstack-infra | 16:14 | |
| jeblair | (be sure to read both comments) | 16:14 |
| EmilienM | mordred: so the current status of ansible/puppet module does not shot any output of puppet runs, right? | 16:14 |
| *** EricGonczer_ has quit IRC | 16:14 | |
| *** daemontool_ has quit IRC | 16:15 | |
| *** annegentle has quit IRC | 16:16 | |
| AJaeger | anteaya: could you review a small cleanup job to remove unused checks as well, please? Nothing critical, just housekeeping: 276267 | 16:16 |
| *** annegentle has joined #openstack-infra | 16:16 | |
| jeblair | EmilienM: i think there was a change to make it an option? | 16:16 |
| EmilienM | jeblair: I don't see any param for that in the system/puppet.py file | 16:17 |
| anteaya | jasondotstar cznewt so are you looking to revive all the salt-formula repos that are currently in the stackforge name space? http://git.openstack.org/cgit/?ofs=600 | 16:17 |
| jeblair | EmilienM: but we turned it off because it made the ansible output unreadable if puppet output a lot of data, which it does if something goes wrong, which is just when you want to be able to read your log files. so we look on-host instead. | 16:17 |
| anteaya | AJaeger: sure | 16:17 |
| jeblair | EmilienM: ah here it is https://review.openstack.org/271086 | 16:17 |
| AJaeger | thanks, anteaya | 16:17 |
| jasondotstar | anteaya: yes. | 16:18 |
| openstackgerrit | Vincent Françoise proposed openstack-infra/project-config: Watcher multinode Tempest job fixup https://review.openstack.org/277923 | 16:18 |
| anteaya | jasondotstar: good so have you got patches up to move those repos? | 16:19 |
| dulek | Gerrit seems slow again and I see 502 errors occasional. | 16:19 |
| jasondotstar | anteaya: we've created new ones | 16:19 |
| fungi | jeblair: ahh, yep, on regressions you pointed out in 223298, makes sense. it looks like all the needed-by changes are still open, so should we revert or try to tweak it as is? | 16:19 |
| fungi | jhesketh: ^ | 16:19 |
| anteaya | jasondotstar: new ones of what? | 16:19 |
| dulek | Last time restart fixed this. | 16:19 |
| jasondotstar | so we'll have to look at what's in the ones on the stackfoge namespace | 16:19 |
| *** amaretskiy1 has left #openstack-infra | 16:20 | |
| jasondotstar | anteaya: there are some that are in the openstack namespace too | 16:20 |
| anteaya | why were new repos created? | 16:20 |
| jeblair | fungi: i think it will be a simple fix. | 16:20 |
| *** isaacb has joined #openstack-infra | 16:20 | |
| EmilienM | jeblair: I'm talking about https://github.com/ansible/ansible-modules-extras/blob/devel/system/puppet.py | 16:20 |
| fungi | dulek: i object to the term "fixed" anytime a reboot is involved (except when having to do with kernels) | 16:20 |
| clarkb | I can confirm via melody that GC time is spiking | 16:20 |
| clarkb | before we restart we should get zaro's connection queuing change in (which updates it to match the gerrit 2.12 default) | 16:21 |
| jasondotstar | anteaya: those are the ones we are currently iterating against | 16:21 |
| fungi | jeblair: are you working on a fix, or should i, or were you thinking of waiting until jhesketh is around in case he wants to hack on it? | 16:21 |
| *** annegentle has quit IRC | 16:21 | |
| anteaya | clarkb: what is the url of the change? | 16:21 |
| clarkb | anteaya: digging that up now | 16:21 |
| *** [HeOS] has joined #openstack-infra | 16:21 | |
| clarkb | oh looks like it did already merge | 16:21 |
| anteaya | jasondotstar: my question is why did you create new ones, or better yet why were your patches to create new ones approved | 16:21 |
| clarkb | so next restart will pick up that new configuration value | 16:22 |
| jeblair | EmilienM: so am i :) | 16:22 |
| anteaya | since we have repos collected dust | 16:22 |
| jeblair | fungi: i am | 16:22 |
| anteaya | clarkb: oh awesome | 16:22 |
| openstackgerrit | Merged openstack-infra/project-config: Add python[3]-jobs and check-requirements to freezer* https://review.openstack.org/269138 | 16:22 |
| fungi | jeblair: thanks! | 16:22 |
| *** HeOS has quit IRC | 16:23 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/jenkins-job-builder: Remove ordereddict support from py26 https://review.openstack.org/240438 | 16:23 |
| *** annegentle has joined #openstack-infra | 16:24 | |
| *** dtantsur|brb is now known as dtantsur | 16:24 | |
| jeblair | EmilienM: if you're using the version in ansible, then i believe the output is reported as the module stdout | 16:24 |
| SotK | Is there any chance someone can kill off the rogue workers on storyboard.o.o *before* we find a way to fix it please? I've so far failed to find a way to reliably reproduce the issue (but have managed to occasionally), and have other things to work on which is making it difficult to find a solution quickly (and the situation is causing delays in our changes making it into production). | 16:25 |
| jeblair | EmilienM: if you're using our version, then it is not, except that we are in the process of adding an option to tell puppet where to log, and then, if you don't set that option, it will be sent to stdout. eventually, this will merge into ansible. | 16:25 |
| jasondotstar | anteaya: the tcpcloud team is our most significant contributor to the salt effort at this point. they've worked on the salt formulas on github for quite sometime. we created the new repos to migrate the work over from their github repos into openstack as new projects | 16:26 |
| isaacb | clarkb, yolanda: Hi, based on our discussion the other day, can you please have a look in: https://review.openstack.org/#/c/262787/ | 16:26 |
| jasondotstar | im saying this not knowing what work was done in the repos inside the stackforge namespace | 16:26 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/infra-manual: Fix location of GerritBot channels config https://review.openstack.org/277933 | 16:27 |
| *** vgridnev has quit IRC | 16:27 | |
| jasondotstar | anteaya: ^ | 16:28 |
| mmedvede | I am looking to get puppet-ciwatch first patch approved, is anyone able to take a peek? https://review.openstack.org/#/c/238606/ | 16:28 |
| *** shashank_hegde has joined #openstack-infra | 16:28 | |
| jasondotstar | cznewt: do you have anything to add? the question was 'why did we create new repos, as opposed to migrating over the stackforge repos already in existance' | 16:29 |
| jeblair | we imported stackforge repos into openstack? | 16:29 |
| anteaya | no, they got recreated | 16:30 |
| anteaya | we now have a set of salt repos in both openstack and stackforge | 16:30 |
| *** ccrouch has quit IRC | 16:30 | |
| *** ccrouch has joined #openstack-infra | 16:30 | |
| jeblair | so we imported them from github into both namespaces? | 16:30 |
| mordred | I think a new set of completely unrelated repos got created that are not based on the repos that were in stackforge | 16:31 |
| pabelanger | that is how I understand it too | 16:31 |
| EmilienM | jeblair: ok I'm using the one in ansible extras. But I'm interested by having your code in the ansible repo too :-) | 16:31 |
| anteaya | which are both called salt formula | 16:31 |
| anteaya | AJaeger: I think I have done the list | 16:31 |
| mordred | EmilienM: ya - we were in sync, then we added some stuff locally that I think I still need to sync to upstream | 16:31 |
| AJaeger | thanks a lot, anteaya | 16:31 |
| anteaya | AJaeger: welcome, thanks for being so organized | 16:31 |
| *** avihou has joined #openstack-infra | 16:32 | |
| jeblair | oh, so we have stackforge/nova-salt-formula and openstack/salt-formula-nova ? | 16:32 |
| EmilienM | mordred: ok so iiuc you first patch your module and then upstream, right? | 16:32 |
| * jeblair goes back to zuul | 16:32 | |
| pabelanger | anteaya: well, technically stackforge is glance-salt-formula and openstack is salt-formula-glance. So, not the same namespace | 16:32 |
| pabelanger | jeblair: ^ | 16:32 |
| anteaya | jeblair: yes | 16:33 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Handle image upload job failures https://review.openstack.org/277941 | 16:34 |
| *** savihou has quit IRC | 16:34 | |
| clarkb | jeblair: greghaynes ^ that should correct the issue, I really should add a test but wanted to get something written before I forgot the problem | 16:34 |
| anteaya | clarkb: so did you restart gerrit and I missed it? | 16:34 |
| clarkb | anteaya: no I havne't | 16:34 |
| anteaya | okay thanks | 16:34 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Notify #openstack-dev of governance changes https://review.openstack.org/277943 | 16:35 |
| AJaeger | anteaya: if you have time: 269809 would be great as well so that annegentle can move forward with the API documents. Could you review, please? | 16:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Make puppet logout destination configurable https://review.openstack.org/271086 | 16:36 |
| greghaynes | clarkb: replied | 16:37 |
| mordred | EmilienM, jeblair: ^^ we need to land that before we can really upstream our changes | 16:37 |
| pleia2 | good morning | 16:37 |
| *** vincentfrancoise has joined #openstack-infra | 16:37 | |
| clarkb | greghaynes: that is what I get for copy pasta | 16:37 |
| clarkb | greghaynes: will fix | 16:37 |
| greghaynes | yep :) | 16:37 |
| EmilienM | mordred: no problem, just asking the workflow. I'm wondering which module should I use now. | 16:37 |
| EmilienM | mordred: for puppet openstack CI... | 16:37 |
| mordred | EmilienM: use the one in ansible | 16:37 |
| *** sridhar_ram has joined #openstack-infra | 16:38 | |
| anteaya | AJaeger: yes, done | 16:38 |
| mordred | EmilienM: our copy in infra is for convenience for us to be able to make changes when we see a production issue and can't wait for upstream to land a patch | 16:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Handle image upload job failures https://review.openstack.org/277941 | 16:38 |
| anteaya | morning pleia2 | 16:38 |
| EmilienM | mordred: ok Sir. | 16:38 |
| *** abregman has quit IRC | 16:38 | |
| *** savihou has joined #openstack-infra | 16:38 | |
| AJaeger | thanks, anteaya. | 16:38 |
| anteaya | welcome | 16:38 |
| AJaeger | good morning, pleia2 ! | 16:38 |
| EmilienM | mordred: just want to make sure all your nice patches in infra module go upstream | 16:38 |
| jasondotstar | jeblair: yes, that's correct | 16:38 |
| pleia2 | been chatting with HPE folks at our sprint venue in ft collins, so I added an agenda item to see if we need anymore info from them | 16:39 |
| jasondotstar | more accurately, we have stackforge/nova-salt-formula and openstack/salt-formula-nova | 16:40 |
| mordred | EmilienM: I'm about to send my one patch there now | 16:40 |
| jasondotstar | jeblair: ^ | 16:40 |
| pleia2 | I'm not renting a car, so my own questions have mostly been travel-based, but other people may have other things | 16:40 |
| *** piet has quit IRC | 16:40 | |
| *** avihou has quit IRC | 16:40 | |
| *** salv-orl_ has joined #openstack-infra | 16:41 | |
| mordred | EmilienM, jeblair: https://github.com/ansible/ansible-modules-extras/pull/1644 | 16:41 |
| clarkb | I just hope that we have weather like they are having this week | 16:41 |
| clarkb | I won't have to dig out any warmer clothing | 16:41 |
| *** egonzalez has quit IRC | 16:41 | |
| *** bpokorny has joined #openstack-infra | 16:42 | |
| EmilienM | mordred: I'm wondering how to return structured logs in ansible output | 16:43 |
| mordred | EmilienM: it's possible to write ansible logging plugins that might give you more flexibility | 16:44 |
| cznewt | anteaya: the stackforge salt repos are not ready and seem retired [as read from README] | 16:44 |
| *** krtaylor has quit IRC | 16:44 | |
| *** DuaneDeC7 has quit IRC | 16:44 | |
| *** salv-orlando has quit IRC | 16:44 | |
| anteaya | pleia2: now I understand what you are saying | 16:44 |
| EmilienM | mordred: I actually prefer syslog, that's why I asked you about this patch. shipit! lol | 16:44 |
| *** bpokorny has quit IRC | 16:45 | |
| anteaya | cznewt: well the point of the retirement is that should anyone want to continue the work they take them out of retirement | 16:45 |
| *** bpokorny has joined #openstack-infra | 16:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Check branch-override in ofagent job-template https://review.openstack.org/274930 | 16:45 |
| vincentfrancoise | Hi everyone. In https://review.openstack.org/#/c/275167/, I created a new "watcher-dashboard" project along with its "watcher-dashboard-core" group. The thing is that no one is currently in, so could one of you put me in? | 16:46 |
| *** piet has joined #openstack-infra | 16:46 | |
| *** yamahata has quit IRC | 16:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove git-commit-message check https://review.openstack.org/276267 | 16:47 |
| anteaya | vincentfrancoise: thanks for your reviews on project-config, I appreciate you taking the time to help | 16:47 |
| openstackgerrit | Merged openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 16:47 |
| vincentfrancoise | anteaya: you're welcome | 16:48 |
| anteaya | vincentfrancoise: I'm not a gerrit admin so can't fulfill your request | 16:49 |
| anteaya | but stand by hopefully someone can when they have a spare moment | 16:49 |
| *** esker has quit IRC | 16:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix default merge failure reports https://review.openstack.org/277958 | 16:49 |
| vincentfrancoise | ok nice | 16:49 |
| anteaya | pabelanger: so are you attending infra sprint? The wikipage looks like it says you are, are you? | 16:49 |
| cznewt | anteaya: but there's not much to continue from, I've checked them and they are not complete, they serve one specific deployment, the target to our formulas is to support variety of deployents | 16:50 |
| anteaya | look it is done | 16:50 |
| jeblair | fungi, clarkb, jhesketh, sdague: https://review.openstack.org/277958 fixes a bug introduced with zuul connections that caused us to stop reporting merge failures on check/gate pipelines. | 16:50 |
| fungi | SotK: i've cleaned up and restarted the storyboard-workers on storyboard.o.o for now | 16:51 |
| anteaya | is it just a waste in my mind, as reusing the repos would have kept us from creating more the almost the same name | 16:51 |
| pabelanger | anteaya: Yes, I need to update passport today. But will be attending | 16:51 |
| anteaya | but we can't delete repos once they are created, so now we have lots | 16:51 |
| anteaya | pabelanger: wow cutting it close, how long will it take you to get your updated passport? | 16:51 |
| Zara | fungi: thank you! :) much appreciated. | 16:51 |
| pabelanger | anteaya: you can do 24hours, just have to pay for it | 16:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix default merge failure reports https://review.openstack.org/277960 | 16:52 |
| fungi | anteaya: cznewt: well, more important is if they shared substantial git history (or if the team working on the new ones could have easily based their work on the old retired repos after simply reverting the retirement commit on HEAD) | 16:52 |
| pabelanger | anteaya: need travel docs to show | 16:52 |
| *** DuaneDeC7 has joined #openstack-infra | 16:52 | |
| *** shashank_hegde has quit IRC | 16:52 | |
| anteaya | pabelanger: really? I can't do that as I have to travel to whitby | 16:52 |
| pabelanger | anteaya: but, 7days is enough time. | 16:52 |
| anteaya | and they have to mail it to me, but you are in ottawa | 16:52 |
| *** scheuran has quit IRC | 16:52 | |
| pabelanger | anteaya: Ya, helps being in Ottawa | 16:52 |
| pabelanger | yup | 16:52 |
| anteaya | so must be faster there | 16:52 |
| pabelanger | I just walk into the passport office for drop off and pickup | 16:53 |
| anteaya | fungi: well radar got reused and entired changed the git history | 16:53 |
| anteaya | pabelanger: nice I'm not allowed to pickup | 16:53 |
| anteaya | I have to get it mailed to me | 16:53 |
| anteaya | pabelanger: guess I'll consider ottawa an option if I want to pickup then | 16:53 |
| anteaya | thanks | 16:53 |
| anteaya | s/entired/entirely | 16:54 |
| cbader | anyone else seeing an issue with eventlet previously set to 0.18.1 and then changed to 0.18.2 and job breaking? | 16:54 |
| anteaya | but if everyone else if fine with folks just creating new gerrit repos, then so be it | 16:55 |
| clarkb | cbader: markus_z sent mail to the dev list about it | 16:55 |
| *** krtaylor has joined #openstack-infra | 16:55 | |
| clarkb | cbader: tl;dr is they deleted 0.18.1 which we hvae set in constraints | 16:55 |
| markus_z | cbader: http://openstack.markmail.org/thread/jwsmdos4pugtp7nw | 16:56 |
| openstackgerrit | Merged openstack-infra/glean: Add missing entry for localhost https://review.openstack.org/267672 | 16:56 |
| *** apoorvad has joined #openstack-infra | 16:56 | |
| *** mikelk has quit IRC | 16:57 | |
| *** zeih_ has joined #openstack-infra | 16:57 | |
| *** jgriffith is now known as jgriffith_away | 16:58 | |
| *** dimtruck is now known as zz_dimtruck | 16:58 | |
| *** annegentle has quit IRC | 16:58 | |
| *** zz_dimtruck is now known as dimtruck | 16:58 | |
| pabelanger | wow, never seen this jenkins error before: https://jenkins07.openstack.org/job/gate-windmill-deploy-centos7-nv/3/consoleText | 16:58 |
| pabelanger | no workspace from node | 16:59 |
| *** jlanoux has quit IRC | 16:59 | |
| *** achanda has joined #openstack-infra | 16:59 | |
| *** nadya has joined #openstack-infra | 16:59 | |
| *** achanda has quit IRC | 16:59 | |
| *** nadya is now known as Guest71587 | 17:00 | |
| *** ccrouch has quit IRC | 17:00 | |
| *** jlanoux has joined #openstack-infra | 17:00 | |
| *** ccrouch has joined #openstack-infra | 17:01 | |
| *** Guest10250 has joined #openstack-infra | 17:01 | |
| *** Guest71587 has quit IRC | 17:02 | |
| anteaya | new one for me too, it couldn't even get the node environment variables loaded | 17:02 |
| *** d0ugal has quit IRC | 17:02 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP Task Links https://review.openstack.org/277882 | 17:03 |
| *** DuaneDeC7 has quit IRC | 17:03 | |
| *** DuaneDeC7 has joined #openstack-infra | 17:03 | |
| *** jaosorior has quit IRC | 17:04 | |
| dims | fungi : clarkb : should i be able to do build_name:"periodic-.*-py27-with-oslo-master" on http://logstash.openstack.org/ ? | 17:04 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add H203 to check that assertIs(Not)None is used https://review.openstack.org/276517 | 17:04 |
| clarkb | dims: just * not .* iirc | 17:04 |
| *** kzaitsev_mb has joined #openstack-infra | 17:04 | |
| cznewt | anteaya: we have more that 3 years of development of the formulas with variety of deployemnt options, there was good deal of research on existing solutions and their scope was always limited to single node deploys | 17:05 |
| *** chrall__ has quit IRC | 17:05 | |
| dims | clarkb : that does not seem to work either | 17:05 |
| fungi | dims: lucene patterns, not regex | 17:05 |
| dims | fungi : ah | 17:05 |
| fungi | dims: do you know if those jobs are indexed? | 17:05 |
| fungi | (into elasticsearch) | 17:06 |
| *** savihou has quit IRC | 17:06 | |
| clarkb | oh drop the quotes | 17:06 |
| clarkb | quotes treat * literally | 17:06 |
| *** daemontool has quit IRC | 17:06 | |
| dims | fungi : y i can see specific ones | 17:06 |
| dims | clarkb : i see, let me try that | 17:06 |
| anteaya | cznewt: like I said, it is done | 17:07 |
| *** isaacb has quit IRC | 17:07 | |
| dims | perfect thanks clarkb and fungi | 17:07 |
| openstackgerrit | Merged openstack-infra/glean: Add link to documentation on network_info.json https://review.openstack.org/276003 | 17:07 |
| openstackgerrit | Merged openstack-infra/glean: Remove the manual start of network interfaces https://review.openstack.org/276610 | 17:08 |
| *** Guest10250 has quit IRC | 17:08 | |
| *** jlanoux has quit IRC | 17:10 | |
| *** piet has quit IRC | 17:10 | |
| *** sdake has quit IRC | 17:11 | |
| anteaya | nibalizer: I'm unclear on why this patch was abandoned and split into two patches: https://review.openstack.org/#/c/262835/ | 17:11 |
| *** gyee has joined #openstack-infra | 17:12 | |
| *** sdake has joined #openstack-infra | 17:13 | |
| anteaya | I'm currently getting gerrit errors on every attempt to take any action | 17:13 |
| anteaya | as clarkb said in backscroll he feels we are in gc | 17:13 |
| anteaya | does any infra-root have time to do a restart? | 17:13 |
| *** ybathia has joined #openstack-infra | 17:14 | |
| *** yamahata has joined #openstack-infra | 17:14 | |
| fungi | anteaya: i can... is the current situation more disruptive than a brief complete outage from the restart will be? | 17:14 |
| fungi | since i rarely use the webui and the client i'm using caches and retries queries, i don't tend to notice | 17:14 |
| anteaya | yes for me it is | 17:14 |
| anteaya | any other folks care to weigh in on their opinion of their current gerrit experience? | 17:15 |
| Zara | yes, for me too | 17:15 |
| anteaya | fungi: fair enough | 17:15 |
| fungi | infra-root: objections to me restarting gerrit right now to alleviate the current webui errors? | 17:16 |
| mordred | fungi: do it | 17:16 |
| openstackgerrit | Timur Sufiev proposed openstack/requirements: Remove version exclusions for selenium https://review.openstack.org/276288 | 17:18 |
| zaro | morning | 17:18 |
| anteaya | morning zaro | 17:18 |
| *** jcoufal has quit IRC | 17:19 | |
| *** samuelBartel has quit IRC | 17:20 | |
| zaro | fungi: a change merged that might alleviate the problem. so might be a good idea to restart anyways. | 17:21 |
| zaro | https://review.openstack.org/275995 | 17:21 |
| fungi | zaro: oh, excellent. have we not restarted since the 5th? | 17:22 |
| fungi | yeah, running since 2016-02-02 | 17:22 |
| fungi | #status notice Gerrit is restarting now, to alleviate current performance impact and WebUI errors. | 17:23 |
| openstackstatus | fungi: sending notice | 17:23 |
| -openstackstatus- NOTICE: Gerrit is restarting now, to alleviate current performance impact and WebUI errors. | 17:24 | |
| fungi | it's on its way back up | 17:24 |
| anteaya | fungi: thank you | 17:24 |
| fungi | claims to be running again | 17:24 |
| *** tiswanso has quit IRC | 17:24 | |
| anteaya | zaro: thanks for the backport, let's hope it is an improvement | 17:24 |
| mriedem | fungi: it is | 17:24 |
| anteaya | gerrit is up for me | 17:24 |
| fungi | webui seems to be working for me anyway | 17:24 |
| zaro | yep, works here too | 17:24 |
| fungi | i guess let's keep an eye on it and see if this helps the next time java gc is running hard for a while | 17:25 |
| openstackstatus | fungi: finished sending notice | 17:25 |
| anteaya | wonderful, thank you | 17:25 |
| *** matrohon has quit IRC | 17:25 | |
| * anteaya crosses her fingers the new patch will be an improvement | 17:26 | |
| *** tiswanso has joined #openstack-infra | 17:26 | |
| zaro | fungi: yep, check back in about 1 week from today. | 17:27 |
| *** DuaneDeC7 has quit IRC | 17:28 | |
| pleia2 | in puppet, is there a difference between using $admin_password, and $admin_password = '', ? | 17:29 |
| pleia2 | we seem to use both, sometimes in the same file, like in http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/init.pp | 17:29 |
| pleia2 | have $mysql_password, and $ldap_password = '', | 17:29 |
| anteaya | jeblair: current zuul used memory is 23G: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 17:29 |
| fungi | pleia2: the difference is between making it a required parameter and having a fallback (empty string in this case) default value when not explicitly defined during instantiation | 17:30 |
| pleia2 | fungi: thank you | 17:30 |
| jeblair | openstackgerrit did not report a change i just pushed | 17:31 |
| jeblair | remote: https://review.openstack.org/277981 Add job tags | 17:31 |
| fungi | jeblair: it also did not report a change i just pushed | 17:31 |
| jeblair | mtreinish, AJaeger: https://review.openstack.org/277981 adds the needed support to zuul for job tags | 17:31 |
| fungi | remote: https://review.openstack.org/277980 Move bindep fallback job-template to infra file | 17:31 |
| AJaeger | thanks, jeblair ! | 17:31 |
| fungi | i'm checking its log to see if it did not reconnect to the stream automatically | 17:32 |
| *** e0ne has quit IRC | 17:32 | |
| *** unicell has quit IRC | 17:32 | |
| *** dahoo has joined #openstack-infra | 17:33 | |
| *** sdake has quit IRC | 17:33 | |
| AJaeger | since ttx now duplicated this change, could some more infra-manual reviewers have a look at this change before it's a month old, please? https://review.openstack.org/#/c/266508/ | 17:33 |
| *** sdake has joined #openstack-infra | 17:33 | |
| *** sfinucan_ has joined #openstack-infra | 17:34 | |
| fungi | jeblair: it did log connection failures to gerrit while we were restarting, but started seeing events again as early as 17:24:43 | 17:34 |
| *** gokrokve_ has joined #openstack-infra | 17:34 | |
| anteaya | AJaeger: I already reviewed | 17:35 |
| fungi | 2016-02-09 17:30:25,630 INFO gerritbot: Sending "James E. Blair proposed openstack-infra/zuul: Add job tags https://review.openstack.org/277981" to #openstack-infra | 17:35 |
| fungi | so why didn't we see that? | 17:35 |
| fungi | no traceback | 17:35 |
| anteaya | :( | 17:37 |
| *** sfinucan has quit IRC | 17:37 | |
| *** placko has quit IRC | 17:37 | |
| *** gokrokve has quit IRC | 17:37 | |
| jeblair | maybe it's netsplit? | 17:37 |
| fungi | could just be bad timing, yeah | 17:37 |
| fungi | though it reported a change in here at 17:18 | 17:38 |
| fungi | so an eerie coincidence if so | 17:38 |
| fungi | it pongs to me on a ctcp ping through freenode too | 17:38 |
| fungi | so i don't _think_ it's netsplit | 17:38 |
| *** ajmiller has quit IRC | 17:39 | |
| *** jaosorior has joined #openstack-infra | 17:39 | |
| *** vincentfrancoise has quit IRC | 17:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add job tags https://review.openstack.org/277987 | 17:40 |
| jeblair | and it's back... | 17:40 |
| fungi | and now it's working again | 17:40 |
| fungi | how strange | 17:40 |
| jeblair | (that's a new push -- not delayed) | 17:40 |
| *** tphummel has joined #openstack-infra | 17:41 | |
| monester | Hi all, who can review my patch to python-gerritlib? | 17:41 |
| *** ajmiller has joined #openstack-infra | 17:42 | |
| *** kgiusti has joined #openstack-infra | 17:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename periodic networking jobs https://review.openstack.org/276367 | 17:43 |
| AJaeger | anteaya: I had to fix my patch due to duplicate definitions ^ | 17:44 |
| *** infra-red has joined #openstack-infra | 17:44 | |
| AJaeger | anteaya: could you review again, please? Now it passes even locally... | 17:44 |
| anteaya | monester: in gerrit when you are logged in, People > List Groups, search for gerritlib-core | 17:44 |
| anteaya | AJaeger: sure | 17:44 |
| monester | anteaya: thx | 17:44 |
| *** ybathia has quit IRC | 17:44 | |
| anteaya | welcome | 17:45 |
| *** EricGonc_ has quit IRC | 17:46 | |
| rbrndt | I'm having some difficulty with my devstack install. Our project requires eventlet, but the install can't find the version from the global constraints (0.18.1) on pipit. I'm not sure how to go about fixing this | 17:46 |
| rbrndt | *pypi | 17:46 |
| *** BobBall is now known as BobBall_AWOL | 17:46 | |
| anteaya | rbrndt: there is a post to -dev | 17:46 |
| *** fedexo has joined #openstack-infra | 17:46 | |
| anteaya | that version of eventlet disappeared from pypi | 17:46 |
| *** fedexo_ has joined #openstack-infra | 17:46 | |
| anteaya | the -dev mailing list that is | 17:47 |
| *** dizquierdo has quit IRC | 17:47 | |
| anteaya | fungi: on the infra cloud section of today's meeting, can we spend any time figuring out how to include folks who haven't been spending a lot of time on the project so they can review? | 17:48 |
| *** jtomasek has quit IRC | 17:48 | |
| anteaya | fungi: I'm trying to find a pathway into the reviews so I can get a picture in my mind and offer something useful and I'm having a hard time | 17:48 |
| fungi | anteaya: sure, if there are any good ideas for ramping up people, that's cool | 17:48 |
| *** sfinucan_ has quit IRC | 17:49 | |
| *** zz_ja is now known as zz_zz_ja | 17:49 | |
| anteaya | fungi: thanks well for me it would be, this is the first patch to review and the others build on it | 17:49 |
| rbrndt | antenna: thanks I'll go look at it | 17:49 |
| anteaya | I can review the add infra-cloud to nodepool patch but I do believe that is the last one not the first | 17:49 |
| anteaya | rbrndt: great | 17:49 |
| rbrndt | when did this start autocorrecting me...? | 17:49 |
| anteaya | rbrndt: not sure | 17:49 |
| anteaya | I have a new friday nick should I want to use it | 17:49 |
| anteaya | rbrndt: thank you | 17:50 |
| fungi | hah, that's a fun autocorrection | 17:50 |
| jeblair | anteaya: it's kind of awesome | 17:50 |
| *** achanda has joined #openstack-infra | 17:50 | |
| anteaya | ha ha ha | 17:50 |
| anteaya | it is indeed | 17:50 |
| anteaya | never thought of it before | 17:50 |
| *** eil397 has joined #openstack-infra | 17:50 | |
| fungi | it's the name of a track on kraftwerk's radio-activity | 17:51 |
| anteaya | very cool | 17:51 |
| fungi | 1975 | 17:51 |
| anteaya | ha ha ha | 17:51 |
| *** Swami has joined #openstack-infra | 17:51 | |
| anteaya | every youtube vid now begins with an ad, so tiresome | 17:53 |
| *** ildikov has quit IRC | 17:53 | |
| *** vgridnev has joined #openstack-infra | 17:53 | |
| fungi | what a great album that was | 17:53 |
| fungi | well, still is | 17:53 |
| anteaya | this track is very interesting | 17:53 |
| anteaya | so far I like it | 17:53 |
| *** jaosorior has quit IRC | 17:53 | |
| *** jordanP has joined #openstack-infra | 17:55 | |
| *** tphummel has quit IRC | 17:55 | |
| *** fedexo has quit IRC | 17:57 | |
| *** fedexo_ has quit IRC | 17:57 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Handle image upload job failures https://review.openstack.org/277941 | 17:57 |
| clarkb | greghaynes: jeblair ^ now with a test that should cover this case | 17:58 |
| greghaynes | clarkb: nice | 17:58 |
| clarkb | it is a bit convoluted but I want to make sure that the background machinery is processing the fails properly | 17:58 |
| *** derekh has quit IRC | 17:58 | |
| *** shardy has quit IRC | 18:00 | |
| *** akshai has quit IRC | 18:00 | |
| cody-somerville | neat. devstack runs twice as fast in lxc container. | 18:00 |
| *** infra-re_ has joined #openstack-infra | 18:00 | |
| SotK | fungi: excellent, thanks! :) | 18:00 |
| clarkb | cody-somerville: compared to what? | 18:00 |
| cody-somerville | clarkb: compared to a VM on my machine. | 18:00 |
| clarkb | cody-somerville: also to answer your question from previously about why we don't use containers is iscsi is still not namespaced | 18:01 |
| clarkb | cody-somerville: and testing cinder + nova is important to us. | 18:01 |
| mordred | fungi, jeblair: https://etherpad.openstack.org/p/5nUo6obvF7 | 18:01 |
| mordred | mirror announce draft | 18:01 |
| mordred | clarkb, pleia2: ^^ | 18:01 |
| jeblair | clarkb: cool. and greghaynes: i saw a traceback earlier that i need to debug. i plan to do that. don't let me forget. :) | 18:02 |
| *** angdraug has quit IRC | 18:02 | |
| *** bnemec has joined #openstack-infra | 18:02 | |
| mordred | cody-somerville: btw - we split the reprepro config to be ubuntu and debian separately - thanks for the feedback | 18:02 |
| greghaynes | jeblair: ok, thanks | 18:02 |
| cody-somerville | clarkb: yea, neither is loopback or multi-device but its okay if you're fine with no isolation. | 18:02 |
| clarkb | cody-somerville: but thats equivalent to what we already do so not sure I see the benefit | 18:02 |
| *** akshai has joined #openstack-infra | 18:02 | |
| *** rossella_s has quit IRC | 18:02 | |
| clarkb | we already run everything not isolated :) | 18:03 |
| *** jordanP has quit IRC | 18:03 | |
| cody-somerville | clarkb: it has benefit for local development. | 18:03 |
| *** abregman has joined #openstack-infra | 18:03 | |
| *** infra-red has quit IRC | 18:03 | |
| mordred | I think the main thing is that container in a vm isn't going to add much | 18:03 |
| mordred | for us | 18:03 |
| mordred | in the gate | 18:03 |
| *** rossella_s has joined #openstack-infra | 18:03 | |
| cody-somerville | aye. | 18:03 |
| cody-somerville | 100% agree. | 18:03 |
| clarkb | and a non isolated container on your laptop isn't really different than what devstack does today | 18:03 |
| mordred | clarkb: it gets you an isolated filesystem | 18:04 |
| mordred | clarkb: without a VM | 18:04 |
| clarkb | both can (and probably will) trash your system aat some point | 18:04 |
| clarkb | mordred: iirc you can't isolate fs with iscsi | 18:04 |
| clarkb | since it touches fs bits | 18:04 |
| clarkb | you can isolate processes? its a very small list of what is actually isolatable | 18:04 |
| cody-somerville | I think it boils down to there is no device namespace (I also heard something about iscsi specific bits in the kernel as well) | 18:05 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added *-npm-publish-draft job https://review.openstack.org/277996 | 18:05 |
| mordred | clarkb: while I agree with the correctness of your statements, I think the organzational isolation that it provides is likely good enough for 90% of local dev use | 18:05 |
| zaro | fungi, anteaya : fix to prevent gerrit from creating duplicate group name as gerrit system group merged upstream. https://gerrit-review.googlesource.com/#/c/74312 | 18:06 |
| cody-somerville | net + fs isolation is probably 80% of whats needed to help avoid messing up your main host. | 18:06 |
| mordred | you do get a chroot-like thing, and the devstack in the container does not screw with the networking stack on your laptop | 18:06 |
| clarkb | cody-somerville: but those are the things you need to not isolate to make it work :) | 18:06 |
| mordred | clarkb: you can TOTALLY isolate the fs | 18:06 |
| jeblair | mordred: overall looks great -- i corrected one typo and made 2 offers of suggestions -- take em or leave em :) | 18:06 |
| mgagne | zaro good one +1 | 18:06 |
| clarkb | mordred: with iscsi and cinder and nova? | 18:06 |
| mordred | clarkb: the container is like a chroot in that | 18:06 |
| *** woodster_ has quit IRC | 18:06 | |
| EmilienM | pabelanger: I'm trying to see the magic to copy logs when using subnodes | 18:06 |
| clarkb | I think you can have a chroot but not islated fs | 18:07 |
| mordred | clarkb: so, you get a chroot in which you can do things to the network which do not screw your wifi | 18:07 |
| mordred | that's good enough for most things | 18:07 |
| EmilienM | I've looked at devstack-gate, I haven't found yet the magic | 18:07 |
| *** eil397 has quit IRC | 18:07 | |
| clarkb | mordred: sort of | 18:07 |
| cody-somerville | Plus I find that kvm in container is much faster than nested kvm | 18:07 |
| *** electrofelix has quit IRC | 18:07 | |
| clarkb | mordred: network definitely can't be isolated with iscis | 18:07 |
| *** nadya has joined #openstack-infra | 18:07 | |
| clarkb | so you still have things touching the root namespace and potentially screwing your wifi | 18:07 |
| *** yaume has quit IRC | 18:07 | |
| zaro | fungi, anteaya : i tried to cherry-pick to our Gerrit fork but it was not trivial so it's probably not a good idea to add that. | 18:07 |
| cody-somerville | clarkb: I'll look further into the iscis stuff and get back to you but I was thinking for local dev I might just use the nfs driver | 18:07 |
| clarkb | for fs I think the problem there is while you can have an effective chroot nothing prevents the container from mounting the host / and having at it | 18:08 |
| *** nadya is now known as Guest9860 | 18:08 | |
| cody-somerville | clarkb: (which may still have namespace issues) | 18:08 |
| cody-somerville | clarkb: oh. my container can absolutely not do that. | 18:08 |
| clarkb | cody-somerville: the nfs driver is just for the storage side | 18:08 |
| clarkb | cody-somerville: iscsi is always required for the nova bits | 18:08 |
| cody-somerville | clarkb: that is why I didn't let it have access to lvm/dm | 18:08 |
| cody-somerville | clarkb: weird. It worked just fine for me. | 18:08 |
| clarkb | cody-somerville: did you use cinder? | 18:08 |
| *** eil397 has joined #openstack-infra | 18:08 | |
| clarkb | (and test that it worked?) | 18:08 |
| zaro | fungi, anteaya, mgagne : it's not really required to fix the duplicate 'Registered Users' group though. it can be fixed with the script i pushed up and the change from mgagne or myself. | 18:09 |
| *** shashank_hegde has joined #openstack-infra | 18:09 | |
| cody-somerville | clarkb: Yes, I used nfs but devstack didn't set it up properly. I didn't test any of the block storage stuff. | 18:09 |
| clarkb | basically cinder has two major components, the backend storage driver which can be lvm, nfs, etc. and the frontend that connects VMs to that storage via iscis | 18:09 |
| mgagne | EmilienM https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L582-L583 | 18:09 |
| clarkb | it is the frontend attach block to VM that is problematic with containers | 18:09 |
| mordred | yes. but there are tons of things you can usefully do with a devstack that do not include attaching vms to volumes over iscsi | 18:10 |
| clarkb | and for networking I am just going off of what sdake blogged about when containering neutron | 18:10 |
| cody-somerville | clarkb: Ok. I'll playa round with that. I just wanted to use NFS instead of LVM as to do lvm would require giving the container access to my volumes on my host. | 18:10 |
| clarkb | it is possible that nova net (devstack default) does work in its own namespace | 18:10 |
| mgagne | zaro yea, can't expect people to upgrade to latest gerrit | 18:10 |
| *** hdd has joined #openstack-infra | 18:10 | |
| mordred | well, you should never run devstack with nova-net | 18:10 |
| sdake | sup clarkb | 18:10 |
| EmilienM | mgagne: thanks | 18:10 |
| mgagne | EmilienM I learned something too today | 18:11 |
| clarkb | sdake: iirc you posted about needing to run neutron in the root net namespcae to get things working in containers? | 18:11 |
| *** gokrokve_ has quit IRC | 18:11 | |
| clarkb | sdake: was that and is it still the case? | 18:11 |
| mordred | since nova-net should never be used by anyone other than the people who are already stuck running it | 18:11 |
| sdake | clarkb yes | 18:11 |
| EmilienM | mgagne: trying to push puppet multinode jobs in our ci | 18:11 |
| anteaya | zaro: yay 74312 merged | 18:11 |
| mgagne | EmilienM just look for subnode jobs, see what scripts *could* copy the logs and go from there =) | 18:11 |
| *** calebb1 is now known as calebb | 18:11 | |
| mgagne | EmilienM nice =) | 18:11 |
| *** lucasagomes is now known as lucas-dinner | 18:11 | |
| clarkb | mordred: ^ so you cna't protect your wifies :) | 18:11 |
| pabelanger | EmilienM: what mgagne said. Just use ansible to sync them to the master node | 18:11 |
| sdake | clarkb it may be possible to run openstack without -net=host, but we didn't attempt it | 18:11 |
| mgagne | EmilienM we have 80 nodes setup already haha :P | 18:11 |
| *** flepied has quit IRC | 18:11 | |
| cody-somerville | clarkb: sdake: You have to expose /dev/net/tun but you can setup private isolated network for neutron. | 18:11 |
| sdake | clarkb i can't confirm openstack would *not* run without net=host, but it is a huge pain in the ass to make it all work correctly | 18:12 |
| mgagne | EmilienM we use bash to copy logs but I guess ansible is a great alternative | 18:12 |
| clarkb | sdake: gotcha | 18:12 |
| sdake | so we optimized with net=host | 18:12 |
| *** zeih_ has quit IRC | 18:12 | |
| clarkb | cody-somerville: with neutron? or nova net? | 18:12 |
| clarkb | or both? | 18:12 |
| sdake | also net-host removes docker-proxy | 18:12 |
| cody-somerville | neutron | 18:12 |
| sdake | ya neutron will definately need some host bindmounts with or without net=host | 18:12 |
| *** ybathia has joined #openstack-infra | 18:13 | |
| sdake | like i said, without net=hsot = possible, but not worth the pain and a super bandwidth killer | 18:13 |
| clarkb | sdake: gotcha, doable but not practical | 18:13 |
| clarkb | at least with docker? | 18:13 |
| *** fedexo has joined #openstack-infra | 18:14 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/integ/multinode: copy logs for localhost machine https://review.openstack.org/278008 | 18:14 |
| EmilienM | mgagne: first step ^ | 18:14 |
| *** fedexo_ has joined #openstack-infra | 18:14 | |
| *** annegentle has joined #openstack-infra | 18:14 | |
| EmilienM | pabelanger: ok thanks :-) | 18:14 |
| openstackgerrit | Khai Do proposed openstack-infra/gerritlib: Initialize the logger https://review.openstack.org/277654 | 18:14 |
| anteaya | mordred: do be clear I personally would not be dead without a per region pypi mirror | 18:15 |
| anteaya | to be clear | 18:15 |
| mordred | anteaya: that's fair | 18:15 |
| mordred | anteaya: how's that? | 18:15 |
| * anteaya looks again | 18:16 | |
| openstackgerrit | Merged openstack-infra/zuul: Fix default merge failure reports https://review.openstack.org/277958 | 18:16 |
| anteaya | that is much more accurate | 18:16 |
| jeblair | mordred: can we drop 'many' from 'many tens of minutes' ? | 18:16 |
| jeblair | mordred: cause it's like two tens of minutes. :) | 18:16 |
| anteaya | as indeed the gate would be in dire straits without the mirrors | 18:16 |
| *** slogan621 has joined #openstack-infra | 18:16 | |
| anteaya | two three | 18:16 |
| jeblair | mordred: or maybe 1.8 tens of minutes. | 18:16 |
| *** e0ne has joined #openstack-infra | 18:17 | |
| jeblair | but certainly not much more than 2 tens of minutes at the most | 18:17 |
| mordred | heh | 18:17 |
| mordred | I want to say 1.8 tens of minutes | 18:17 |
| cody-somerville | speaking of wheels, would it make sense to add docs to devstack to tell people how to use our wheels mirrors? compiling numpy takes awhile no matter what you're on. | 18:17 |
| anteaya | ha ha ha | 18:18 |
| *** rhallisey has quit IRC | 18:18 | |
| mordred | cody-somerville: well, they're not (at the moment) intended for non-gate humans to use | 18:18 |
| anteaya | mordred: looks good to me | 18:19 |
| *** rhallisey has joined #openstack-infra | 18:19 | |
| *** annegentle has quit IRC | 18:19 | |
| anteaya | zaro: alright what is the status of the script you submitted to gerrit to fix the duplicate 'Registered Users' group? | 18:19 |
| mordred | cody-somerville: we could investigate and discuss providing endpoints for non-gate humans | 18:19 |
| clarkb | cody-somerville: mordred in the past we hvae explicitly told people not to use them | 18:20 |
| mordred | yah | 18:20 |
| clarkb | cody-somerville: mordred part of that is selfish | 18:20 |
| clarkb | eg we just turned off the old ones and didn't have to notify anyone | 18:20 |
| mordred | I think now that we have afs we _could_ spin up a non-region-specific web gateway that's just an AFS client pretty easily - but like clarkb says, that increases the surface area we have to care about if we make structural changes | 18:21 |
| *** infra-red has joined #openstack-infra | 18:21 | |
| jeblair | i guess if we made it an actual devstack option, that might alleviate it a bit... | 18:21 |
| *** jsavak has quit IRC | 18:21 | |
| zaro | fungi, anteaya: https://review.openstack.org/#/c/270537/ and (https://review.openstack.org/#/c/270628/ or https://review.openstack.org/#/c/248126/) | 18:22 |
| slogan621 | What remains to be done for https://review.openstack.org/#/c/275873/ ? We have 3 +1, jenkins +1 as well. Is it now just basically wait for a infra project owner to approve? | 18:22 |
| fungi | mordred: "1.8 tens of minutes" is fun wording | 18:22 |
| *** infra-red has quit IRC | 18:22 | |
| *** piet has joined #openstack-infra | 18:23 | |
| *** piet has quit IRC | 18:23 | |
| jeblair | it will probably tell us if anyone reads it :) | 18:23 |
| *** infra-red has joined #openstack-infra | 18:23 | |
| *** annegentle has joined #openstack-infra | 18:23 | |
| *** infra-re_ has quit IRC | 18:23 | |
| anteaya | mordred: this is zaro's patch with alters the gerrit db can you take a look? or add it to your pile of patches to take a look at, please? https://review.openstack.org/#/c/270537/5 | 18:23 |
| anteaya | s/with/which | 18:24 |
| fungi | mordred: announcement lgtm | 18:24 |
| *** fedexo_ has quit IRC | 18:24 | |
| *** fedexo has quit IRC | 18:24 | |
| mordred | should we wait until we've cut new images with the apt-mirror consumption patch applied? | 18:24 |
| *** EricGonczer_ has joined #openstack-infra | 18:25 | |
| AJaeger | mordred: well written | 18:25 |
| anteaya | zaro mgagne can the two of you decide between you which way you want to go either 270628 or 248126 if they are mutually exclusive? | 18:25 |
| *** daemontool has joined #openstack-infra | 18:25 | |
| mordred | jeblair: yah - if we did that and made a mirror.openstack.org it might not be too tarrible | 18:25 |
| anteaya | zaro mgagne it will save time trying to get reviewers attention to get the solution in place | 18:26 |
| *** infra-red has quit IRC | 18:28 | |
| fungi | mordred: sure, might want to hold off until we're actually using the apt mirror in most places (for ubuntu anyway) so that your announcement doesn't serve as a precursor to some implementation disaster we've overlooked ;) | 18:29 |
| mordred | fungi: +100 | 18:30 |
| fungi | tuesday: here's this great new thing we're using, wednesday: oh the humanity! | 18:30 |
| *** esker has joined #openstack-infra | 18:30 | |
| fungi | easier to explain if things happen the other way 'round | 18:31 |
| *** ajmiller has quit IRC | 18:33 | |
| *** sabeen3 has quit IRC | 18:33 | |
| *** infra-red has joined #openstack-infra | 18:34 | |
| *** jgriffith_away is now known as jgriffith | 18:34 | |
| *** sridhar_ram has quit IRC | 18:35 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Install docker for tests https://review.openstack.org/278021 | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| *** fedexo has joined #openstack-infra | 18:36 | |
| *** fedexo_ has joined #openstack-infra | 18:36 | |
| *** ildikov has joined #openstack-infra | 18:36 | |
| *** ajmiller has joined #openstack-infra | 18:36 | |
| *** akshai has quit IRC | 18:36 | |
| jeblair | fungi: i made some last-minute agenda mods | 18:37 |
| fungi | jeblair: thanks for the heads up | 18:37 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 18:38 |
| *** akshai has joined #openstack-infra | 18:38 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 18:38 |
| jeblair | that change number starts with a 1. | 18:38 |
| *** kushal has joined #openstack-infra | 18:38 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/ciwatch: Add link to source for project under Contact Us https://review.openstack.org/278022 | 18:38 |
| fungi | at least it's not <6 digits! | 18:38 |
| *** ociuhandu has quit IRC | 18:39 | |
| *** baoli_ has quit IRC | 18:40 | |
| *** annegentle has quit IRC | 18:40 | |
| *** annegentle has joined #openstack-infra | 18:41 | |
| *** eil397 has quit IRC | 18:41 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont pass xattrs to tar if its unsupported https://review.openstack.org/177559 | 18:42 |
| *** infra-re_ has joined #openstack-infra | 18:42 | |
| clarkb | FYI osic will be doing reboots for firmware patching today. Expect funnyness if testing the new accounts | 18:43 |
| *** infra-red has quit IRC | 18:44 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add tests for building *-minimal images https://review.openstack.org/181162 | 18:44 |
| *** rhallisey has quit IRC | 18:45 | |
| jeblair | clarkb: any word on the ips? | 18:45 |
| SergeyLukjanov | fungi, hi, I'll be unable to attend meeting - have a conflicting one, but I'm ready to do renaming on weekdays pacific time | 18:45 |
| clarkb | jeblair: they have said they are itentioanlly being stingy with them, I have reiterated our requirement for FIP per host or ipv6 and said we can use as many as they can give us | 18:46 |
| fungi | SergeyLukjanov: okay, thanks. i'll bring it up (it's still on our agenda for today) | 18:46 |
| openstackgerrit | Fabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements https://review.openstack.org/251674 | 18:46 |
| *** matrohon has joined #openstack-infra | 18:47 | |
| *** rhallisey has joined #openstack-infra | 18:48 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont pass xattrs to tar if its unsupported https://review.openstack.org/177559 | 18:49 |
| *** mriedem has quit IRC | 18:49 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Unified Mirrors https://review.openstack.org/252678 | 18:50 |
| *** mriedem has joined #openstack-infra | 18:51 | |
| *** tphummel has joined #openstack-infra | 18:51 | |
| *** [1]Thelo has joined #openstack-infra | 18:52 | |
| *** angdraug has joined #openstack-infra | 18:53 | |
| *** Thelo has quit IRC | 18:54 | |
| *** [1]Thelo is now known as Thelo | 18:54 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d https://review.openstack.org/259734 | 18:55 |
| *** flepied has joined #openstack-infra | 18:55 | |
| *** annegentle has quit IRC | 18:56 | |
| openstackgerrit | Khai Do proposed openstack-infra/jeepyb: Don't create new groups for system ones https://review.openstack.org/248126 | 18:56 |
| zaro | anteaya, mgagne ^ consoliated our changes | 18:56 |
| *** sridhar_ram has joined #openstack-infra | 18:56 | |
| *** krtaylor has quit IRC | 18:57 | |
| *** thorst_ is now known as thorst | 18:57 | |
| anteaya | zaro: thank you, do you want to abandon https://review.openstack.org/#/c/270628/ then? | 18:57 |
| *** matrohon has quit IRC | 18:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rework devstack-jobs templating https://review.openstack.org/278034 | 18:57 |
| anteaya | thanks | 18:57 |
| *** Guest8858 has joined #openstack-infra | 18:57 | |
| zaro | anteaya: already did | 18:59 |
| AJaeger | team, the change above (278034) reworks devstack-jobs, we now generate 360 jobs less, since we only have a few periodic etc. jobs. What do you think of this rework? It passes the tests locally... | 18:59 |
| anteaya | zaro: sorry I must have had a lag, thank you for consolidating | 18:59 |
| pabelanger | AJaeger: looks like voting is open for Austin talks. Going to mention that in the meeting | 18:59 |
| anteaya | AJaeger: will take a look after the stretch of meetings | 18:59 |
| *** flepied has quit IRC | 18:59 | |
| jhesketh | Morning | 18:59 |
| anteaya | AJaeger: thank you | 18:59 |
| anteaya | jhesketh: morning | 18:59 |
| *** alivigni has quit IRC | 19:00 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| AJaeger | thanks, anteaya. | 19:00 |
| anteaya | welcome :) | 19:00 |
| AJaeger | pabelanger: Let's collect some votes then ;) | 19:00 |
| *** Guest8858 is now known as annegentle_ | 19:00 | |
| *** akshai has quit IRC | 19:01 | |
| *** jsavak has joined #openstack-infra | 19:01 | |
| *** tiswanso has quit IRC | 19:01 | |
| *** akshai has joined #openstack-infra | 19:02 | |
| *** tiswanso has joined #openstack-infra | 19:02 | |
| anteaya | oh please don't play vote for me | 19:02 |
| *** alivigni has joined #openstack-infra | 19:02 | |
| anteaya | playing voting is open is fine, but playing vote for me is just has a domino effect | 19:03 |
| *** ociuhandu has joined #openstack-infra | 19:03 | |
| openstackgerrit | Carl Baldwin proposed openstack-infra/irc-meetings: Update chairs for Neutron's L3 sub-team meeting https://review.openstack.org/278038 | 19:05 |
| *** tdurakov_ has joined #openstack-infra | 19:05 | |
| *** fawadkhaliq has quit IRC | 19:06 | |
| *** eil397 has joined #openstack-infra | 19:06 | |
| *** jsavak has quit IRC | 19:08 | |
| *** krtaylor has joined #openstack-infra | 19:08 | |
| mtreinish | I just got a couple of unstable results from: https://review.openstack.org/#/c/267824/ | 19:08 |
| *** jsavak has joined #openstack-infra | 19:08 | |
| *** dtantsur is now known as dtantsur|afk | 19:09 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/irc-meetings: nova bugs team meeting https://review.openstack.org/278043 | 19:09 |
| *** daemontool_ has joined #openstack-infra | 19:10 | |
| *** daemontool__ has joined #openstack-infra | 19:11 | |
| *** sabeen1 has joined #openstack-infra | 19:12 | |
| *** daemontool has quit IRC | 19:12 | |
| *** daemontool_ has quit IRC | 19:14 | |
| *** alex_xu has quit IRC | 19:15 | |
| *** esker has quit IRC | 19:16 | |
| *** esker has joined #openstack-infra | 19:17 | |
| *** dims_ has joined #openstack-infra | 19:17 | |
| *** dims has quit IRC | 19:17 | |
| *** alex_xu has joined #openstack-infra | 19:18 | |
| *** esker has quit IRC | 19:19 | |
| *** sabeen3 has joined #openstack-infra | 19:21 | |
| *** sabeen1 has quit IRC | 19:21 | |
| anteaya | nibalizer: for context it is simple things like complete commit messages, for instance: https://review.openstack.org/#/c/262835/1 | 19:22 |
| anteaya | nibalizer: I have zero idea why this patch was abandoned and what was gained by spliting it into two | 19:22 |
| anteaya | and the two new patches don't tell me in the commit message either | 19:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove non-existing jobs from devstack-jobs https://review.openstack.org/278051 | 19:25 |
| *** dims_ has quit IRC | 19:27 | |
| dmsimard | Could anyone point me at where the magic happens to have the jenkins console output be in html with "linkable" timestamps ? | 19:27 |
| anteaya | I think you want an oslo library | 19:27 |
| *** Sukhdev has joined #openstack-infra | 19:27 | |
| AJaeger | anteaya: FYI 278051 should be a trivial cleanup | 19:28 |
| anteaya | take a look in nova, I believe they use an oslo logging library | 19:28 |
| anteaya | AJaeger: okay thanks | 19:28 |
| *** ybathia has quit IRC | 19:28 | |
| dmsimard | anteaya: huh. I'm not following how an oslo library can help with setting up the console.html page, I would assume that to be done through jenkins ? | 19:29 |
| dmsimard | This is what I mean, clicky timestamps: http://logs.openstack.org/32/276932/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/17624f4/console.html#_2016-02-08_19_26_08_083 | 19:29 |
| anteaya | it isn't it is done via a library | 19:29 |
| anteaya | I get what you mean, clicky timestamps | 19:30 |
| *** dims has joined #openstack-infra | 19:30 | |
| dmsimard | interesting | 19:30 |
| *** eil397 has quit IRC | 19:30 | |
| *** kzaitsev_mb has quit IRC | 19:31 | |
| *** tzn has quit IRC | 19:31 | |
| *** eil397 has joined #openstack-infra | 19:31 | |
| *** jtomasek_ has joined #openstack-infra | 19:31 | |
| *** ybathia has joined #openstack-infra | 19:32 | |
| *** eil397 has quit IRC | 19:33 | |
| *** esker has joined #openstack-infra | 19:34 | |
| *** esker has quit IRC | 19:34 | |
| *** mingyu_ has quit IRC | 19:34 | |
| *** esker has joined #openstack-infra | 19:34 | |
| *** mingyu has joined #openstack-infra | 19:35 | |
| *** eil397 has joined #openstack-infra | 19:36 | |
| *** ayoung has joined #openstack-infra | 19:37 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Support stable requirements https://review.openstack.org/278056 | 19:37 |
| slogan621 | anteaya: What happens next for https://review.openstack.org/#/c/275873/ ? Is it just in need of project approval? | 19:38 |
| *** kzaitsev_mb has joined #openstack-infra | 19:38 | |
| slogan621 | Ajaeger: Same question, what is the next step for a project that is in this state: https://review.openstack.org/#/c/275873/ | 19:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Added bower install to post-install hook https://review.openstack.org/278057 | 19:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Corrected license https://review.openstack.org/278058 | 19:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Corrected repository field. https://review.openstack.org/278059 | 19:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Corrected package author https://review.openstack.org/278060 | 19:39 |
| anteaya | slogan621: well you commit message could be formatted consistently with other commit messages | 19:39 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove unused templates from devstack-gate.yaml https://review.openstack.org/278061 | 19:40 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add Create Story Button in Worklists https://review.openstack.org/269656 | 19:40 |
| *** sabeen3 has left #openstack-infra | 19:40 | |
| AJaeger | and one more cleanup - templates that are unused since over a year (278061) | 19:41 |
| anteaya | AJaeger: ack, thank you | 19:41 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Realize infracloud admins https://review.openstack.org/268366 | 19:41 |
| *** dims has quit IRC | 19:41 | |
| AJaeger | thanks, anteaya | 19:41 |
| *** dims has joined #openstack-infra | 19:42 | |
| slogan621 | anteaya: we'll need to pass that along to the developer to change his commit message then, could you maybe comment on the bug and cite an example he can follow (or give me an example or link to an example and I will post the comment)? | 19:42 |
| slogan621 | s/bug/review/ | 19:42 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Realize infracloud admins https://review.openstack.org/268366 | 19:42 |
| anteaya | slogan621: after the infra and tc meeting | 19:42 |
| slogan621 | ok :-) sorry to interrupt | 19:43 |
| nibalizer | anteaya: so 268366 is the only patch any moree | 19:44 |
| nibalizer | what doesn't make sense there? | 19:44 |
| anteaya | This is a re-up of I31e0504f68e196b0b249b5b7b750861a752679ac | 19:44 |
| mtreinish | mordred: so I just read: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086176.html I wanted to check if there was anything special I needed to do for: https://pypi.python.org/pypi/Tempest/ | 19:44 |
| anteaya | what am I supposed to understand from that line? | 19:45 |
| mtreinish | I haven't done anything with it except add openstackci as another owner | 19:45 |
| anteaya | why was the other patch not worthy of moving forward? | 19:45 |
| *** tdurakov_ has quit IRC | 19:45 | |
| mtreinish | but when I'm ready to push a release any pre-steps I need to do | 19:45 |
| *** sridhar_ram is now known as sridhar_ram_afk | 19:45 | |
| nibalizer | anteaya: that the first patch attempting to do this work had that change-id, but that was abandoned and now this patch lives on | 19:45 |
| *** mriedem has quit IRC | 19:45 | |
| mordred | mtreinish: that's a flexible heatmap generator | 19:45 |
| nibalizer | anteaya: it was proposed to the wrong repo | 19:45 |
| mordred | mtreinish: but no - there is nothing special | 19:46 |
| *** dims_ has joined #openstack-infra | 19:46 | |
| mtreinish | mordred: it was, but it's the same name as tempest. I got the previous owner to give up the slot | 19:46 |
| mtreinish | so we can push new tempest releases to pypi | 19:46 |
| mtreinish | it's part of my evil plan: https://review.openstack.org/275966 | 19:46 |
| mordred | mtreinish: neat! | 19:47 |
| AJaeger | mtreinish: the infra manual should explain everything | 19:47 |
| *** mriedem has joined #openstack-infra | 19:47 | |
| AJaeger | mtreinish: http://docs.openstack.org/infra/manual/drivers.html | 19:47 |
| mtreinish | AJaeger: it explains how to take over an existing project on pypi which was different code with a different release versioning? | 19:47 |
| *** fedexo_ has quit IRC | 19:48 | |
| *** fedexo has quit IRC | 19:48 | |
| AJaeger | mtreinish: different versioning? Ooh, better ask stuffit (?) | 19:48 |
| anteaya | nibalizer: okay great, thank you, proposed to the wrong repo is useful for me | 19:48 |
| sdague | in my ideal world there would be a "hold please" button on projects somewhere so we could temporarily hold +A from moving to the gate when there was a systemic fail | 19:48 |
| *** dims has quit IRC | 19:49 | |
| daemontool__ | AJaeger, using python-jobs, do you know how can I enable the gate-{name}-docs gate job only for master and leave the other gate job run for all the branches? | 19:49 |
| AJaeger | sdague: +1 | 19:49 |
| mtreinish | AJaeger: yeah, the previous Tempest heat map project used the date as the release version | 19:49 |
| *** e0ne has quit IRC | 19:49 | |
| AJaeger | daemontool__: http://jaegerandi.blogspot.de/2016/02/templates-in-openstacks-zuul.html | 19:49 |
| sdague | fyi, all nova patches are fail, because eventlet 0.18.2 in unit tests | 19:49 |
| mtreinish | while openstack tempest uses an int (which pbr covnerts to a normal semver triplet) | 19:49 |
| AJaeger | daemontool__: does that help? | 19:50 |
| AJaeger | daemontool__: ah, that does not explain it - but just add a condition for -docs similar as written | 19:51 |
| daemontool__ | AJaeger, fan-tas-tic ty | 19:53 |
| *** sabeen1 has joined #openstack-infra | 19:53 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 19:55 |
| *** achanda has quit IRC | 19:55 | |
| *** flepied has joined #openstack-infra | 19:55 | |
| *** esker has quit IRC | 19:56 | |
| anteaya | sdague: I read your comment in -nova and now your comment in here makes sense | 19:57 |
| *** sridhar_ram_afk is now known as sridhar_ram | 19:57 | |
| *** dims_ has quit IRC | 19:57 | |
| anteaya | you would like a global button so that no new patches are added to the gate when something like a constrained dependency disappears from pypi | 19:57 |
| daemontool__ | AJaeger, but would that limit a specific job part of python-jobs, i.e. limit only gate{name}-docs called from python-jobs? | 19:59 |
| *** dprince has quit IRC | 19:59 | |
| AJaeger | daemontool__: the conditions only work on jobs, not on templates | 20:00 |
| *** ccrouch has quit IRC | 20:00 | |
| *** ccrouch1 has joined #openstack-infra | 20:00 | |
| *** dprince has joined #openstack-infra | 20:00 | |
| *** rockyg has joined #openstack-infra | 20:01 | |
| mordred | infra-root: anybody have an issue with me cutting a new release of glean? | 20:01 |
| fungi | mordred: no objection from me | 20:01 |
| yolanda | fine for me | 20:02 |
| mordred | looking at the outstanding changes I'm going to call it 1.5.0 | 20:02 |
| *** akshai has quit IRC | 20:02 | |
| *** ifarkas has quit IRC | 20:02 | |
| daemontool__ | does a job setting override the template setting? Like if I use python-jobs template and gate-{name}-docs as job, will the settings of the job override the job called from the template? | 20:02 |
| nibalizer | mordred: no object | 20:02 |
| *** austin81 has joined #openstack-infra | 20:03 | |
| *** annegentle_ has quit IRC | 20:03 | |
| nibalizer | infra-root: can has reviews on this https://review.openstack.org/#/c/234535/11 | 20:03 |
| *** EricGonczer_ has quit IRC | 20:03 | |
| *** annegentle_ has joined #openstack-infra | 20:03 | |
| daemontool__ | AJaeger, the problem I have now, is that I want all the jobs in the template python-jobs execute on all the branchs, but the job gate-{name}-docs called from the template run against master only | 20:04 |
| clarkb | daemontool__: because of the way that the yaml parser works one of them will win and its parse order dependent iirc | 20:04 |
| clarkb | daemontool__: you should avoid doing that to avoid future confusion (if parse order changes for example) | 20:05 |
| AJaeger | daemontool__: conditions *only* match against jobs, never against templates | 20:05 |
| *** jgriffith has quit IRC | 20:05 | |
| daemontool__ | ok | 20:05 |
| *** EricGonczer_ has joined #openstack-infra | 20:05 | |
| AJaeger | daemontool__: jsut add a condition for gate-your-repo-docs | 20:05 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Handle image upload job failures https://review.openstack.org/277941 | 20:05 |
| daemontool__ | clarkb, AJaeger ty | 20:05 |
| *** akshai has joined #openstack-infra | 20:05 | |
| *** eil397 has quit IRC | 20:06 | |
| *** jgriffith has joined #openstack-infra | 20:06 | |
| *** jgriffith is now known as Guest55421 | 20:06 | |
| lifeless | sigmavirus24: sure | 20:06 |
| *** annegentle_ has quit IRC | 20:06 | |
| *** annegentle_ has joined #openstack-infra | 20:07 | |
| clarkb | sdague: anteaya you could remove all the members of the core group | 20:07 |
| clarkb | then add them back again when ready | 20:07 |
| clarkb | well leave at least one person to repopulate | 20:07 |
| anteaya | clarkb: good point | 20:07 |
| anteaya | that would prevent any +A'ing from the group | 20:07 |
| sdague | clarkb: I suppose, I mostly want hold so that patches in the gate would all get pulled out into a holding bin as well | 20:07 |
| anteaya | and is fast and reversable | 20:08 |
| *** rcernin has joined #openstack-infra | 20:08 | |
| sdague | instead of having to go and pluck | 20:08 |
| sdague | and unwind that | 20:08 |
| sdague | anyway, like I said, in an ideal world | 20:08 |
| *** rfolco has quit IRC | 20:08 | |
| *** jsavak has quit IRC | 20:09 | |
| *** piet has joined #openstack-infra | 20:09 | |
| *** kzaitsev_mb has quit IRC | 20:10 | |
| lifeless | dstufft: thoughts on https://bugs.launchpad.net/pbr/+bug/1542383 appreciated | 20:10 |
| openstack | Launchpad bug 1542383 in PBR "wsgi_scripts and python wheels" [Undecided,New] | 20:10 |
| *** jsavak has joined #openstack-infra | 20:12 | |
| *** piet has quit IRC | 20:13 | |
| *** sabeen3 has joined #openstack-infra | 20:13 | |
| *** piet has joined #openstack-infra | 20:13 | |
| openstackgerrit | Merged openstack-infra/system-config: Add playbook to run JJB on all jenkins masters https://review.openstack.org/277539 | 20:13 |
| *** tiswanso has quit IRC | 20:14 | |
| *** tiswanso has joined #openstack-infra | 20:15 | |
| SergeyLukjanov | anteaya yeah, I'm in Mirantis office in Sunnyvale ;) | 20:15 |
| SergeyLukjanov | pleia2 I'm ok with 22UTC | 20:15 |
| pleia2 | SergeyLukjanov: great, thank you :) | 20:16 |
| pleia2 | SergeyLukjanov: also, welcome to the valley! | 20:16 |
| pleia2 | (I'm up in SF) | 20:16 |
| SergeyLukjanov | pleia2 thx, cool :) | 20:16 |
| anteaya | SergeyLukjanov: you had said sunnyvale before and I guessed thatwas were the office was, cool | 20:16 |
| SergeyLukjanov | I'll be here for a few month | 20:16 |
| anteaya | wonderful | 20:17 |
| *** sabeen1 has quit IRC | 20:17 | |
| SergeyLukjanov | so, Friday 22UTC | 20:17 |
| SergeyLukjanov | just to confirm - Feb 12? :) | 20:17 |
| anteaya | yes this friday | 20:18 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova wsgi unit test eventlet 0.18.2 bug 1543766 https://review.openstack.org/278069 | 20:18 |
| openstack | bug 1543766 in OpenStack Compute (nova) "nova.tests.unit.test_wsgi.TestWSGIServerWithSSL fails with testtools.matchers._impl.MismatchError: 'OK\r\n' != 'PONG' and eventlet 0.18.2" [High,Confirmed] https://launchpad.net/bugs/1543766 | 20:18 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Set conditions for gate-{name}-docs on freezer* https://review.openstack.org/278070 | 20:18 |
| daemontool__ | AJaeger, something like that would work? https://review.openstack.org/#/c/278070/1/zuul/layout.yaml | 20:19 |
| *** krtaylor has quit IRC | 20:19 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 20:19 |
| AJaeger | daemontool__: yeah, should - +2 | 20:21 |
| *** Guest55421 is now known as jgriffith | 20:24 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet-ceph: do not run integ jobs for scenario002 https://review.openstack.org/278072 | 20:25 |
| *** EricGonczer_ has quit IRC | 20:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove {name} jobs from devstack-jobs https://review.openstack.org/278073 | 20:27 |
| *** Sukhdev has quit IRC | 20:27 | |
| AJaeger | and one more cleanup - a few broken entries | 20:28 |
| slogan621 | anteaya: something like this? https://github.com/openstack-infra/project-config/commit/0fad865d442b4fb864d009b14b5f16e0d7ec1aa7 | 20:29 |
| *** Sukhdev has joined #openstack-infra | 20:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tempest/experimental: run puppet scenarios on centos7 https://review.openstack.org/278080 | 20:31 |
| pleia2 | oh hey, what time are we starting on sprint days? | 20:31 |
| pleia2 | 9? 10? | 20:31 |
| *** tzn has joined #openstack-infra | 20:31 | |
| *** krtaylor has joined #openstack-infra | 20:31 | |
| anteaya | pleia2: I'm usually there by 9am | 20:32 |
| anteaya | we'll probably get rolling when folks show up | 20:32 |
| *** ybathia has quit IRC | 20:32 | |
| anteaya | we tend to be morning get things going type of folks | 20:32 |
| pleia2 | anteaya: ok, going to see if it's easy for them to give us coffee+little breakfast snacks, and knowing *when* would help | 20:32 |
| daemontool__ | AJaeger, brilliant thanks a lot | 20:32 |
| sigmavirus24 | lifeless: I'm happy to chat about that too if mattt isn't still around | 20:32 |
| pleia2 | in darmstadt we started at 10, but maybe that was just a europe thing :) | 20:32 |
| *** tzn has quit IRC | 20:32 | |
| *** EricGonczer_ has joined #openstack-infra | 20:33 | |
| *** tcammann has joined #openstack-infra | 20:33 | |
| anteaya | slogan621: well the commit message on http://git.openstack.org/cgit/openstack-infra/project-config/commit/?id=0fad865d442b4fb864d009b14b5f16e0d7ec1aa7 is a bit thin but at least the sentence length is more along the lines of what is a resonable format expectation | 20:33 |
| openstackgerrit | Merged openstack/requirements: bump eventlet version from 0.18.1 to 0.18.2 https://review.openstack.org/277912 | 20:33 |
| slogan621 | anteaya: So, replace "Adding" with "Add", remove "and jobs". And the rest is ok? Nothing depends on the change, so don't see a need for "Needed-By:" | 20:33 |
| *** tzn has joined #openstack-infra | 20:33 | |
| anteaya | pleia2: did we do 10am in darmstadt | 20:33 |
| anteaya | slogan621: can we continue this after the tc meeting is finished? | 20:33 |
| slogan621 | oh, sorry, should I monitor in #openstack-meetings? | 20:33 |
| *** NobodyCa1 has joined #openstack-infra | 20:34 | |
| lifeless | sigmavirus24: sure; see my comment on the bug | 20:34 |
| * sigmavirus24 looks | 20:34 | |
| *** esp has joined #openstack-infra | 20:35 | |
| anteaya | slogan621: or wait until the top of the hour, whatever works for you | 20:35 |
| daemontool__ | clarkb, or anyone if you don't mind +A on https://review.openstack.org/#/c/278070/1 | 20:35 |
| *** |-paul-| has joined #openstack-infra | 20:35 | |
| dstufft | lifeless: what is a "wsgi_script", is this like a console_script ? | 20:36 |
| * slogan621 heads out for a taco and pepsi | 20:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tempest/experimental: run puppet scenarios on centos7 https://review.openstack.org/278080 | 20:36 |
| dstufft | the only special entry points are console_script and gui_script afaik | 20:36 |
| *** salv-orl_ has quit IRC | 20:36 | |
| AJaeger | dstufft: mtreinish had some questions about releasing tempest on pypi - could you read the scrollback from an hour ago, please? | 20:37 |
| pleia2 | tacos++ | 20:37 |
| *** SumitNaiksatam has joined #openstack-infra | 20:38 | |
| *** esp has quit IRC | 20:39 | |
| *** tcammann has quit IRC | 20:39 | |
| clarkb | nibalizer: crinkle +2 to managing the controller and computes with ansible puppet | 20:39 |
| nibalizer | ty | 20:39 |
| lifeless | dstufft: it is; we have a custom writer hooked in from pbr for it | 20:40 |
| lifeless | dstufft: sets up mod_wsgi glue automatically | 20:40 |
| clarkb | daemontool__: do the docs not exist back on those branches? | 20:40 |
| *** NobodyCa1 has quit IRC | 20:41 | |
| daemontool__ | nope | 20:41 |
| daemontool__ | unless I backport | 20:41 |
| daemontool__ | that changes from master | 20:41 |
| clarkb | (I approved the change was just looking for more of the why) | 20:41 |
| gtmanfred | SpamapS: mordred can i get one of yall to review this? thanks! https://review.openstack.org/#/c/268404/ | 20:41 |
| daemontool__ | sure | 20:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Sort devstack-jobs list https://review.openstack.org/278084 | 20:42 |
| openstackgerrit | Merged openstack-infra/ciwatch: Add link to source for project under Contact Us https://review.openstack.org/278022 | 20:42 |
| *** calebb has quit IRC | 20:43 | |
| *** maishsk has joined #openstack-infra | 20:43 | |
| dstufft | AJaeger: I'm not sure the question. If the version numbers aren't equal then just delete everything and start uploading. You won't be able to re-use version numbers but once the old versions are deleted you won't have to worry about ordering issues | 20:44 |
| dstufft | I'm not sure I understand the question* | 20:44 |
| clarkb | please don't delete... | 20:44 |
| clarkb | eventlet just broke a bunch of people because they deleted | 20:44 |
| dstufft | eh | 20:44 |
| *** weshay_xchat has quit IRC | 20:45 | |
| dstufft | This is "some unrelated thing on PyPI was called tempest, and now some new thing on PyPI is going to be temptest" | 20:45 |
| clarkb | hrm | 20:45 |
| *** Sukhdev has quit IRC | 20:45 | |
| clarkb | maybe its better to avoid the collision? | 20:45 |
| AJaeger | mtreinish: ^ | 20:45 |
| dstufft | Dunnno, your other option is to use epoch versions | 20:45 |
| sigmavirus24 | lifeless: so launchpad is timing out while I try to finish my thoughts on that | 20:45 |
| dstufft | that doesn't work in older setuptools/pip though at all | 20:46 |
| sigmavirus24 | dstufft: epoch versions are too ugly for openstack. we've had this discussion already | 20:46 |
| dstufft | *shrug* | 20:46 |
| sigmavirus24 | Also it's too much of a hassle for downstream packagers to represent our epochs along side their epochs or something | 20:46 |
| AJaeger | dstufft: understood, thanks - hope it helps mtreinish | 20:46 |
| * sigmavirus24 shrugs | 20:46 | |
| *** piet has quit IRC | 20:46 | |
| dstufft | those are your options :] | 20:46 |
| mtreinish | dstufft: so I was planning to just push a new version of tempest to pypi. But I guess it's naive of me to expect things to just work :) | 20:46 |
| *** lxsli has joined #openstack-infra | 20:47 | |
| *** esp has joined #openstack-infra | 20:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:47 | |
| dstufft | mtreinish: depends on ordering | 20:47 |
| dstufft | does your temptest version scheme overlap with the old temptest version scheme? | 20:47 |
| dstufft | and if not, does your temptest version scheme sort as older than the old tempest version scheme | 20:48 |
| *** tcammann_ has joined #openstack-infra | 20:48 | |
| *** NobodyCa1 has joined #openstack-infra | 20:48 | |
| *** tcammann has joined #openstack-infra | 20:48 | |
| mtreinish | well the current version of Tempest on pypi is 2010.09.26 and the next tempest version will be 10.0.0 | 20:48 |
| dstufft | So the answer to the first is No, but the second is Yes | 20:48 |
| mtreinish | (although there is nothing uploaded for 2010.09.26) | 20:48 |
| dstufft | Is there anything uploaded at all? | 20:49 |
| *** ccrouch1 has quit IRC | 20:49 | |
| dstufft | if there's ntohing uploaded it doesn't matter | 20:49 |
| sdague | clarkb: this is also a super old dead project | 20:49 |
| mtreinish | dstufft: https://pypi.python.org/pypi/Tempest/2009.10.31b | 20:49 |
| mtreinish | that release has something | 20:49 |
| *** deva_ has joined #openstack-infra | 20:49 | |
| dstufft | mtreinish: Ok so | 20:49 |
| dstufft | 10.0.0 will be < than that | 20:49 |
| dstufft | and thus pip/setuptools will prefer the 2009 release (sort of, it'll be a pre-release so it won't install it by default, but you'll get confusing results from --pre or so) | 20:50 |
| dstufft | so you have to either delete those old versions, or use epochs, or change your version scheme to be > 2009 | 20:50 |
| sdague | it really feels like delete should be fine in this circumstance | 20:51 |
| sdague | they are different things | 20:51 |
| mtreinish | dstufft: ok I can delete the old stuff. I was hoping to avoid having to do that in order to preserve history, etc. | 20:51 |
| *** sabeen1 has joined #openstack-infra | 20:51 | |
| mtreinish | but if I have to that shouldn't be a problem | 20:51 |
| *** esp has quit IRC | 20:51 | |
| dstufft | I don't have a problem with deletion | 20:51 |
| sdague | and it will only be massively confusing to see versions unrelated to current software | 20:51 |
| AJaeger | mtreinish: if you leave it in with the same name, it causes confusion... | 20:51 |
| clarkb | it mostly depends on if there is anything out there using the old version | 20:51 |
| dstufft | I'm just laying out options fwiw | 20:51 |
| dstufft | pick whatever one you like the most (or hate the least) | 20:51 |
| sdague | clarkb: their source code is only on google code, which is dead | 20:52 |
| sdague | and has been for some time | 20:52 |
| clarkb | sdague: that doesn't mean that something else isn't consuming it | 20:52 |
| dstufft | lifeless: re wsgi_scripts, I think you just need to generate the script at wheel build time. | 20:52 |
| mtreinish | sdague: well google code and gitorious (which is also dead and the actual code is deleted) | 20:52 |
| clarkb | I'm just saying you can break people by doing this | 20:52 |
| sdague | clarkb: you can | 20:52 |
| clarkb | in the same way that people break us and we get mad about | 20:52 |
| *** maishsk has quit IRC | 20:52 | |
| mtreinish | clarkb: the pypi owner gave up control of the repo | 20:52 |
| sdague | but people break us by deleting code they published this year | 20:52 |
| *** tcammann_ has quit IRC | 20:52 | |
| *** salv-orlando has joined #openstack-infra | 20:53 | |
| mtreinish | so he's ok with it at least | 20:53 |
| sdague | not last decade | 20:53 |
| *** esker has joined #openstack-infra | 20:53 | |
| dstufft | I can tel you if anyone is downloading that thing | 20:53 |
| sigmavirus24 | lifeless: threw an idea into the issue (since I'm toying with a different project's wheel file that uses scripts= with setuptools.setup) | 20:53 |
| clarkb | I don't think we can complain at eventlet or whoever if we do this | 20:53 |
| sdague | and it would be massively confusing to people to do a pip search and get 2 completely different software stacks | 20:53 |
| clarkb | instead we will just have to deal with it | 20:53 |
| *** krtaylor has quit IRC | 20:53 | |
| *** maishsk has joined #openstack-infra | 20:53 | |
| *** tcammann has quit IRC | 20:53 | |
| sdague | clarkb: I completely disagree | 20:53 |
| *** lxsli has quit IRC | 20:53 | |
| *** jsavak has quit IRC | 20:53 | |
| *** NobodyCa1 has quit IRC | 20:53 | |
| sdague | there are not absolutes, this is shades of grey | 20:53 |
| clarkb | its super simple to solve though | 20:54 |
| *** sabeen3 has quit IRC | 20:54 | |
| sdague | if eventlet deleted 0.12 from their repo | 20:54 |
| clarkb | call it openstack-tempest | 20:54 |
| clarkb | done | 20:54 |
| mtreinish | dstufft: the counts on pypi page show 15 in the last day? | 20:54 |
| *** deva_ has quit IRC | 20:54 | |
| *** jsavak has joined #openstack-infra | 20:54 | |
| *** ccrouch has joined #openstack-infra | 20:54 | |
| sdague | mtreinish: I wonder how many of those people are people trying to install our tempest by mistake | 20:54 |
| mtreinish | sdague: I just assumed it was mirrors or something | 20:55 |
| sdague | yeh, probably | 20:55 |
| dstufft | https://caremad.io/s/PbqzFMsUlG/ | 20:55 |
| mtreinish | clarkb: well except it'll conflict with tempest if the pypi on tempest was installed | 20:55 |
| dstufft | this is since 1-14-2016 | 20:55 |
| dstufft | or so | 20:55 |
| lifeless | I guess we don't need to worry about windows for this | 20:56 |
| sdague | clarkb: right, we also have to rename all the tempest code. If you are really concerned about a collision | 20:56 |
| clarkb | I am less worried about the collision in the code because yo ucan separate the installs | 20:56 |
| slogan621 | Ajaeger: thanks! | 20:56 |
| clarkb | but if you try to intsall one or the other and get the wrong thing and even worse can no longer install the item you want that is bad | 20:57 |
| sdague | clarkb: well, I believe you a wrong. I think that if people really want 8 year old dead code, that's what the wayback machine is for | 20:57 |
| clarkb | sdague: 1k installs happened in the last month | 20:58 |
| *** rlandy has quit IRC | 20:58 | |
| mtreinish | clarkb: well, at least 2 of those were me :) | 20:58 |
| * AJaeger just counted jobs defined in JJB and jobs used in zuul - we have 8800 in JJB and 3100 are not used in zuul... | 20:59 | |
| *** esker has quit IRC | 20:59 | |
| clarkb | AJaeger: wow! | 20:59 |
| dstufft | I (obviously) have no way of determining if those were accidental or not | 20:59 |
| clarkb | AJaeger: how many of those come from gratuitous use of python-jobs? | 20:59 |
| clarkb | my guess is a good portion | 21:00 |
| mtreinish | dstufft: yeah | 21:00 |
| dstufft | (Personally I'd just delete the old code and move on, but with my PyPI hat on, I have no opinion on what y'all do) | 21:00 |
| *** rlandy has joined #openstack-infra | 21:00 | |
| mtreinish | I'm frantically googling try to find another reference to it anywhere and am only finding the google coee page | 21:00 |
| mtreinish | which tells you to install it from the tarball: https://code.google.com/archive/p/image-tempest/wikis/PythonInstall.wiki | 21:00 |
| sigmavirus24 | lifeless: worry about windows for what? | 21:01 |
| clarkb | maybe create a tempest-old-and-gone on pypi and dump the old releases there? at least you can point people to it if they really need it | 21:01 |
| lifeless | sigmavirus24: wsgi entry points | 21:01 |
| *** infra-red has joined #openstack-infra | 21:01 | |
| *** eil397 has joined #openstack-infra | 21:01 | |
| sigmavirus24 | lifeless: uhh | 21:01 |
| lifeless | sigmavirus24: scripts on windows need to be .exe files | 21:01 |
| sigmavirus24 | is Nova using them? | 21:01 |
| AJaeger | clarkb: I expect the majority, currently playing with it... | 21:01 |
| sigmavirus24 | also I'm not sure scripts will put those files in the right place | 21:01 |
| *** nikhil_k is now known as nikhil_ | 21:01 | |
| lifeless | sigmavirus24: there's special case logic in pip to build .exe's out of scripts w/in wheels, but I don't recall extactly how its all wired up | 21:01 |
| mtreinish | clarkb: I'd probably do image-tempest, which is the project name on google code | 21:01 |
| mtreinish | but that's what I was thinking | 21:01 |
| *** daemontool has joined #openstack-infra | 21:02 | |
| sigmavirus24 | lifeless: I think Nova can be run on windows so if Nova needs them then maybe not? | 21:02 |
| lifeless | sigmavirus24: apache on windows is very rare :) | 21:02 |
| sigmavirus24 | Ah good point | 21:02 |
| *** nikhil_ is now known as nikhil_k | 21:02 | |
| lifeless | sigmavirus24: wsgi entry points should really be named 'mod_wsgi' entry points | 21:02 |
| *** daemontool__ has quit IRC | 21:02 | |
| * sigmavirus24 never grokked nova on windows but was told it was a thing | 21:02 | |
| dstufft | if they're platform specific you can still do that | 21:02 |
| sigmavirus24 | lifeless: that makes a lot more sense | 21:02 |
| dstufft | you just need to generate platform specific wheels | 21:02 |
| AJaeger | clarkb: 2500 comes from python-jobs | 21:03 |
| lifeless | dstufft: so, keystone on windows should work; keystone on windows under apache - I'll wait for the bug reports | 21:03 |
| *** mikelk has joined #openstack-infra | 21:03 | |
| *** dims has joined #openstack-infra | 21:03 | |
| *** krotscheck is now known as krotscheck_dcm | 21:03 | |
| *** e0ne has joined #openstack-infra | 21:03 | |
| sdague | mtreinish: honestly, I'd wait until someone sends an email | 21:03 |
| sdague | and then repost it | 21:03 |
| dstufft | lifeless: seems fine to me | 21:04 |
| sdague | it's not like an automated system will handle it anyway | 21:04 |
| mtreinish | sdague: well I'll have to archive it somewhere then | 21:04 |
| AJaeger | clarkb: 396 times unused pypy | 21:04 |
| mtreinish | and not forget where that is | 21:04 |
| mtreinish | because it's not like the repo exists on gitorious anymore | 21:04 |
| anteaya | slogan621: you have what you need? | 21:04 |
| sdague | mtreinish: yeh, put it in a directory on your computer | 21:04 |
| AJaeger | so, 430 repos define pypy but 396 do not use it... | 21:05 |
| *** infra-re_ has quit IRC | 21:05 | |
| anteaya | AJaeger: awesome | 21:05 |
| clarkb | AJaeger: wow | 21:05 |
| AJaeger | time to remove pypy for good? | 21:05 |
| anteaya | AJaeger: what gerrit topic are you using for your cleanup patches? | 21:05 |
| anteaya | reduce-jobs? | 21:05 |
| *** sridhar_ram has quit IRC | 21:06 | |
| *** Guest9860 has quit IRC | 21:06 | |
| AJaeger | anteaya: cleanup-devstack was what I was working on. Planned as a small spring cleanup... | 21:06 |
| AJaeger | https://review.openstack.org/#/q/topic:cleanup-devstack | 21:07 |
| anteaya | ah cool, I'll search on that | 21:07 |
| anteaya | perfect thans | 21:07 |
| anteaya | put a k in there | 21:07 |
| *** krtaylor has joined #openstack-infra | 21:07 | |
| anteaya | still winter here but I like your thinking | 21:07 |
| AJaeger | compare pypy with python34: 430 repos and 219 unused | 21:07 |
| *** eil397 has quit IRC | 21:08 | |
| AJaeger | or python27: 520 repos and 95 unused | 21:08 |
| anteaya | I could use you today in my house, I set two mouse traps yesterday thinking I had one mouse, checked a trap last night, dead mouse (clean kill, yay) and went to bed | 21:08 |
| slogan621 | anteaya: I have a +2, but looking for one from you so I suppose the thing to do is nail this commit message. Can I propose one here, get us to agree, and then comment on the review? The developer is likely asleep at this point and so I am acting as his ears and eyes, so to speak. | 21:08 |
| anteaya | today my other mousetrap is missing, can't even find it in the room where I placed it :( | 21:08 |
| anteaya | slogan621: yeah tidy the commit message please | 21:09 |
| mtreinish | anteaya: they're evolving... | 21:09 |
| anteaya | mtreinish: must be | 21:09 |
| *** sridhar_ram has joined #openstack-infra | 21:09 | |
| anteaya | I'm staying upstairs for now | 21:09 |
| anteaya | slogan621: here is a great page on commit messages: https://wiki.openstack.org/wiki/GitCommitMessages | 21:10 |
| nibalizer | mtreinish: so why is os-tempest pushing out the existing tempest on pypi? | 21:10 |
| nibalizer | that seems really odd and clunky | 21:10 |
| anteaya | slogan621: you should find all you need for a clear, useful, nicely formatted commit message there | 21:10 |
| mtreinish | nibalizer: we've wanted to host tempest on pypi for a long time. It's either take the space from the dead project or rename tempest | 21:12 |
| mtreinish | renaming is not a small undertaking, and the owner of the pypi project had no issues giving up the spot on pypi | 21:12 |
| nibalizer | but packages have different names than the software all the time | 21:13 |
| sdague | nibalizer: and are massively confusing when they do | 21:13 |
| dims | mordred : i got some details from the maintainer here : https://github.com/eventlet/eventlet/commit/5bf0a6f32b3e4459b38ad1895c9eb4b0b483dae1#commitcomment-15987613 | 21:16 |
| *** ybathia has joined #openstack-infra | 21:16 | |
| nibalizer | sdague: i argree that it can be confusing | 21:16 |
| *** alivigni has quit IRC | 21:16 | |
| nibalizer | but wiping out an existing release with completely different software doesn't seem like its ever the right decision | 21:16 |
| jlvillal | sdague: There is a small issue with PS4 I think. https://bugs.launchpad.net/devstack/+bug/1543749 | 21:17 |
| openstack | Launchpad bug 1543749 in devstack "devstack installation with Ironic fails on RHEL7" [Undecided,New] - Assigned to Surojit Pathak (suro-patz) | 21:17 |
| mordred | dims: cool - that still doesn't explain actually physically removing 0.18.1 though | 21:17 |
| *** alivigni has joined #openstack-infra | 21:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove tempest-dsvm-test-accounts-src from lib-forward-testing https://review.openstack.org/278091 | 21:18 |
| sdague | nibalizer: I disagree, it's a dead project. Leaving the field of dead everywhere because it existed in a previous decade to confuse people that are used to names meaning the new thing is just another way to surprise everyone. | 21:18 |
| mgagne | I have a successful (and merged) change stuck in zuul status interface (in gate pipeline). in fact, it takes a long time before it goes. any idea? is it waiting for an event from gerrit? | 21:19 |
| dims | fungi : AJaeger : anteaya : i owe you an update on moving the py27/34 jobs from travis to our CI - here's the etherpad where i am tracking things - https://etherpad.openstack.org/p/dims-periodic-jobs | 21:20 |
| clarkb | mgagne: how long are we talking? a couple minutes while results and event queues get processed and the response fro gerrit on whether or not it actually merged are not uncommon | 21:20 |
| clarkb | mgagne: but if you are talking hours then that would be abnormal | 21:20 |
| *** yolanda_ has joined #openstack-infra | 21:20 | |
| mgagne | clarkb a couple of minutes yea. I'm curious about the reasons | 21:21 |
| mgagne | clarkb because when change is in check pipeline, it's gone in no time | 21:21 |
| openstackgerrit | Merged openstack-infra/release-tools: process pypi flag for release announcements https://review.openstack.org/276469 | 21:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove gate-nova-pylint https://review.openstack.org/278094 | 21:21 |
| nibalizer | sdague: looking at dstufft's data, it looks like its been pip installed 2k times in less than a month | 21:21 |
| nibalizer | that sounds decidedly un-dead | 21:21 |
| fungi | mgagne: zuul may be polling git trying to figure out whether the change is really merged. i think we recalled recently that we're currently polling a replica rather than the internal jgit in gerrit for that so perhaps your relication is lagging? | 21:21 |
| clarkb | fungi: yup | 21:22 |
| AJaeger | dims: glad to see the progress! | 21:22 |
| clarkb | ya so it is basically spending time checking that it merged properly | 21:22 |
| clarkb | it may not need to be so cautious, I pushed a patch to hit gerrit rather than the mirror | 21:22 |
| clarkb | but it still checks | 21:22 |
| AJaeger | anteaya: enough cleanup for today | 21:22 |
| * AJaeger waves good night | 21:22 | |
| mgagne | fungi it's replicated to github already. is there an other replication I should care about? | 21:22 |
| *** weshay_xchat has joined #openstack-infra | 21:23 | |
| nibalizer | sdague: i think i see your point, but it looks like people are using it | 21:23 |
| anteaya | AJaeger: good night | 21:23 |
| anteaya | AJaeger: thanks for clean up | 21:23 |
| sdague | nibalizer: I am doubtful of that | 21:23 |
| clarkb | well something is downloading it, hard to rule out mirroring etc completely | 21:23 |
| nibalizer | sdague: how do you explain the 2k downloads? | 21:23 |
| fungi | mgagne: in our case zuul is polling a mirror at https://review.openstack.org/p/<reponame> | 21:24 |
| anteaya | dims: you don't owe me anything, but thank you for the etherpad | 21:24 |
| openstackgerrit | Craig Vyvial proposed openstack-infra/project-config: run trove experimental run scenario tests https://review.openstack.org/278095 | 21:24 |
| * anteaya loves a chance to get the big picture | 21:24 | |
| clarkb | mgagne: but also it has to process each result and event one at a time | 21:24 |
| clarkb | in order to maintain a consistent internal state | 21:24 |
| nibalizer | dstufft: could you pull the same data for a package that is actually super-dead-broken-not real? | 21:24 |
| nibalizer | that way we'd see the numbers that are just dumb bots pulling everything? | 21:24 |
| *** daemontool_ has joined #openstack-infra | 21:24 | |
| slogan621 | anteaya: what is the best way to share the proposed change? pastebin? | 21:24 |
| clarkb | the result queue must drop to zero before the event queue is processed | 21:25 |
| slogan621 | commit line < 50, lines <= 72 | 21:25 |
| clarkb | which can also introduce delay if you have a continuous feed of results | 21:25 |
| slogan621 | no period at the end :-) | 21:25 |
| anteaya | slogan621: well I would suggest you comment on the patch so the owner can make the change to fix the commit message | 21:26 |
| odyssey4me | hmm, I thought this would be sorted out today - I did a pip wheel build with the index pointing at the infra wheel mirror, and the extra-index pointing at the pypi mirror, and it fails on oslo.concurrency as the wheel mirror doesn't have the version needed. I see that the pypi mirror does though. any thoughts? ref: http://logs.openstack.org/67/265867/12/check/gate-openstack-ansible-dsvm-commit/e4a49b1/console.html#_201 | 21:26 |
| odyssey4me | 6-02-09_17_54_12_337 | 21:26 |
| *** erlon has quit IRC | 21:26 | |
| anteaya | slogan621: you can practice your reviews | 21:26 |
| *** daemontool has quit IRC | 21:27 | |
| clarkb | odyssey4me: we are only building what global requirements wants iirc | 21:27 |
| slogan621 | anteaya: nod. And I hope you can comment on my comment | 21:27 |
| mgagne | could it be linked to window-floor config? I'm not sure about its purpose | 21:27 |
| clarkb | odyssey4me: so if you have otherwise pinned or are using older commits that could explain it | 21:27 |
| odyssey4me | clarkb yep, I see that in the wheel mirror - what I don't understand is why the fallback to the pypi mirror isn't working | 21:27 |
| anteaya | slogan621: I'll let you have a round first, you seemed to understand the message of the wikipage | 21:28 |
| clarkb | mgagne: no, that is just configuring how many changes can be "active" in the gate at one time | 21:28 |
| *** matrohon has joined #openstack-infra | 21:28 | |
| fungi | clarkb: odyssey4me: the upper-constraints.txt in particular, i believe | 21:28 |
| craige | o/ | 21:28 |
| *** dkranz has quit IRC | 21:28 | |
| odyssey4me | fungi clarkb yeah, the wheel mirror only has the upper contraints and the wheel mirror has normalised names - the pypi mirror does not have normalised names... I'm wondering if that is the issue. dstufft thoughts? | 21:29 |
| clarkb | odyssey4me: we have our indexes the other way around | 21:29 |
| clarkb | odyssey4me: you could just grab it off the host? | 21:29 |
| clarkb | pypi mirror proper is main index, extra is wheels | 21:29 |
| mgagne | takes about ~12m for change to get removed from UI | 21:29 |
| *** alivigni has quit IRC | 21:29 | |
| dstufft | nibalizer: pick a name and I'll look it up | 21:30 |
| dstufft | odyssey4me: bandersnatch makes both normalized and unnormalized names, if openstack does something else besides that I dunno, but it should make the normalized name available. | 21:30 |
| mgagne | "INFO zuul.DependentPipelineManager: Reported change <Change 0x7f8fdc098790 3,1> status: all-succeeded: True, merged: False" | 21:30 |
| mgagne | looks to be it | 21:30 |
| fungi | odyssey4me: one of these things is not like the other | 21:31 |
| odyssey4me | dstufft so the wheel mirror has http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/oslo-concurrency/ and the pypi mirror has http://mirror.bhs1.ovh.openstack.org/pypi/simple/oslo.concurrency/ | 21:31 |
| fungi | Failed to find 'hacking' at http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/hacking/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}. | 21:31 |
| *** dprince has quit IRC | 21:31 | |
| *** jgriffith is now known as jgriffith_away | 21:31 | |
| fungi | what's non-normalized about "hacking" there? | 21:31 |
| odyssey4me | my constraints at this sha is looking for oslo.concurrency<=3.3.0 | 21:31 |
| odyssey4me | fungi this is what is: http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/hacking/ :) (404) | 21:32 |
| slogan621 | anteaya: posted my comment (including the link so the developer can review guidelines, thanks for that) | 21:32 |
| clarkb | we do need to rewrite oslo.concurrency to oslo-concurrency iirc | 21:32 |
| odyssey4me | ie hacking is not on the wheel mirror | 21:32 |
| *** alivigni has joined #openstack-infra | 21:33 | |
| *** achanda has joined #openstack-infra | 21:33 | |
| fungi | odyssey4me: righth, we don't put hacking and some other linters in the constraints list, so they won't get built in our wheel mirror, however there are wheels of those already in our pypi mirror because we upload them to pypi (they're just pure python) | 21:33 |
| anteaya | slogan621: welcome, thanks for reading and following up, nice work | 21:33 |
| odyssey4me | fungi yep, so I think that's a red herring | 21:33 |
| slogan621 | anteaya: and thanks for the help with the learning curve. | 21:34 |
| dstufft | fungi: nothing | 21:34 |
| odyssey4me | fungi that warning appears to just relate to there being nothing in the primary index... I think the issue here is that pip isn't finding oslo.concurrency in the extra index | 21:35 |
| nibalizer | dstufft: just looking at the download stats of other things in pypi, many packages have 0-0-0 | 21:35 |
| clarkb | odyssey4me: yes because we don't rewrite it to oslo-concurrency | 21:36 |
| *** jsavak has quit IRC | 21:36 | |
| dstufft | fungi: it 404'd, can't tell the difference between "we needed to fallback to try and discover the name because we don't know the un-normalized name for it" and "fell back because it legitly doesn't exist" | 21:36 |
| anteaya | slogan621: you are welcome, thanks for digging in | 21:36 |
| dstufft | and pip 8 (I think? might be pip 9) just removes that fallback | 21:36 |
| odyssey4me | clarkb nor does pypi? https://pypi.python.org/simple/oslo.concurrency | 21:36 |
| *** tpsilva has quit IRC | 21:36 | |
| openstackgerrit | Craig Vyvial proposed openstack-infra/project-config: run trove experimental run scenario tests https://review.openstack.org/278095 | 21:36 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update chairs for Neutron's L3 sub-team meeting https://review.openstack.org/278038 | 21:37 |
| anteaya | slogan621: some folks believe I'm telling them to go away, I'm not, I'm trying to help, and this is how I do it | 21:37 |
| anteaya | slogan621: thanks for sticking with it | 21:37 |
| *** tiswanso has quit IRC | 21:37 | |
| *** jsavak has joined #openstack-infra | 21:37 | |
| clarkb | odyssey4me: https://pypi.python.org/simple/oslo-concurrency/ | 21:37 |
| clarkb | it has to do with name normalization, if you use new enough pip I blieve it will do it client side | 21:37 |
| clarkb | dstufft: ^ is that right? | 21:38 |
| odyssey4me | clarkb oh interesting, so pypi does both normalised and non-normalised names | 21:38 |
| dstufft | clarkb: yea | 21:38 |
| odyssey4me | dstufft this is using pip 7.1.2 - what's new enough? | 21:38 |
| mordred | odyssey4me: you should always use latest pip | 21:38 |
| *** baoli has joined #openstack-infra | 21:38 | |
| odyssey4me | mordred that would be nice, but breaking changes do not for a stable environment make :) | 21:39 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: networking-ovn: Add an install job for networking-ovn + kuryr https://review.openstack.org/278101 | 21:39 |
| mordred | odyssey4me: I hear that - but openstack assumes latest pip and does not make any attempt to work with older pip | 21:39 |
| mordred | odyssey4me: so if you're installing from pip, I strongly recommend installing pip from pip | 21:39 |
| dstufft | PyPI redirects the non-cannonical name to the cannonical name, previously that was "whatever the person originally registered the name as" but that meant that give a command like ``pip install Django`` we didn't know if we need /simple/django/ or /simple/Django/ or /simple/DjAnGo/. It worked no matter what on PyPI proper because of the redirects, but that was hard to do on static mirrors, and failing meant we had to fetch /simple/ | 21:40 |
| dstufft | and scan that for links... which is a 4MB download | 21:40 |
| *** ybathia has quit IRC | 21:40 | |
| dstufft | so we switched it so the cannonical name was the normalized name | 21:40 |
| *** vgridnev has quit IRC | 21:40 | |
| dstufft | and pip always fetches the cannonical name | 21:40 |
| dstufft | and the fallback to /simple/ was deprecated | 21:40 |
| *** infra-red has quit IRC | 21:41 | |
| *** jsavak has quit IRC | 21:41 | |
| cody-somerville | hmm... mordred, jeblair: Were you able to get openafs-client-dkms to compile on Ubuntu Trusty? | 21:42 |
| odyssey4me | dstufft so what would the correct approach to use that will work with pip7+ ? | 21:42 |
| *** davideagnello has quit IRC | 21:42 | |
| pleia2 | come to think of it, I've only used it on the 6 month releases | 21:42 |
| dstufft | odyssey4me: use the normalized name in the repository | 21:42 |
| *** tiswanso has joined #openstack-infra | 21:43 | |
| *** regXboi has quit IRC | 21:43 | |
| *** davideagnello has joined #openstack-infra | 21:43 | |
| prometheanfire | odyssey4me: ohai | 21:43 |
| dstufft | PEP 503 has the simple repository documented | 21:43 |
| dstufft | follow that :] | 21:43 |
| *** achanda has quit IRC | 21:43 | |
| odyssey4me | dstufft so if requirements.txt has 'oslo.concurrency' then we should convert that before executing the pip install/pip wheel? | 21:43 |
| dstufft | odyssey4me: no | 21:44 |
| dstufft | odyssey4me: pip will convert it | 21:44 |
| odyssey4me | ah, so the repo itself is the issue here | 21:44 |
| dstufft | the repository, e.g. http://mirror.bhs1.ovh.openstack.org/pypi/simple/oslo-concurrency/ | 21:44 |
| odyssey4me | in this case, the pypi mirror needs the names normalised | 21:44 |
| dstufft | not http://mirror.bhs1.ovh.openstack.org/pypi/simple/oslo.concurrency/ | 21:44 |
| dstufft | if the mirror is built using a recent enough bandersnatch, it should have *both* oslo.concurrency and oslo-concurrency, for compatability sake | 21:45 |
| *** nikhil_k is now known as nikhil | 21:45 | |
| *** ybathia has joined #openstack-infra | 21:45 | |
| dstufft | if it's built using some custom tooling, then adjust that tooling :] | 21:45 |
| clarkb | it is bandersnatch | 21:46 |
| clarkb | the wheel mirror is alrady correct with oslo-concurrency | 21:46 |
| sdague | infra-root: https://review.openstack.org/#/c/278089 can we get that promoted to fix the nova gate | 21:46 |
| *** jsavak has joined #openstack-infra | 21:46 | |
| odyssey4me | dstufft for our own repo, which is consumes other repositories, I'll check against pep503 compliance... in this case the issue is bandersnatch as its the infra pypi mirror :) | 21:46 |
| odyssey4me | so clarkb if you can point me at something, I'll make the codes if I can fathom what's there :) | 21:47 |
| jeblair | cody-somerville: yes -- but without the lts enablement stack. | 21:47 |
| clarkb | odyssey4me: we may just need to update bandersnatch where we are building the mirror now | 21:47 |
| clarkb | and I have no idea where we build the mirror now | 21:47 |
| cody-somerville | jeblair: so an older version of the kernel? | 21:47 |
| jeblair | cody-somerville: the _trusty_ version of the kernel | 21:47 |
| clarkb | would have to dig backwards in puppet to find it | 21:47 |
| odyssey4me | clarkb ah | 21:48 |
| dstufft | (it's also possible that bandersnatch isn't doing it correctly in all cases) | 21:48 |
| *** achanda has joined #openstack-infra | 21:48 | |
| odyssey4me | I noticed, also, that the wheel mirror is ubuntu only at this point - is that right? | 21:48 |
| lifeless | dstufft: does bandersnatch do it itself, or is it in th efeed data? | 21:48 |
| jeblair | cody-somerville: (i'm not sure that was ultimately a factor, however, having it enabled, installing some things, and then removing it, and then installing openafs definitely doesn't work, which is what was happening with some of our puppet configuration) | 21:48 |
| dstufft | lifeless: bandersnatch does it itself | 21:48 |
| clarkb | odyssey4me: yes since we have to do per distro release and arch builds of wheels | 21:48 |
| lifeless | clarkb: ^ so yes | 21:48 |
| jeblair | cody-somerville: (our solution to that problem was to not use it at all) | 21:48 |
| clarkb | odyssey4me: currently only doing that for ubuntu 14.04 on x86_64 | 21:48 |
| *** alivigni has quit IRC | 21:49 | |
| *** sdake has quit IRC | 21:49 | |
| dstufft | https://bitbucket.org/pypa/bandersnatch/src/31c0ffcc8d5e8300339acc7e41894356f83a35ba/src/bandersnatch/package.py?at=default&fileviewer=file-view-default#package.py-26 | 21:49 |
| fungi | cody-somerville: fwiw, i had no problems with apt-get install openafs-client-dkms, but i was using a relatively current debian/sid | 21:49 |
| dstufft | heh | 21:49 |
| dstufft | this is probably an safe_name thing | 21:49 |
| dstufft | I think it allows . | 21:49 |
| *** |-paul-| has quit IRC | 21:50 | |
| fungi | cody-somerville: though in theory we're installing it on our mirror servers and mirror build workers | 21:50 |
| fungi | which are ubuntu-trusty | 21:50 |
| dstufft | probably replacing the call to pkg_resources.safe_name with the normalize function from PEP 503 will be enough (maybe should move that to the packaging lib, though it seems silly to add a 1 line function) | 21:50 |
| jeblair | fungi: ^ (without lts enablement) | 21:51 |
| lifeless | dstufft: +1 on adding it | 21:51 |
| clarkb | in theory we always use latest bandersnatch | 21:51 |
| clarkb | but it is working from an existing mirror each time | 21:51 |
| *** amrith is now known as _amrith_ | 21:51 | |
| jeblair | clarkb: we use a fork of bandersnatch | 21:51 |
| clarkb | we do? | 21:52 |
| clarkb | :/ | 21:52 |
| jeblair | clarkb: we're waiting for https://bitbucket.org/pypa/bandersnatch/pull-requests/16/add-option-to-dir-hash-index-files to land | 21:52 |
| *** FallenPegasus has joined #openstack-infra | 21:52 | |
| clarkb | oh for some reason I thought that merged | 21:52 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: networking-ovn: Add an install job for networking-ovn + kuryr https://review.openstack.org/278101 | 21:52 |
| jeblair | clarkb: all of the previous review comments have been addressed | 21:52 |
| jeblair | clarkb, dstufft: so i don't expect it to be a problem, but no one has looked at it yet | 21:53 |
| fungi | jeblair: ahh, yep, sorry. replying to things in scrollback without first reading all of scrollback again | 21:53 |
| jeblair | clarkb: there have been no bandersnatch commits since our fork | 21:53 |
| openstackgerrit | Merged openstack-infra/irc-meetings: nova bugs team meeting https://review.openstack.org/278043 | 21:54 |
| clarkb | we will need to do the hg equivalent of a rebase if bandersnatch fixes this | 21:55 |
| *** ccrouch has quit IRC | 21:55 | |
| clarkb | or get them to merge both things | 21:55 |
| jeblair | clarkb: aiui there is no hg equivalent of a rebase | 21:55 |
| jeblair | 21:44 < dstufft> the repository, e.g. http://mirror.bhs1.ovh.openstack.org/pypi/simple/oslo-concurrency/ | 21:56 |
| jeblair | 21:44 < odyssey4me> in this case, the pypi mirror needs the names normalised | 21:56 |
| jeblair | 21:44 < dstufft> not http://mirror.bhs1.ovh.openstack.org/pypi/simple/oslo.concurrency/ | 21:56 |
| mordred | maybe if they fix this, they'll also merge our patch so we can stop using a fork | 21:56 |
| mordred | that would be neat | 21:56 |
| jeblair | gah | 21:56 |
| fungi | they just add commits to address merge conflicts when tying them together? | 21:56 |
| *** claudiub has quit IRC | 21:56 | |
| jeblair | clarkb: so can you summarise the issue? | 21:56 |
| dstufft | Hg has rebase | 21:56 |
| dstufft | it's in an extension though | 21:56 |
| greghaynes | I think rebase is just an extension in hg | 21:56 |
| greghaynes | jinx | 21:56 |
| greghaynes | like everything else | 21:57 |
| clarkb | jeblair: the issue is that oslo.concurrency is canonically oslo-concurrency so needs to be at /simple/oslo-concurrency | 21:57 |
| pleia2 | called and confirmed that the courtyard has a local shuttle too (thanks cody-somerville for confirming the other two) | 21:57 |
| mordred | clarkb: and this is breaking our mirror? | 21:57 |
| clarkb | jeblair: our wheel mirror does that just fine, but our bandersnatch mirror does not. apparently because the . is extra special in the normalization code | 21:57 |
| mordred | ah | 21:57 |
| jeblair | even though pypi itself does? | 21:57 |
| clarkb | pip is smart enough that things work | 21:57 |
| dstufft | pkg_resources uses a different regex from pip and PyPI | 21:57 |
| *** ccrouch has joined #openstack-infra | 21:57 | |
| clarkb | but it doesn't use the wheel for some reason | 21:58 |
| dstufft | and bandersnatch uses pkg_resources | 21:58 |
| clarkb | it uses the sdist on the bandersnatch mirror that isn't canonicaly pathed | 21:58 |
| *** jpeeler has quit IRC | 21:58 | |
| anteaya | dims: updated your etherpad with a link to the keystone patch | 21:58 |
| *** _amrith_ is now known as amrith | 21:58 | |
| dims | anteaya : thanks | 21:58 |
| clarkb | I am guessing because the oslo.conccurency name more closely matches what is in our reqs? | 21:58 |
| anteaya | welcome | 21:58 |
| mgagne | moved our first project under Gerrit/Zuul w00t | 21:58 |
| anteaya | dims: nice clear etherpad, well done | 21:59 |
| anteaya | dims: any leads on how to address the neutron issue(s)? | 21:59 |
| mordred | clarkb: so, in theory we can fix this in our fork of bandersnatch that is in jeblair's repo by changing pkg_resources.safe_name to something else, yeah? | 21:59 |
| clarkb | mordred: yes | 21:59 |
| jeblair | clarkb: so it sounds like two issues, possibly separate? 1) bandersnatch doesn't do oslo-concurrency; 2) we aren't using the wheel for a reason that may or may not be related to (1)? | 21:59 |
| anteaya | pleia2: thanks for the update | 21:59 |
| dims | anteaya : y still toying with it. hopefully will come up with something soon | 22:00 |
| clarkb | jeblair: yes | 22:00 |
| jeblair | whew | 22:00 |
| mordred | dstufft: what's the thing we want to use instead of pkg_resources.safe_name() ? | 22:00 |
| *** jsavak has quit IRC | 22:00 | |
| anteaya | dims: okay let me know if you need a hand to find some neutron folks | 22:00 |
| fungi | "normalize function from PEP 503" | 22:00 |
| *** esker has joined #openstack-infra | 22:00 | |
| dims | anteaya : ack thanks | 22:00 |
| clarkb | jeblair: but it also sounds like odyssey4me's thing is writing its own pip.conf so could be completely separate issue too | 22:00 |
| anteaya | dims: and for reference, I'm so tired these days trying to figure out what people are saying, I do favour properly spelled words | 22:01 |
| *** camunoz has quit IRC | 22:01 | |
| anteaya | dims: or an attempt as I am a poor speller | 22:01 |
| mordred | clarkb: this is only a problem for odyssey4me ? | 22:01 |
| anteaya | dims: and you're welcome | 22:01 |
| jeblair | clarkb: oh, yeah, odyssey4me is going to fix that | 22:01 |
| clarkb | mordred: I havne't confirmed if it is an issue outside of that job | 22:01 |
| mordred | ok | 22:01 |
| dstufft | mordred: digging into this a bit sec | 22:01 |
| dims | :) | 22:01 |
| fungi | i believe odyssey4me is saying that the proposed change to _not_ do special stuff any longer is running into this https://review.openstack.org/#/c/265867/ | 22:01 |
| anteaya | :) | 22:01 |
| mordred | dstufft: awesome. thanks! | 22:01 |
| odyssey4me | mordred well, I'm trying to use the infra pypi/wheel mirror for our builds, and finding bugs :) | 22:01 |
| clarkb | odyssey4me: yes but are you using our mirror config? | 22:02 |
| clarkb | from pip.conf? | 22:02 |
| clarkb | what you described was different than what we configure | 22:02 |
| odyssey4me | clarkb the particular brokenness is happening in this command execution: http://logs.openstack.org/67/265867/12/check/gate-openstack-ansible-dsvm-commit/e4a49b1/console.html#_2016-02-09_17_54_12_283 | 22:02 |
| *** rossella_s has quit IRC | 22:03 | |
| *** Sukhdev has joined #openstack-infra | 22:03 | |
| fungi | ahh, yeah looking at the actual patch in 265867 it looks a lot less like what i anticipated from the commit message | 22:03 |
| clarkb | yup that config is different than ours | 22:03 |
| jeblair | odyssey4me: eventually you are going to use the actual pip.conf files we write, right? | 22:03 |
| fungi | i was expecting ^ that | 22:03 |
| *** matrohon has quit IRC | 22:03 | |
| clarkb | for us extra index url is the wheel mirror and index url is the bandersnatch mirror | 22:03 |
| *** yolanda_ has quit IRC | 22:03 | |
| *** rossella_s has joined #openstack-infra | 22:04 | |
| jeblair | because duplicating infra code in your project is not the best long term strategy. | 22:04 |
| clarkb | so you hvae flipped the order which may or may not play a part here | 22:04 |
| odyssey4me | jeblair yes, this is part of the process to test whether that is an option for us - which it currently is not due to this problem | 22:04 |
| fungi | i'm not too inclined to dig much deeper on troubleshooting behavior from tests of that change | 22:04 |
| dstufft | bleh | 22:04 |
| dstufft | I Hate computers | 22:04 |
| anteaya | me too | 22:04 |
| jeblair | and, to be perfectly honest, this is why -- we spent months working through all of this and i don't think we should be solving it again | 22:04 |
| mordred | odyssey4me: how hard would it be for you to test upgrading your build to latest pip before doing your build to see if that makes it go away? | 22:04 |
| dims | LOL | 22:04 |
| *** annegentle_ has quit IRC | 22:04 | |
| dstufft | a refactor lost half of the normalization at some point in pip | 22:04 |
| fungi | odyssey4me: where is the test of using our pip.conf? | 22:04 |
| clarkb | odyssey4me: wait, it would install it from sdist? is that a problem? | 22:04 |
| mordred | so - reading odyssey4me's patch | 22:04 |
| fungi | i don't see that actually tested with this change | 22:04 |
| mordred | he's passing in the same parameters to his currently parameterized ansible playbook | 22:05 |
| cloudnull | computers are terrible | 22:05 |
| *** esker has quit IRC | 22:05 | |
| *** sridhar_ram has quit IRC | 22:05 | |
| *** annegentle_ has joined #openstack-infra | 22:05 | |
| mordred | it looks like the playbook is not structured to take a pip.conf file as input | 22:05 |
| mordred | so he's doing what he can to get the values passed in to the right parameters | 22:05 |
| odyssey4me | nope, it's not - that a next step option | 22:05 |
| clarkb | mordred: no he is not | 22:05 |
| anteaya | pleia2: could I trouble you for a review of this patch which fixs a bug in our implementation of gerrit, whereby jeepyb can create gerrit groups that are system groups, the upstream patch has merged but backporting is more effort than it is worth: https://review.openstack.org/#/c/270537/5 | 22:05 |
| clarkb | mordred: the config is inverted | 22:05 |
| mordred | clarkb: I know | 22:05 |
| *** FallenPegasus has quit IRC | 22:05 | |
| mordred | odyssey4me: I suggest trying 2 things | 22:05 |
| clarkb | if we uninverted it that would be a trivial step one | 22:05 |
| pleia2 | anteaya: sure, looking | 22:05 |
| slogan621 | nothing wrong with computers - it's users that are often the problem :-) | 22:06 |
| anteaya | pleia2: thank you | 22:06 |
| odyssey4me | so step back a patch set, and it was inverted the other way | 22:06 |
| mordred | odyssey4me: a) inverting the order of your parameters | 22:06 |
| odyssey4me | :) | 22:06 |
| *** annegentle_ has quit IRC | 22:06 | |
| mordred | odyssey4me: and b) upgrading pip to latest in your thing | 22:06 |
| *** abregman is now known as abregman|afk | 22:06 | |
| *** annegentle_ has joined #openstack-infra | 22:06 | |
| mordred | odyssey4me: that gets you closest to the way the other things are working | 22:06 |
| odyssey4me | ok, not a patch set - several | 22:06 |
| clarkb | odyssey4me: and is there some problem installing from an sdist? | 22:06 |
| odyssey4me | mordred so upgrading pip is not an option for the stable branches - which means we're stuck with horribleness - so making this work like this is ideal | 22:07 |
| *** annegentle_ has quit IRC | 22:07 | |
| clarkb | because we definitely install oslo.concurrency eselwhere I would just expect it to sdist | 22:07 |
| odyssey4me | clarkb it's slower, apparently | 22:07 |
| odyssey4me | but yeah, I can invert it and see what happens | 22:07 |
| clarkb | odyssey4me: it wont be that much slower because oslo.concurrency doesn't compile anything | 22:08 |
| mordred | odyssey4me: ok. so - just to be clear - openstack does not support old pip _at_all_ | 22:08 |
| clarkb | odyssey4me: your big win with these wheels is on stuff that needs to link to other libs | 22:08 |
| clarkb | lxml etc | 22:08 |
| fungi | yeah, wheeling oslo.concurrency should be very fast, especially if all its other requirements are available as wheels | 22:08 |
| mordred | odyssey4me: so whatever your policy is on upgrading pip you may want to rethink otherwise you are setting yourself up to be screwed in an unsupported way | 22:08 |
| mordred | as much as I understand why your policy probablyis the way it is | 22:08 |
| openstackgerrit | Carl Baldwin proposed openstack-infra/irc-meetings: A meeting for discussing and coordinating routed networks https://review.openstack.org/278114 | 22:08 |
| *** sridhar_ram has joined #openstack-infra | 22:09 | |
| odyssey4me | mordred understood, but pip 8.0 broke us for several days... and made all previous tags unusable... so we had to make a policy call for stable branches | 22:09 |
| sdague | fungi, et al - https://review.openstack.org/#/c/278089 - will really help prevent everyone recheck grinding nova patches | 22:09 |
| mordred | odyssey4me: totally - believe me - I understand that pain | 22:10 |
| mordred | :) | 22:10 |
| sdague | it looks like there are no nova patches in gate, so even just pushing it into the gate queue will be helpful | 22:10 |
| openstackgerrit | Merged openstack-infra/release-tools: Do not mention "independent" in announcements https://review.openstack.org/277902 | 22:11 |
| odyssey4me | thank you all for your advice and for digging into it | 22:11 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/irc-meetings: Remove PTL office hours https://review.openstack.org/278116 | 22:11 |
| odyssey4me | hopefully bandersnatch can be fixed for pep503 - meanwhile, we'll see what we can do to work with this | 22:11 |
| mordred | odyssey4me: good luck! | 22:12 |
| odyssey4me | I'm trying to get something that works better for gating, but also doesn't implement radical changes to how the playbooks work, otherwise backports will not be possible (and we get stuck with horrible things) | 22:12 |
| clarkb | odyssey4me: what I would expect is for the sdist to be used via the bandersnatch mirror which is likely how everything ese is working now | 22:12 |
| clarkb | can probably find some logs to confirm that from other jobs | 22:12 |
| mordred | right - but if they're on old pip, then it's not going to be doing pre-normalization | 22:12 |
| *** calebb has joined #openstack-infra | 22:13 | |
| mordred | our wheel mirror DOES normalize | 22:13 |
| jeblair | yeah, honestly, using the pip.conf we write is the answer | 22:13 |
| *** burgerk_ has joined #openstack-infra | 22:13 | |
| mordred | it's not | 22:13 |
| mordred | it's not going to fix him | 22:13 |
| mordred | at all | 22:13 |
| jeblair | okay, we've looped back again. mordred: why don't you explain? | 22:13 |
| clarkb | mordred: I thought 7 was new enough to fix the names | 22:13 |
| mordred | the problem is the combo of old pip and a wheel mirror that properly normalizes | 22:13 |
| pleia2 | "I have breakfast/coffee/tea and lunches ordered for everyday. Would you like me to have any coffee/tea brought in the afternoon as a pick me up? Any snacks?" <-- I said yes :) | 22:13 |
| jeblair | okay, so use our pip.conf *and* use our pip. | 22:13 |
| pleia2 | so we should be well fed | 22:13 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job https://review.openstack.org/264991 | 22:13 |
| odyssey4me | clarkb I'll switch the index order now, which hopefully will work - otherwise I'll work out another plan. I'm thinking that we may have to introduce the option of multiple fallbacks or something - then the pip.conf from the host can be used | 22:13 |
| jeblair | mordred: yeah? | 22:14 |
| dstufft | well | 22:14 |
| mordred | so - I honestly do not think the pip.conf is a problem - the code he's got is going to wind up with a pip.conf that is identical (he's already tested inverting the order of the parameters) | 22:14 |
| dstufft | pip < 8 does the fallback to /simple/ | 22:14 |
| mordred | that he's only getting the _tarball_ | 22:14 |
| jeblair | mordred: no, this is something we need to be very clear on -- jobs _must_ use our pip.conf | 22:14 |
| *** thorst has quit IRC | 22:14 | |
| mordred | jeblair: I understand that | 22:14 |
| mordred | however | 22:14 |
| dstufft | so if the /simple/ index is correctly handled an older pip with the wrong normalization rules will work ok | 22:15 |
| jeblair | mordred: we need to be able to change that without having to politely ask every project to please update their scripts | 22:15 |
| clarkb | http://logs.openstack.org/41/277941/4/check/gate-dsvm-nodepool/5d87d90/logs/devstacklog.txt.gz#_2016-02-09_21_28_01_816 the other jobs pick up the wheel from the bandersnatch mirror | 22:15 |
| mordred | I'm saying that that is not the actual techincal problem in this specific case | 22:15 |
| mordred | so while we can work on getting him to do that | 22:15 |
| *** nadya has joined #openstack-infra | 22:15 | |
| jeblair | mordred: right, you're saying it's the older pip | 22:15 |
| mordred | implying that it will solve his problem is not correct | 22:15 |
| *** thorst has joined #openstack-infra | 22:15 | |
| mordred | I do think he should do that | 22:15 |
| odyssey4me | well, the problem is also that we're not building from head every time - we're doing sha updates on a schedule in order to help manage from being side-tracked - we test at a sha for 2 weeks, then tag, then bump sha's and go again | 22:15 |
| jeblair | mordred: i understand that which is why i'm trying to work with you on evolving a response | 22:15 |
| *** nadya is now known as Guest17394 | 22:15 | |
| odyssey4me | so the wheels we want are from that sha's upper-constraints, not from last night's upper-constraints | 22:15 |
| fungi | sdague: i went ahead and pushed it into the gate pipeline | 22:16 |
| sdague | fungi: thank you | 22:16 |
| mordred | ah | 22:16 |
| fungi | sdague: most of the changes ahead of it are probably about to report, so i'm hesitant to promote it | 22:16 |
| sdague | fungi: sure, and none of them are nova changes | 22:16 |
| *** burgerk has quit IRC | 22:16 | |
| *** thorst_ has joined #openstack-infra | 22:16 | |
| fungi | that too | 22:16 |
| sdague | which means they won't fail out in front of us | 22:17 |
| sdague | at least not for this reason | 22:17 |
| sdague | end of gate is fine in this case | 22:17 |
| clarkb | and that path is completely not normalized | 22:17 |
| odyssey4me | and jeblair my very first attempt was based on grabbing the values from /etc/pip.conf on the host: https://review.openstack.org/#/c/265867/1/scripts/gate-check-commit.sh | 22:17 |
| clarkb | so I would expect this to just work if the index url is for bandersnatch mirror | 22:17 |
| fungi | but mainly in the interest of shaving off an extra ~hour of wasted nodes from nova devs recheck-spamming their failing patches because we'll never be able to convince them to not do that | 22:17 |
| *** burgerk_ has quit IRC | 22:17 | |
| jeblair | odyssey4me: right, there's a pip version issue involved here, i stand corrected | 22:17 |
| odyssey4me | and I will return to that, instead of using the current mechanism - but I can only refine the method once I have a pypi mirror and wheel mirror that works for our needs :) | 22:18 |
| *** camunoz has joined #openstack-infra | 22:18 | |
| clarkb | I am almost positive that this will work if index-url is set to bandersnatch | 22:18 |
| clarkb | because oslo.concurrency == oslo.concurrency which is the path in use there and there is a wheel published | 22:19 |
| odyssey4me | yep, I'm going to kick that off now as a test clarkb :) | 22:19 |
| *** achanda has quit IRC | 22:19 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: networking-ovn: Add an install job for networking-ovn + kuryr https://review.openstack.org/278101 | 22:19 |
| *** EricGonczer_ has quit IRC | 22:19 | |
| *** thorst has quit IRC | 22:19 | |
| *** Guest17394 has quit IRC | 22:19 | |
| *** achanda has joined #openstack-infra | 22:21 | |
| *** mikelk has quit IRC | 22:21 | |
| clarkb | our wheel mirror doesnt have its own /simple/ | 22:21 |
| clarkb | but since the bandersnatch mirror wasn't the main index it wasn't used, the wheel waas? | 22:21 |
| *** thorst_ has quit IRC | 22:21 | |
| clarkb | anyways I think that is what confused it there | 22:21 |
| fungi | that makes sense | 22:24 |
| *** HeOS__ has quit IRC | 22:24 | |
| *** HeOS_ has joined #openstack-infra | 22:24 | |
| jeblair | odyssey4me: i don't believe we plan to quickly delete wheels from our mirrors as requirements update; however, i will not promise that they will stay there. we may chose to delete them a few hours after they are no longer relevant in any requirements branch. or we may rebuild the mirror from scratch. | 22:25 |
| *** rockyg has quit IRC | 22:25 | |
| clarkb | however the bandersnatch mirrors should be complete, but that may mean falling back on sdists | 22:25 |
| jeblair | odyssey4me: so, on your long-term map, i think there are 3 divergences which we should try to correct: 1) the config files [well under way]; 2) the version of pip; 3) the requirements versions | 22:25 |
| odyssey4me | clarkb you're right - actually I think the wheel mirror needs to be fed into a --find-links CLI parameter, not an index | 22:26 |
| lifeless | sigmavirus24: so your bug: I think you should have enough to start on a patch ? | 22:26 |
| jeblair | odyssey4me: things 2 and perhaps especially 3, may be longer term conversations with lots of people on what exactly you're solving, and how to deal with that in the systems we've developed as a project | 22:26 |
| *** FallenPegasus has joined #openstack-infra | 22:26 | |
| clarkb | odyssey4me: it is working fine for us as an extra index | 22:26 |
| fungi | clarkb: jeblair: looks like osic went ahead and gave us 100 floating ips quota | 22:26 |
| anteaya | wooot | 22:27 |
| sigmavirus24 | lifeless: yeah, I'll need to stick my head into PBR though | 22:27 |
| clarkb | oh fun I will have to pick up my testing there then | 22:27 |
| jeblair | fungi: good start! ;) | 22:27 |
| odyssey4me | clarkb ok, there goes that idea :p | 22:27 |
| *** rhallisey has quit IRC | 22:27 | |
| lifeless | sigmavirus24: ayup | 22:27 |
| odyssey4me | jeblair I'd be very happy to discuss how we can both serve each other's needs better at any time. It might be good for us to set something up at the summit, but I'd be happy to converse earlier. | 22:28 |
| *** abregman|afk is now known as abregman | 22:29 | |
| openstackgerrit | Volodymyr Samotiy proposed openstack-infra/project-config: Add openstack/broadview-lib project https://review.openstack.org/275873 | 22:29 |
| *** miqui has joined #openstack-infra | 22:30 | |
| odyssey4me | I do think it would be best for us to give you a clearer idea of what we're doing, because I think that'll help you advise us about how we can perhaps move towards something better that can suit both projects. | 22:30 |
| jeblair | odyssey4me: agreed -- with the exception that we're one project. :) | 22:30 |
| jeblair | odyssey4me: welcome to openstack :) | 22:30 |
| odyssey4me | jeblair of course :) | 22:31 |
| *** julim has quit IRC | 22:32 | |
| odyssey4me | jeblair in some ways I'd rather we figure something out before the summit - anything that isn't already in place in mitaka is going to live in a stable branch for another year (or more)... so the sooner the better :) | 22:32 |
| odyssey4me | I'd rather get good habits into Liberty and Mitaka, and drop bad habits when Kilo dies in a fire. | 22:33 |
| dstufft | https://github.com/pypa/packaging/pull/60 | 22:33 |
| *** esker has joined #openstack-infra | 22:34 | |
| odyssey4me | afk for a bit | 22:36 |
| jeblair | greghaynes: the traceback in https://review.openstack.org/#/c/273703/9 looks interleaved? | 22:38 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for nova wsgi unit test eventlet 0.18.2 bug 1543766 https://review.openstack.org/278069 | 22:38 |
| openstack | bug 1543766 in OpenStack Compute (nova) liberty "nova.tests.unit.test_wsgi.TestWSGIServerWithSSL fails with testtools.matchers._impl.MismatchError: 'OK\r\n' != 'PONG' and eventlet 0.18.2" [High,In progress] https://launchpad.net/bugs/1543766 - Assigned to Matt Riedemann (mriedem) | 22:38 |
| jeblair | greghaynes: Thread.__target appears to be called inside of "_sleep(delay)" which i highly doubt | 22:38 |
| *** thorst has joined #openstack-infra | 22:39 | |
| greghaynes | jeblair: oy | 22:39 |
| greghaynes | jeblair: I have some code I can repro it again with, it just takes a while | 22:39 |
| *** dims_ has joined #openstack-infra | 22:39 | |
| jeblair | greghaynes: maybe it was a copy/paste error? | 22:39 |
| greghaynes | jeblair: I did confirm that submitJob is blocking forever though, which is when I went on that debugging | 22:39 |
| greghaynes | jeblair: likely | 22:39 |
| greghaynes | jeblair: the thread dump was huge so I tried to grab the good parts | 22:39 |
| jeblair | greghaynes: it looks like two complete copies of the same traceback; one stuck inside of the other | 22:40 |
| jeblair | greghaynes: so i think i can un-interleave it | 22:40 |
| *** salv-orl_ has joined #openstack-infra | 22:40 | |
| jeblair | greghaynes: this looks like the actual thing: http://paste.openstack.org/show/486495/ does that look right to you? | 22:40 |
| *** dims has quit IRC | 22:40 | |
| greghaynes | jeblair: yep | 22:41 |
| prometheanfire | greghaynes: mind taking a look at this 4 line patch? https://review.openstack.org/#/c/276578/ | 22:41 |
| jeblair | greghaynes: cool, i'll go with that, thanks | 22:41 |
| *** gibi_ has quit IRC | 22:41 | |
| prometheanfire | and this 2 line one (with file rename) https://review.openstack.org/#/c/276581/ | 22:41 |
| prometheanfire | ianw: around? | 22:41 |
| greghaynes | prometheanfire: yea, ill have a look | 22:41 |
| *** gibi has joined #openstack-infra | 22:41 | |
| prometheanfire | thanks | 22:42 |
| ianw | prometheanfire: yep | 22:42 |
| clarkb | woot my change for upload fails passed one of the two unitest jobs. Looks like unrelated failure in the cover job but I am investigating | 22:42 |
| clarkb | jeblair: greghaynes ^ | 22:42 |
| prometheanfire | mordred: is stuff 'stable' enough for a new glean tag? | 22:42 |
| *** thorst has quit IRC | 22:43 | |
| greghaynes | clarkb: fun | 22:43 |
| *** salv-orlando has quit IRC | 22:43 | |
| prometheanfire | clarkb: yay having to fix unrelated errors | 22:43 |
| ianw | promethanfire: we were just discussing this a bit ago and i think mordred is going to tag a 1.5 | 22:44 |
| prometheanfire | ah, nice | 22:44 |
| mordred | oh - yeah | 22:44 |
| mordred | I've got it ready to push - just didn't want to do it while I was in meetings | 22:44 |
| mordred | done | 22:44 |
| prometheanfire | yay, don't have to override to git anymore | 22:44 |
| ianw | greghaynes: did you write the growroot stuff | 22:44 |
| greghaynes | ianw: I did | 22:45 |
| greghaynes | ianw: should I be scared? | 22:45 |
| greghaynes | ianw: I make no claim that it is good | 22:45 |
| ianw | greghaynes: yes, because i am about to mention systemd | 22:45 |
| prometheanfire | LOL | 22:45 |
| prometheanfire | greghaynes: I'm sorry | 22:45 |
| *** achanda has quit IRC | 22:45 | |
| *** esker has quit IRC | 22:46 | |
| ianw | greghaynes: because when we run it under systemd at boot, ID_PART_ENTRY_NUMBER isn't yet showing up in udevadm | 22:46 |
| greghaynes | ianw: fun. Is it just running too early? | 22:46 |
| greghaynes | ianw: that can run really late... | 22:46 |
| prometheanfire | it can run at any time really | 22:46 |
| ianw | greghaynes: so i think i've just done it wrong -- what i think needs to happen is it needs a udev rule that matches on added block devices | 22:47 |
| *** Sukhdev has quit IRC | 22:47 | |
| ianw | and, if the block device is the root one, and it's not embiggened, embiggen it | 22:47 |
| *** gibi has quit IRC | 22:47 | |
| *** FallenPegasus has quit IRC | 22:47 | |
| *** tzn has quit IRC | 22:47 | |
| greghaynes | ianw: hrm, it needs to run after that device has been mounted too, though (how else does it know that its the root device) | 22:48 |
| *** gibi has joined #openstack-infra | 22:48 | |
| ianw | greghaynes: hmm, i *think* if the device rule then has a "wants" or similar on multi-user, it should be at the right point ... | 22:49 |
| *** pradk has quit IRC | 22:49 | |
| greghaynes | ianw: is there no systemd thing for 'run after all local filesystems are mounted'? | 22:50 |
| greghaynes | because really thats what i'd use | 22:50 |
| *** baoli has quit IRC | 22:50 | |
| ianw | greghaynes: it's not the mounting, it's the udev triggers that add all those flags being probed for | 22:50 |
| greghaynes | Yea, but those have to be before the mount | 22:50 |
| greghaynes | right? | 22:51 |
| ianw | greghaynes: i don't think so, it's looking at all this /dev/disk/by-uuid/... stuff that is setup by udev rules | 22:52 |
| clarkb | http://logs.openstack.org/41/277941/4/check/nodepool-coverage/94b508c/console.html#_2016-02-09_22_31_35_994 it looks like dropping the db hung and timed out | 22:52 |
| clarkb | greghaynes: that is a known issue with cleanups right? | 22:52 |
| greghaynes | ianw: aye, ok | 22:52 |
| *** esker has joined #openstack-infra | 22:52 | |
| *** sridhar_ram1 has joined #openstack-infra | 22:53 | |
| *** sridhar_ram has quit IRC | 22:53 | |
| greghaynes | clarkb: yep | 22:53 |
| greghaynes | ianw: so ya, if you can express that somehow in systemd then SGTM, I lack the fu to do that. | 22:53 |
| *** tiswanso has quit IRC | 22:54 | |
| clarkb | greghaynes: you should watch kung fu panda and learn :) | 22:54 |
| *** annegentle_ has joined #openstack-infra | 22:54 | |
| *** amitgandhinz has quit IRC | 22:54 | |
| greghaynes | clarkb: Is that what I'm missing? | 22:54 |
| clarkb | yes, this is why google gifted you a copy | 22:55 |
| clarkb | or was it amazon? | 22:55 |
| *** baoli has joined #openstack-infra | 22:55 | |
| greghaynes | hah, amazon | 22:55 |
| olaph | greghaynes is the dragon warrior? | 22:56 |
| greghaynes | haha | 22:56 |
| *** tongli has quit IRC | 22:56 | |
| jeblair | greghaynes: do you have any information from the gearman server side? | 22:57 |
| greghaynes | jeblair: I do not - that was in a test case | 22:57 |
| *** Sukhdev has joined #openstack-infra | 22:57 | |
| *** dingyichen has joined #openstack-infra | 22:57 | |
| greghaynes | jeblair: I hacked the thread dumping in to the base class | 22:57 |
| jeblair | greghaynes: it does have a timeout | 22:57 |
| greghaynes | jeblair: so I have gear logs, I can repro it in a little bit and paste logs if that is better | 22:57 |
| *** dimtruck is now known as zz_dimtruck | 22:57 | |
| jeblair | greghaynes: we have set it to be very high though | 22:57 |
| jeblair | greghaynes: higher than the timeout for running tests | 22:58 |
| greghaynes | jeblair: ah | 22:58 |
| greghaynes | jeblair: so that is it, then | 22:58 |
| jeblair | greghaynes: so i do not believe that is a deadlock | 22:58 |
| *** rcernin has quit IRC | 22:58 | |
| *** Sukhdev has quit IRC | 22:58 | |
| greghaynes | jeblair: Is there no way gear client can discover that the socket reconnected or disconnected and kill the wait()? | 22:58 |
| jeblair | greghaynes: it's the same timeout bump we made to avoid disconnects | 22:58 |
| *** Sukhdev has joined #openstack-infra | 22:59 | |
| jeblair | greghaynes: perhaps... let me see | 23:00 |
| *** jaypipes has quit IRC | 23:00 | |
| *** Guest88685 has quit IRC | 23:00 | |
| *** thorst has joined #openstack-infra | 23:00 | |
| jeblair | greghaynes: it's not clear to me what actually happened though | 23:01 |
| *** Sukhdev has quit IRC | 23:01 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Set conditions for gate-{name}-docs on freezer* https://review.openstack.org/278070 | 23:02 |
| jeblair | greghaynes: because it should not have tried to submit the job to the old disconnected connection | 23:02 |
| jeblair | greghaynes: that is marked as invalid before jobs are notified | 23:02 |
| greghaynes | jeblair: Yea, it only happens about 1 of 10 test runs for me locally | 23:02 |
| greghaynes | jeblair: Let me repro it and get a better dump | 23:03 |
| jeblair | greghaynes: so the fact that nodepool's handleDisconnect was called means that the connection was taken out of the client's list of active connections | 23:03 |
| *** daemontool__ has joined #openstack-infra | 23:03 | |
| jeblair | greghaynes: so it would have been submitted to a replacement connection | 23:03 |
| jeblair | greghaynes: why that one didn't respond i don't know | 23:03 |
| *** eharney has quit IRC | 23:03 | |
| jeblair | greghaynes: (or if we can confirm it somehow was submitted to the old connection, that would be a bug) | 23:03 |
| *** Sukhdev has joined #openstack-infra | 23:04 | |
| greghaynes | jeblair: http://paste.ubuntu.com/15005243/ | 23:04 |
| *** daemontool_ has quit IRC | 23:04 | |
| jeblair | greghaynes: when you said a while, i was imagining more than 1 minute | 23:05 |
| greghaynes | jeblair: usually I have to run it in a while loop and go do something else | 23:05 |
| greghaynes | I got lucky | 23:06 |
| jeblair | greghaynes: are you just running that one test? | 23:06 |
| greghaynes | jeblair: yep | 23:06 |
| jeblair | greghaynes: (i have run it a few times and have not hit it yet) | 23:06 |
| dims_ | clarkb : +1 on kung fu panda :) | 23:06 |
| greghaynes | jeblair: I left the command at the top | 23:06 |
| jeblair | greghaynes: ah, yep, that's what i'm running | 23:07 |
| jeblair | greghaynes: your workstation probably has the magic timing | 23:07 |
| greghaynes | jeblair: Yep | 23:07 |
| greghaynes | jeblair: It seems to happen when nodepool reconnects to gear at about the same time we send the submitJob | 23:07 |
| greghaynes | If you look on line 204 | 23:08 |
| greghaynes | jeblair: I have to go afk for a bit, ty for having a look | 23:08 |
| jeblair | greghaynes: np,ty | 23:09 |
| *** Sukhdev has quit IRC | 23:09 | |
| *** thorst has quit IRC | 23:09 | |
| *** Sukhdev has joined #openstack-infra | 23:09 | |
| *** thorst has joined #openstack-infra | 23:09 | |
| *** xyang1 has quit IRC | 23:09 | |
| *** Sukhdev has quit IRC | 23:10 | |
| *** Sukhdev has joined #openstack-infra | 23:10 | |
| clarkb | mordred: educate me, what would you expect to lock up a drop database in mysql resulting in a job timeout | 23:10 |
| clarkb | mordred: something with an open transaction? | 23:11 |
| *** Sukhdev has quit IRC | 23:12 | |
| mordred | hrm | 23:12 |
| *** jtomasek_ has quit IRC | 23:12 | |
| mordred | yah - maybe so | 23:12 |
| *** kzaitsev_mb has quit IRC | 23:13 | |
| clarkb | mordred: nodepool tests occasionally timeout and they seem to be stuck on the drop database cleanup for the db fixture | 23:13 |
| *** Sukhdev has joined #openstack-infra | 23:13 | |
| *** thorst has quit IRC | 23:14 | |
| *** camunoz has quit IRC | 23:14 | |
| *** jpr has quit IRC | 23:15 | |
| *** Jeffrey4l has joined #openstack-infra | 23:15 | |
| *** nadya has joined #openstack-infra | 23:16 | |
| *** nadya is now known as Guest31937 | 23:17 | |
| mordred | oh lovely | 23:17 |
| clarkb | we create a new db per test | 23:17 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: use the name we derive from the project instead of the path https://review.openstack.org/278143 | 23:17 |
| clarkb | so I don't think this is multiple tests interfering with each other | 23:17 |
| clarkb | we must have some background thread that is not dying holding a transaction? | 23:17 |
| *** gokrokve has joined #openstack-infra | 23:19 | |
| *** daemontool_ has joined #openstack-infra | 23:20 | |
| *** baoli has quit IRC | 23:20 | |
| *** Guest31937 has quit IRC | 23:21 | |
| *** yamamoto_ has joined #openstack-infra | 23:21 | |
| *** mriedem has quit IRC | 23:22 | |
| *** daemontool__ has quit IRC | 23:22 | |
| *** yamamoto_ has quit IRC | 23:23 | |
| *** gokrokve has quit IRC | 23:23 | |
| fungi | if rvasilets and redixin had a problem with the speed of reviews in project-config, it would have been nice of them to pitch in rather than ranting like in https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/8835 | 23:24 |
| fungi | but it seems they're more interested in doing their own thing rather than participating | 23:24 |
| fungi | this disappoints me | 23:24 |
| *** hashar has joined #openstack-infra | 23:25 | |
| * Clint fixates on "numeros" | 23:25 | |
| prometheanfire | greghaynes: how goes the reviews :P | 23:25 |
| *** zz_dimtruck is now known as dimtruck | 23:25 | |
| fungi | anybody recall if they ever voiced concerns or had suggestions which didn't involve replacing what we have with something they wrote from scratch? | 23:25 |
| fungi | not really sure how seriously to take this | 23:26 |
| *** abregman is now known as abregman|ZzZzZz | 23:26 | |
| clarkb | fungi: not really, when rally ci was first proposed I tried to suss out the motivations behind it | 23:26 |
| clarkb | I distinctly remember "simpler config" and not using java | 23:26 |
| anteaya | fungi: I don't recall a serious prposal or spec that was offered | 23:26 |
| fungi | aside from the fact that most of the highlights there are slated to be solved with zuul v3 | 23:26 |
| anteaya | nor to I recall a meeting item agenda on the topic | 23:27 |
| jeblair | yeah, considering that 20 people raised their hands for helping on zuulv3 at the summit, and 3 raised their hands when it was actually time to start coding | 23:27 |
| jeblair | if someone is seriously interested in hacking on ci stuff, we certainly have room for them to do so | 23:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Add poll.poll() timeout https://review.openstack.org/274445 | 23:27 |
| *** camunoz has joined #openstack-infra | 23:27 | |
| *** daemontool_ has quit IRC | 23:28 | |
| dims_ | boris-42 : around? what's up with rally-ci? ^^^ | 23:28 |
| clarkb | mordred: what is the best way to display the open transactions? show engine innodb status? | 23:28 |
| clarkb | mordred: I think I will add that to our cleanup | 23:28 |
| clarkb | mordred: then next time it happens we can review the state of the owrld | 23:28 |
| jeblair | i am also very surprised to hear that the doors of openstack ci are all closed. | 23:28 |
| mordred | I have nothing productive to add on the topic of rally ci | 23:29 |
| *** daemontool has joined #openstack-infra | 23:29 | |
| clarkb | mordred: wha about on the topic of mysql? :P | 23:29 |
| mordred | but I might decide, should this talk get accepted, to go mercilessly heckle | 23:29 |
| anteaya | mordred: your time is better spent elsewhere, being more productive | 23:29 |
| jeblair | i will entertain the idea that our doors are too open. | 23:29 |
| mordred | clarkb: not much useful there either - I think your innodb status idea is a good one | 23:29 |
| mordred | jeblair: ++ | 23:29 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool: Add 'image-api-use-tasks' provider entry for shade https://review.openstack.org/272097 | 23:29 |
| anteaya | jeblair: I can support that argument | 23:30 |
| *** Sukhdev has quit IRC | 23:30 | |
| *** amrith is now known as _amrith_ | 23:30 | |
| *** abregman|ZzZzZz has quit IRC | 23:31 | |
| anteaya | fungi: are there any good propsals you have come across? | 23:31 |
| clarkb | Access denied; you need the PROCESS privilege for this operation hrm | 23:31 |
| *** Sukhdev has joined #openstack-infra | 23:31 | |
| clarkb | thta was wit a root connection, I worry that the test user may not hvae perms, but Iwill test that explicitly | 23:32 |
| anteaya | clarkb: is this oisc | 23:32 |
| anteaya | or raxtel? | 23:32 |
| *** Sukhdev has quit IRC | 23:32 | |
| clarkb | neither, debugging nodepool test failures | 23:33 |
| clarkb | it does work with the test user so will add that to nodepool and recheck a bunch | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Show innodb status before dropping db https://review.openstack.org/278146 | 23:36 |
| clarkb | mordred: ^ we will see if that gets us somewhere | 23:36 |
| clarkb | and I derped the copy pasta again | 23:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Show innodb status before dropping db https://review.openstack.org/278146 | 23:37 |
| anteaya | ah | 23:38 |
| anteaya | there are some good proposals | 23:39 |
| *** gordc has quit IRC | 23:39 | |
| jeblair | greghaynes: it looks like the server enqueued a response, but the client never got it | 23:39 |
| *** tzn has joined #openstack-infra | 23:40 | |
| *** annegentle_ has quit IRC | 23:40 | |
| jeblair | greghaynes: that could be because the server never actually sent it, or the client didn't get notified correctly, or there was some weird network error (seems unlikely over loopback). | 23:40 |
| jeblair | greghaynes: but lines 206 and 209 look like the the server received, processed, and responded to the request | 23:41 |
| anteaya | fungi: some feedback on the vote-for-speakers app, if I select Other at the bottom of the drop down list, I get no content as a response | 23:42 |
| *** EricGonczer_ has joined #openstack-infra | 23:45 | |
| fungi | anteaya: 7337 and 7578 of course but 8020,7155,8611,8338,8396,7618,7542,8081,7634,7334,7935 are pretty infra-heavy topics in the upstream dev track too (and 6956 but it looks like it's a clueless talking head), 8087 and 8013 are stable branch related, 7813 is qa-ish (on static analyzers), | 23:46 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove PTL office hours https://review.openstack.org/278116 | 23:46 |
| *** Sukhdev has joined #openstack-infra | 23:47 | |
| anteaya | fungi: awesome, thank you for the list | 23:47 |
| fungi | anteaya: in summary, i don't think we have a shortage of openstack infra/ci talks proposed | 23:47 |
| anteaya | agreed we have no shortage | 23:47 |
| fungi | if anything, we're in danger of dominating the submissions in that track | 23:47 |
| *** annegentle_ has joined #openstack-infra | 23:47 | |
| anteaya | my concern is what the listener will expect from us as a result of hearing all the talks | 23:47 |
| anteaya | so I hope the selections committee for the upstream development track has some good conversations curating that track | 23:48 |
| fungi | anteaya: for the ones which are specifically about the systems/services we're developing and maintaining, i'm pretty confident we'll manage to collectively present an accurate and consistent picture | 23:48 |
| anteaya | yay! | 23:49 |
| fungi | since the people proposing those are for the most part actively engaged in the team | 23:49 |
| anteaya | oh good | 23:49 |
| anteaya | yes I see, many good proposals | 23:49 |
| anteaya | there are more than a few third party ci proposals too | 23:49 |
| fungi | indeed there are | 23:49 |
| fungi | popular topic | 23:49 |
| anteaya | hope they set reasonable expectations for operators | 23:49 |
| odyssey4me | clarkb it seems that your suspicion was right - the repo build completed and used both sources... the whole build failed, so I'll figure out why tomorrow - thanks for your help! | 23:49 |
| anteaya | and operator's managers | 23:49 |
| pleia2 | wow, so many talks | 23:50 |
| anteaya | so many | 23:50 |
| *** yamamoto_ has joined #openstack-infra | 23:50 | |
| fungi | i'm pleasantly surprised by the overall quality of proposals in that track | 23:50 |
| pleia2 | yeah, it was a great idea | 23:50 |
| *** eil397 has joined #openstack-infra | 23:50 | |
| fungi | the ones i'm unsurprisingly disappointed in will likely get nixed by the track chairs anyway regardless of votes | 23:51 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:51 | |
| anteaya | it is up to the selection committee for each track how much weight they give to votes | 23:52 |
| fungi | though the process is slightly altered this time | 23:52 |
| anteaya | when I chaired it was easy to spot the voting that was obviously gamed since they had 10x more than the other submissions | 23:52 |
| anteaya | oh? | 23:52 |
| anteaya | chairs can still spot gamed voting, yes? | 23:52 |
| anteaya | and make decisions accordingly? | 23:53 |
| pleia2 | yep | 23:53 |
| fungi | chairs got a chance to do a first pass pre-vote to horse trade or otherwise shuffle poorly-targeted proposals prior to opening voting | 23:53 |
| anteaya | great | 23:53 |
| anteaya | oh yeah, I remember that bit | 23:53 |
| fungi | so hopefully the voting will also be slightly more useful to chairs than in previous iterations | 23:53 |
| pleia2 | voting is a useful bit of criteria for figuring out what people want to see | 23:53 |
| *** shashank_hegde has quit IRC | 23:53 | |
| anteaya | yes, once the gamed voting ones are removed or accounted for | 23:54 |
| *** austin81 has left #openstack-infra | 23:54 | |
| pleia2 | quality, accuracy, expertise, etc is what the track leads get to figure out | 23:54 |
| anteaya | yes | 23:54 |
| *** c-jm has joined #openstack-infra | 23:54 | |
| *** Sukhdev has quit IRC | 23:54 | |
| fungi | and, to some extent, trustworthiness of the numbers from the vote | 23:54 |
| anteaya | right | 23:54 |
| jeblair | greghaynes: got it | 23:56 |
| jeblair | greghaynes: of the choices i listed above, the relevant one is "the client didn't get notified correctly" | 23:56 |
| jeblair | greghaynes, clarkb: and the answer is right inside the traceback here: http://paste.openstack.org/show/486495/ | 23:57 |
| *** hashar has quit IRC | 23:57 | |
| jeblair | greghaynes, clarkb: you are submitting a job to gearman from within a callback from gear notifying you that the gearman server has disconnected. that callback is in the thread that processes input and output. | 23:57 |
| openstackgerrit | Russell Sim proposed openstack-infra/gerritlib: Implemented add and remove reviewer functions https://review.openstack.org/278154 | 23:57 |
| *** sabeen1 has quit IRC | 23:58 | |
| jeblair | greghaynes, clarkb: so it's not going to process the server response until the callback returns, but the callback is waiting for acklowledgement of the submit_job | 23:58 |
| *** annegentle_ has quit IRC | 23:59 | |
| greghaynes | jeblair: oh! | 23:59 |
| jeblair | greghaynes, clarkb: it would be a deadlock except for the timeout | 23:59 |
| greghaynes | jeblair: wow | 23:59 |
| jeblair | greghaynes, clarkb: so probably, best not to submit gearman jobs from inside gear callbacks :) | 23:59 |
| clarkb | I guess so | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!