| *** thuc has joined #openstack-infra | 00:00 | |
| *** thuc_ has joined #openstack-infra | 00:01 | |
| *** mrda has quit IRC | 00:01 | |
| *** thuc_ has quit IRC | 00:02 | |
| harlowja | hmmm, did someone kill wiki.openstack.org? | 00:02 |
|---|---|---|
| *** thuc_ has joined #openstack-infra | 00:02 | |
| harlowja | (Cannot contact the database server) :-/ | 00:02 |
| fungi | harlowja: mysql update--apache probably needs a restart there | 00:03 |
| fungi | fixing | 00:03 |
| *** thuc_ has quit IRC | 00:03 | |
| harlowja | cool | 00:03 |
| *** thuc_ has joined #openstack-infra | 00:03 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add cross-project unit tests for oslo libraries https://review.openstack.org/85487 | 00:03 |
| *** mrda has joined #openstack-infra | 00:03 | |
| *** timrc is now known as timrc-afk | 00:04 | |
| dhellmann | fungi, jeblair : that version includes a tool for checking the configuration ^^ (it shows that I need to add more jobs) | 00:04 |
| *** thuc_ has quit IRC | 00:04 | |
| *** thuc has quit IRC | 00:04 | |
| *** thuc has joined #openstack-infra | 00:05 | |
| *** jergerber has joined #openstack-infra | 00:09 | |
| *** jergerber has quit IRC | 00:09 | |
| *** nati_uen_ has joined #openstack-infra | 00:11 | |
| *** nati_ueno has quit IRC | 00:12 | |
| *** nati_ueno has joined #openstack-infra | 00:12 | |
| *** UtahDave has quit IRC | 00:15 | |
| *** nati_uen_ has quit IRC | 00:16 | |
| *** msabramo has joined #openstack-infra | 00:18 | |
| *** SumitNaiksatam has quit IRC | 00:18 | |
| jeblair | #status alert All of the project infrastructure hosts are being restarted for security updates. | 00:20 |
| openstackstatus | jeblair: sending alert | 00:20 |
| clarkb | I guess that is my go ahead? | 00:20 |
| * clarkb reboots logstash-worker nodes | 00:21 | |
| openstackstatus | NOTICE: All of the project infrastructure hosts are being restarted for security updates. | 00:21 |
| *** ChanServ changes topic to "All of the project infrastructure hosts are being restarted for security updates." | 00:21 | |
| fungi | should i take care in rebooting the gerrit and jenkins servers? we need to put zuul on hold first? | 00:21 |
| openstackstatus | jeblair: finished sending alert | 00:22 |
| jeblair | fungi: how about i save the zuul queue status | 00:22 |
| fungi | holding | 00:22 |
| *** weshay has joined #openstack-infra | 00:22 | |
| jeblair | mordred: please don't reboot eavesdrop | 00:22 |
| jeblair | fungi: done, all clear | 00:23 |
| fungi | rebooting the world | 00:24 |
| *** openstackgerrit has quit IRC | 00:24 | |
| mordred | jeblair: kk | 00:26 |
| clarkb | elasticsearch cluster is yellow | 00:27 |
| clarkb | and recovering | 00:27 |
| fungi | note, regenerating the gerrit ssh api/git host key will be potentially disruptive for all devs | 00:28 |
| fungi | we might want to consider that one an acceptible risk | 00:28 |
| clarkb | all worker nodes were kicked and had A-C worker processes started | 00:28 |
| mordred | fungi: noted | 00:28 |
| *** amotoki has joined #openstack-infra | 00:29 | |
| *** nosnos has joined #openstack-infra | 00:29 | |
| clarkb | I am not seeing any jobs queue on logstash.o.o but I am not expecting taht to be a real issue right now (we must be waiting for jenkins to do stuff) | 00:29 |
| jeblair | fungi: is gerrit back up? | 00:30 |
| fungi | jenkins07 is back up and responding... and that was the last one i rebooted so checking the others now | 00:30 |
| jeblair | (asking if i should re-enqueue the zuul changes now) | 00:30 |
| fungi | gerrit does not seem to be yet | 00:30 |
| fungi | checking it's tonsils now | 00:30 |
| fungi | ssh: connect to host review port 22: Connection refused | 00:30 |
| fungi | quick, to the rackspace | 00:30 |
| clarkb | is it fscking? | 00:30 |
| fungi | entirely likely | 00:31 |
| jeblair | i had 2 nodes also not come back up yet, the others did | 00:31 |
| mordred | fungi: I've got the same for graphite and groups | 00:31 |
| *** mbacchi has quit IRC | 00:31 | |
| mordred | and cacti | 00:31 |
| mordred | the others are rebooted | 00:31 |
| jeblair | oh, one of them just came back up, so yeah, fsck seems likely | 00:31 |
| NobodyCam | I assume review.openstack.org is rebooting too? | 00:32 |
| jeblair | yes | 00:32 |
| NobodyCam | :) | 00:32 |
| fungi | i think a recent browser update must have knocked out my ability to use the web-based virtual console in the rackspace dashboard | 00:32 |
| NobodyCam | oh Ty | 00:32 |
| *** aconrad has quit IRC | 00:32 | |
| mordred | awesome | 00:32 |
| fungi | "Could not connect to console in the expected amount of time. Please refresh the page or try again later." | 00:32 |
| jeblair | fungi: nova console-log or whatever it's called work? | 00:32 |
| fungi | finding out! | 00:33 |
| *** khyati_ has quit IRC | 00:34 | |
| fungi | ERROR: There is no such action: os-getConsoleOutput (HTTP 400) (Request-ID: req-35fd729c-7978-4c3a-ac23-fff6848ea7f0) | 00:34 |
| fungi | must not be supported? | 00:34 |
| *** blamar has quit IRC | 00:34 | |
| jeblair | hrm. it is an old server | 00:34 |
| *** blamar has joined #openstack-infra | 00:34 | |
| jeblair | i'll try the web console | 00:35 |
| clarkb | let me know if I should check anything | 00:35 |
| fungi | please do. still refusing ssh and flying blind as to why now | 00:35 |
| mordred | all up execpt for graphite now | 00:36 |
| *** timrc-afk is now known as timrc | 00:36 | |
| fungi | i'm guessing fsck picked up an error and is prompting to enter single-user mode to repair it | 00:36 |
| jeblair | fungi: i think it has a really big filesystem... | 00:36 |
| mordred | graphite is now back up | 00:37 |
| fungi | mordred: please re-stop apache on graphite if you haven't | 00:37 |
| jeblair | Proxy reports "500 Internal Server Error" | 00:37 |
| fungi | whee! | 00:37 |
| mordred | stoppped | 00:37 |
| jeblair | fungi: why is apache stopped on graphite? | 00:37 |
| fungi | oh, graphite | 00:37 |
| mordred | restarting | 00:38 |
| fungi | mordred: sorry about that. meant cacti | 00:38 |
| mordred | stopped on cacti | 00:38 |
| mordred | restated on graphite | 00:38 |
| fungi | too many graphing things | 00:38 |
| mordred | yah | 00:38 |
| jeblair | fungi: review is up | 00:38 |
| fungi | review.o.o finally came up | 00:39 |
| fungi | and is responding on webui | 00:39 |
| fungi | yep | 00:39 |
| fungi | 00:39:27 up 14 min | 00:39 |
| fungi | weird | 00:39 |
| fungi | oh, right, spent that long running fsck | 00:39 |
| fungi | that's time since kernel started, not time since runlevel change | 00:40 |
| fungi | jenkins.o.o is still down too | 00:40 |
| mordred | as we rekey on things and re-enroll in puppet - should we enroll in the salt master too? (two birds, one stone?) | 00:41 |
| mordred | and/or - I bet we should re-CA salt master? | 00:41 |
| *** thuc has quit IRC | 00:41 | |
| *** thuc has joined #openstack-infra | 00:42 | |
| clarkb | mordred: I don't think we should mix things | 00:42 |
| fungi | all the jenkins masters except jenkins.o.o are up and responding again | 00:42 |
| mordred | clarkb: k | 00:43 |
| fungi | jenkins.o.o acts like it's still running a fsck too | 00:43 |
| mordred | clarkb: we should probably re-CA the salt master though and re-kick the things that are connected | 00:43 |
| *** alexpilotti has quit IRC | 00:44 | |
| *** arosen has quit IRC | 00:44 | |
| fungi | potentially good news, a friend says that centos 6.4 runs too old of an openssl to be affected | 00:44 |
| clarkb | fungi: we don't use that version though | 00:44 |
| fungi | oh | 00:44 |
| clarkb | I will confirm | 00:44 |
| fungi | pooples | 00:44 |
| fungi | openssl-1.0.1e-16.el6_5.4.x86_64 | 00:45 |
| fungi | yeah | 00:45 |
| *** thuc has quit IRC | 00:46 | |
| fungi | affected | 00:47 |
| fungi | since we're on 6.5 now | 00:48 |
| clarkb | yup | 00:48 |
| dstufft | centos/rhel finally updated their OpenSSL, just in time for heartbleed | 00:49 |
| dstufft | :| | 00:49 |
| fungi | jenkins.o.o is *still* not back up | 00:49 |
| fungi | dstufft: just wondering when the glacier of centos security updates will catch up to this one | 00:49 |
| dstufft | fungi: I'm in #centos at the moment, we're testing a openssl build on warehouse.python.org I think | 00:50 |
| dstufft | ernest is doing it not me, I'm just observing | 00:50 |
| jeblair | all my hosts are up now | 00:50 |
| dstufft | (ernest is one of the python.org infra team members w/ me) | 00:50 |
| fungi | dstufft: oh, awesome! maybe things are faster there than they used to be | 00:51 |
| clarkb | if centos updates I can take that on, and try to do it with as little impact as possible | 00:51 |
| dstufft | PyPI runs on CentOS | 00:51 |
| dstufft | So I have an invested interest :] | 00:51 |
| fungi | all my hosts are back online *except* jenkins.o.o (next-gen vm but not performance flavor) | 00:52 |
| clarkb | fungi: jeblair: looking into puppet CA stuff now. I think we should kill CA, create new certs for all nodes then copy those out to our nodes | 00:52 |
| clarkb | http://projects.puppetlabs.com/projects/1/wiki/certificates_and_security#Regenerate-Certificates-for-Puppet-Master | 00:53 |
| fungi | still refusing connections, which makes me suspect fsck is running | 00:53 |
| fungi | nova console-log doesn't work on it either | 00:53 |
| jeblair | fungi: i think it is also a big old server | 00:54 |
| fungi | 30gb standard instance, yep | 00:54 |
| fungi | same as review.o.o | 00:55 |
| mordred | clarkb: I support that plan | 00:55 |
| mattoliverau | fungi: can you access the console via the web ui? | 00:55 |
| clarkb | blkperl wrote a blog http://blkperl.github.io/replace-puppet-ca.html | 00:55 |
| clarkb | blkperl: nibalizer whic process is more correct? | 00:55 |
| phschwartz | Do we have a good guide to setting up and working with zuul | 00:55 |
| jeblair | phschwartz: now isn't a good time; we're dealing with the openssl issue | 00:56 |
| fungi | mattoliverau: recently i've been having no luck wit the web comsole--i think something changed with my browser to break it | 00:56 |
| nibalizer | ya blkperls blog | 00:56 |
| fungi | mattoliverau: icedtea plugin used to deal with it fine | 00:56 |
| nibalizer | the puppetlabs wiki is super out of date | 00:56 |
| nibalizer | are you replacing CA? | 00:56 |
| phschwartz | jeblair: sorry I can wait. Just connected to my bot and should have read. | 00:56 |
| nibalizer | or just revoking all the things? | 00:56 |
| fungi | mattoliverau: but i get connection timeouts after the little progress indicator spins for a while | 00:57 |
| *** yamahata has joined #openstack-infra | 00:57 | |
| mattoliverau | fungi: you might have to.. *shudder* install sun/oracle java plugin. | 00:57 |
| clarkb | nibalizer: I think we should go ahead and replace CA | 00:57 |
| nibalizer | clarkb: i agree | 00:57 |
| nibalizer | (as you already know) | 00:57 |
| fungi | mattoliverau: yeah, i don't think i have anywhere safe enough to install that these days | 00:58 |
| nibalizer | you can skip a ton of the steps on blkperls blog because you don't have a separate CA server | 00:58 |
| jeblair | mattoliverau: yeah, i won't be doing that either :( | 00:58 |
| mordred | clarkb: before we kill the old one - puppet cert list -a | grep '^\+' | awk '{print $2}' | sed 's/"//g' - will give us a list of the current hosts | 00:58 |
| mordred | that we've accepted | 00:58 |
| clarkb | mordred: can you put that list in /root somewere? | 00:58 |
| nibalizer | you have shared master/CA server so you'll bea ble to nuke the /var/lib/puppet/ssl dirs on everything, run puppet master on the server, then restart apache on the master, and start running puppet agent -t | 00:58 |
| jeblair | mattoliverau: fortunately, i _think_ the nova console log command works with newer hosts, but these are old ones | 00:58 |
| fungi | mattoliverau: "Could not connect to console in the expected amount of time. Please refresh the page or try again later." | 00:58 |
| mordred | clarkb: it is now in /root/allcerts.txt | 00:59 |
| jeblair | mattoliverau: i got an error 500 from the proxy | 00:59 |
| clarkb | nibalizer: mordred: then I figure we moev /var/lib/puppet/ssl aside just incase we wedge outselves | 00:59 |
| clarkb | then delete it next week or something | 00:59 |
| mordred | clarkb: ++ | 00:59 |
| nibalizer | clarkb: sounds good | 01:00 |
| mattoliverau | jeblair: so is the console not up at all.. that may mean it isn't even fscking | 01:00 |
| jeblair | i restarted nodepool again because it wasn't emitting statsd | 01:00 |
| *** timrc is now known as timrc-afk | 01:00 | |
| mattoliverau | fungi: ^^ | 01:00 |
| jeblair | mattoliverau: we saw this on earlier hosts that eventually came up | 01:00 |
| fungi | mattoliverau: well, review.o.o did the same thing, and eventually came up (starting with an uptime of ~15 minutes, which suggests it was fsck taking that long to complete) | 01:01 |
| mattoliverau | jeblair: awesome, what good is a console, that you can't access | 01:01 |
| *** gyee has quit IRC | 01:01 | |
| mattoliverau | fungi: yeah, sounds like it. would be nice to get a console to see it checking tho :( | 01:01 |
| jeblair | or answer any questions it may have... | 01:01 |
| jeblair | fungi: have you tried nova console log with jenkins? it's probably the same vintage so it probably won't work... but still worth a shot? | 01:02 |
| fungi | jeblair: i did, first in fact, before trying the web comsole | 01:02 |
| fungi | console | 01:02 |
| mattoliverau | fungi: is this a rackspace cloud server? If so, maybe I should find someone to poke and say the console is teh broken. | 01:02 |
| clarkb | mordred: I am updating etherpad with puppet plan can you look at it and fill in holes? | 01:03 |
| mordred | clarkb: yah | 01:03 |
| jeblair | fungi, mattoliverau: jenkins.o.o is answering pings, so it's _probably_ fscking. the idea that it's waiting at a prompt is worrying though. | 01:03 |
| *** yaguang has joined #openstack-infra | 01:03 | |
| phschwartz | mattoliverau: If it is I can poke it | 01:03 |
| fungi | mattoliverau: i can't rule out the possibility that it's my browser, though now it's getting further (i actually see "status: connecting to dfw.servers.console.rackspacecloud.com, port 443..." indefinitely when i try | 01:03 |
| jeblair | phschwartz, mattoliverau: it is | 01:04 |
| jeblair | fungi: i saw that for a long time before i hit the proxy error | 01:04 |
| phschwartz | would you like me to take a look at the logs for the HV and the console? | 01:04 |
| jeblair | phschwartz: i think that would be great, fungi? | 01:04 |
| fungi | phschwartz: if you don't mind, that would be awesome... this is jenkins.openstack.org 5d35421b-95aa-4e55-adf0-3a1a3ea8dca2 | 01:05 |
| mordred | clarkb: what if ... | 01:05 |
| mattoliverau | jeblair: ok, so the networking is up, so obviously as at least mounted the / disk so it can get it's ip configuration. | 01:05 |
| fungi | jeblair: yep, now i'm getting the proxy error rather than the timeout i was getting earlier | 01:05 |
| fungi | 'network error: proxy reports "500 internal server error"' | 01:06 |
| fungi | so i guess previously the web client was hitting timeouts reaching the proxy | 01:06 |
| *** blamar has quit IRC | 01:07 | |
| phschwartz | fungi: what data center is it in? | 01:07 |
| fungi | phschwartz: dfw | 01:07 |
| *** aconrad has joined #openstack-infra | 01:08 | |
| phschwartz | fungi: Not reporting any errors, let me see if I can get the console up | 01:09 |
| fungi | yeah, the rackspace web dashboard seemed to think that server was just fine | 01:09 |
| *** sarob has quit IRC | 01:10 | |
| phschwartz | fungi: The hv thinks it is too. | 01:10 |
| *** sarob has joined #openstack-infra | 01:10 | |
| phschwartz | fungi: I can't get the console up. Going to try and get one of our cloud server ops to take a look | 01:11 |
| fungi | thanks phschwartz | 01:11 |
| *** blamar has joined #openstack-infra | 01:11 | |
| *** Sukhdev has quit IRC | 01:13 | |
| clarkb | mordred: why don't we start the puppet process fi no one needs puppet for stuff right now | 01:14 |
| clarkb | mordred: then we can work out entire process on a couple nodes | 01:14 |
| *** sarob has quit IRC | 01:15 | |
| fungi | clarkb: that seems reasonable | 01:15 |
| fungi | phschwartz: i can get to the console now | 01:15 |
| fungi | and the server's miraculously up | 01:15 |
| mattoliverau | fungi: what | 01:15 |
| mattoliverau | whats the uptime | 01:15 |
| mattoliverau | fungi: ^ | 01:15 |
| fungi | 01:15:49 up 50 min | 01:15 |
| mattoliverau | wow | 01:15 |
| mordred | jeblair: clarkb and I are haing a disagreement :) | 01:16 |
| phschwartz | fungi: hmm, maybe the HV was having an issue. But I don't see any nagios reports of issues. | 01:16 |
| fungi | i think it was fsck'ing that whole time | 01:16 |
| *** nosnos has quit IRC | 01:16 | |
| mattoliverau | must've been, and you can't get a console while it's happening :( | 01:16 |
| phschwartz | fungi: that sounds like a reasonable assumption | 01:16 |
| *** atiwari has quit IRC | 01:16 | |
| mordred | jeblair: we may need you to weigh in and break the tie - and I need to go afk my ride is here - back online soon | 01:16 |
| phschwartz | mattoliverau: I should have been able to get a console so I am going to have an admin do a once over of the hvm just incase. | 01:17 |
| fungi | phschwartz: yeah, it was responding to icmp echo request, but refusing connections on port 22 | 01:17 |
| phschwartz | defn sounds like single user mode. | 01:17 |
| fungi | which leads me to think it was in secure runlevel the whole time | 01:17 |
| *** blamar has quit IRC | 01:17 | |
| fungi | so apparently the web console does actually work in my browser, just not at the times when it would actually be useful to me ;) | 01:18 |
| fungi | clarkb: so what is the meat of the disagreement? | 01:18 |
| mattoliverau | fungi: yeah, so it's a console fail | 01:18 |
| *** jeblair has quit IRC | 01:18 | |
| clarkb | fungi: if we remove existing puppet CA how do we regenerate certs for all the nodes. Do we generate on master then push to all nodes or have each node call home and regenerate certs that way | 01:18 |
| clarkb | fungi: I am arguing for having the master do it so that launch works without needing to copy all of the files back to the master later | 01:19 |
| clarkb | fungi: but that requires ssh -A to do file copy onto nodes | 01:19 |
| fungi | clarkb: if the clients can call home, they need their local key material replaced first before they submit a signing request to the master | 01:19 |
| clarkb | fungi: correct | 01:19 |
| clarkb | they actually need it replaced in both cases | 01:19 |
| mattoliverau | phschwartz: it would be an interesting test to see if the console is always unavailable during a fsck or while in single user mode. tho that sounds stupidly wrong. Maybe getting an admin to take a look is a good idea. | 01:20 |
| clarkb | process would be something like, do master CA, for each agent stop agent, delete existing certs, either call home or copy in new data, start agent | 01:20 |
| fungi | sure, but in the generate-on-master case the keys themselves would be generated on the master and pushed over the network rather than generated on the agent side | 01:20 |
| clarkb | fungi: correct | 01:20 |
| *** aconrad has quit IRC | 01:20 | |
| mattoliverau | phschwartz: And it's a good thing your around as you have access to see these things. Rackspace doesn't give me that kind of access since I left the UK. | 01:21 |
| fungi | in a pki scenario it's generally preferred to generate the key on the client since the mastyer doesn't actually need a copy. on the other hand when replacing servers we've abused the key copy on the master before | 01:21 |
| clarkb | fungi: we need a copy in this case | 01:21 |
| *** timrc-afk is now known as timrc | 01:21 | |
| fungi | phschwartz: yeah, thanks a bunch for looking into it! | 01:22 |
| fungi | clarkb: oh, does the server generate the cert from the key rather than from a csr? | 01:22 |
| clarkb | fungi: no, it will work the way you describe but then all of our launch node automation will break | 01:22 |
| fungi | in which case puppet sort of missed the point of separating keys and certs | 01:22 |
| fungi | oh, our automation | 01:22 |
| fungi | well, yes, it does assume generating keys on the master | 01:22 |
| phschwartz | mattoliverau: I am a racker on an Openstack dev team so I have the access, but getting an admin involved is a good step here. | 01:23 |
| fungi | mainly because they're generated before the agent exists | 01:23 |
| clarkb | fungi: right bceause we need to presign | 01:23 |
| clarkb | otherwise initial boot breaks | 01:23 |
| *** jeblair has joined #openstack-infra | 01:23 | |
| phschwartz | I am trying to get much more involved with infra for the future so I will be around a lot to look into things and will make sure there is a way for people to reach me if needed. | 01:23 |
| fungi | as a side note, i wonder whether it would be at all trivial to correct that, but it's a debate for later | 01:23 |
| jeblair | sorry i dropped out | 01:24 |
| mattoliverau | phschwartz: I'm a remote racker in Australia working on upstream openstack. I don't get access, I guess cause I'm upstream. | 01:24 |
| fungi | phschwartz: thanks for having an interest in our infrastructure in particular! i'll be thrilled to give you pointers to all the fun bits when we're not in the middle of a firefight | 01:24 |
| phschwartz | :) | 01:25 |
| phschwartz | mattoliverau: hmm, you should have it | 01:25 |
| fungi | clarkb: so i agree. given the current situation with the launch automation, the distinction is academic for now | 01:25 |
| clarkb | jeblair: trying to sort out the best way to recert all of our puppet agents | 01:25 |
| phschwartz | mattoliverau: and were you at rax.io? | 01:25 |
| clarkb | jeblair: and didn't want tostart that surgery until we all agree on a course of action there | 01:25 |
| mattoliverau | phschwartz: I was indeed, I was the only openstack dev from Australia whom was there :) | 01:25 |
| fungi | clarkb: so i say we emulate what the current launch script does, unless it's sufficiently easier not to | 01:26 |
| phschwartz | mattoliverau: did you stay at aloft? | 01:26 |
| mattoliverau | I did :) | 01:26 |
| clarkb | fungi: the problem is we don't have access from master to agent via ssh at this point | 01:26 |
| phschwartz | mattoliverau: ok, I remember you now. lol | 01:26 |
| clarkb | we only have that during launch | 01:26 |
| clarkb | so we would need to ssh -A or similar | 01:26 |
| clarkb | it is all pretty ugly | 01:26 |
| fungi | clarkb: right, though one of us can grant ourselves read access to the keys and ferry them via ssh | 01:27 |
| mattoliverau | phschwartz: lol, nice :) | 01:27 |
| jeblair | what was mordred arguing? | 01:27 |
| clarkb | jeblair: mordred wanted to use puppet agent --test on the nodes to have them regenerate cert on the node side, then sign on the master side | 01:27 |
| clarkb | jeblair: which would work fine except launch node scripts expect certs to exist on the masters | 01:27 |
| clarkb | s/masters/master/ | 01:28 |
| *** julim has quit IRC | 01:28 | |
| *** signed8bit has quit IRC | 01:28 | |
| clarkb | also puppetdb is going to be a special case | 01:28 |
| fungi | larkb: as in 'ssh ci-puppetmaster cat /path/to/key | ssh wherever "cat > key"' and then do a local copy and delete on te destination host | 01:28 |
| *** matsuhashi has joined #openstack-infra | 01:28 | |
| clarkb | fungi: ya that would work too | 01:29 |
| jeblair | fungi, clarkb: i like that | 01:30 |
| fungi | probably easier than trying to set up some temporary ssh push access to the agents after there is no master relationship | 01:30 |
| clarkb | ok let me update etherpad with that then we can start doing this | 01:31 |
| jeblair | i'm going to reset the status and then reboot eavesdrop | 01:31 |
| jeblair | #status ok All services should be back up | 01:32 |
| openstackstatus | jeblair: sending ok | 01:32 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 01:32 | |
| fungi | clarkb: i did something similar in section 4 of https://etherpad.openstack.org/p/git-server-rebuild to preserve ssh host keys between server replacementd | 01:34 |
| fungi | piping through sudo like that you can avoid the local copy/delete cycle on the destination too, possibly | 01:35 |
| *** SumitNaiksatam has joined #openstack-infra | 01:35 | |
| jeblair | i think statusbot is done, but it hit another rate limit on the channel announcements | 01:36 |
| jeblair | i think it got the topics though | 01:36 |
| *** openstack has joined #openstack-infra | 01:48 | |
| jeblair | the irc bots are all running | 01:48 |
| clarkb | fungi: I think we can do an explicit generate | 01:48 |
| fungi | i guess 'puppet cert --generate ci-puppetmaster.openstack.org' will just do what's needed for this? | 01:49 |
| clarkb | fungi: yup it should | 01:49 |
| fungi | lgtm then | 01:49 |
| clarkb | I have put #'s in front of nodes that in allcerts.txt that we shouldn't regenerate certs for | 01:49 |
| clarkb | having more eyes on that list to make sure I didn't exclude live nodes or miss dead nodes would be great | 01:49 |
| jeblair | clarkb: remove ask.o.o | 01:50 |
| fungi | yeah, we don't have ssh access to ask | 01:51 |
| clarkb | done | 01:51 |
| * clarkb takes a short break before breaking everything | 01:51 | |
| jeblair | remove centos6.slave.openstack.org ci-master.openstack.org citest.openstack.org devstack-launch* | 01:51 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow 'Build on the same node' in trigger-builds builder https://review.openstack.org/85668 | 01:52 |
| jeblair | remove jenkins2.openstack.org large1.openstack.org large2.openstack.org | 01:53 |
| fungi | wheezydev.slave | 01:53 |
| jeblair | remove precise.slave.openstack.org | 01:54 |
| fungi | review2, review.paas.hpcloud.net, review-dev2 | 01:54 |
| fungi | quantal.slave | 01:54 |
| jeblair | precise3k-1.slave.openstack.org precisepy3k* | 01:54 |
| jeblair | precisetest.slave.openstack.org | 01:54 |
| fungi | puppet-dashboard, puppet-bootstrap | 01:54 |
| fungi | i think we've met from both ends now | 01:55 |
| jeblair | storyboard-dev.openstack.org | 01:55 |
| fungi | ohm right, that did just get deleted recently | 01:55 |
| jeblair | translate.openstack.org | 01:55 |
| jeblair | which is actually translate-dev.o.o... | 01:56 |
| *** bhuvan has quit IRC | 01:57 | |
| fungi | fedora18.slave | 01:57 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Support Matrix Tie Parent Plugin in wrappers https://review.openstack.org/85642 | 01:58 |
| clarkb | ok all those have been removed | 01:58 |
| fungi | precise.slave | 01:59 |
| clarkb | done | 01:59 |
| fungi | pypi and pypi.slave | 01:59 |
| fungi | no, not pypi.slave | 01:59 |
| fungi | just pypi | 01:59 |
| *** changbl has joined #openstack-infra | 02:00 | |
| clarkb | isn't pypi.o.o a thing? | 02:00 |
| clarkb | hrm maybe not | 02:00 |
| fungi | it moved to static a while back and got deleted | 02:00 |
| fungi | pypi.slave is definitely still a thing | 02:00 |
| clarkb | ya | 02:00 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Fix some minor typos in the trigger documentation https://review.openstack.org/85885 | 02:00 |
| fungi | pypi is just a vhost on static now though | 02:01 |
| *** wenlock has joined #openstack-infra | 02:01 | |
| fungi | so that's 73 systems | 02:01 |
| clarkb | fungi: jeblair: mordred: if I start this now will you guys be around to help out with the re certing? | 02:01 |
| clarkb | I definitely don't want to do it all in one giant automated script that will break and need redoing | 02:01 |
| fungi | clarkb: i'll try to help out, but i'm rapidly running out of steam (nearing bedtime) | 02:02 |
| clarkb | at least not for the first few | 02:02 |
| *** david-lyle has joined #openstack-infra | 02:02 | |
| clarkb | well I am going to stop apache on the master now | 02:02 |
| fungi | k | 02:02 |
| clarkb | ok puppet master is now using a new cert | 02:04 |
| *** sandywalsh has quit IRC | 02:05 | |
| clarkb | gah puppetdb is cranky | 02:06 |
| clarkb | going to try fixing that by hand out of bad | 02:06 |
| clarkb | *out of band | 02:06 |
| *** alff has quit IRC | 02:06 | |
| *** alff_ has quit IRC | 02:06 | |
| blkperl | clarkb: theres a script that should fix that | 02:06 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Minor fixes in the installation documentation https://review.openstack.org/85886 | 02:07 |
| blkperl | clarkb: http://docs.puppetlabs.com/puppetdb/latest/puppetdb-faq.html#puppetdb-is-complaining-about-a-truststore-or-keystore-file-what-do-i-do | 02:07 |
| clarkb | right I think yo ucovered that in your blog | 02:08 |
| clarkb | Iam doing that now | 02:08 |
| blkperl | :) | 02:08 |
| clarkb | that didn't fix it | 02:09 |
| clarkb | tehre were warnings about /var/lib/puppetdb not matcing certs in /var/lib/puppet wonder if it nooped in that case ... | 02:10 |
| clarkb | I hate puppetdb right now | 02:10 |
| clarkb | why wouldn't tis be like any other agent? | 02:10 |
| clarkb | and why does puppetdb break the master? | 02:10 |
| *** mattoliverau has quit IRC | 02:10 | |
| clarkb | this is far too tightly coupled | 02:10 |
| *** sweston has quit IRC | 02:11 | |
| clarkb | wtf it didn't create a /var/lib/puppet/ssl | 02:11 |
| clarkb | ugh | 02:12 |
| clarkb | oh its in /etc/puppetd | 02:12 |
| fungi | clarkb: oh, that could be why the rekeying instructions did find -exec rm instead of rm -fr | 02:12 |
| clarkb | is that a symlink the warnings are about /var/lib/puppetdb | 02:12 |
| clarkb | fungi: no, I think this is just being stupid | 02:12 |
| fungi | ahh | 02:12 |
| clarkb | ugh, and those depends on /var/lib/ssl there is a cyclic dependency | 02:14 |
| clarkb | so stupid | 02:14 |
| clarkb | I am going t omake puppetdb a special case and generate its stuff local to it | 02:14 |
| * mgagne brings popcorn | 02:16 | |
| clarkb | mgagne: :P | 02:17 |
| clarkb | honestly though the chicken and egg here is not very amusing | 02:17 |
| clarkb | puppet won't run without puppetdb, puppetdb won't run without certs | 02:17 |
| mgagne | clarkb: I know one of our team had this exact problem and they cursed at their monitors for days. Unfortunately, I don't have the exact info on how they fixed it | 02:18 |
| mgagne | clarkb: trying to find where they put their work :-/ | 02:18 |
| clarkb | I am ready to uninstall puppetdb | 02:18 |
| blkperl | I don't think it needs to run, it just needs to make the correct certs | 02:19 |
| jeblair | clarkb: i'm semi-around | 02:19 |
| jeblair | clarkb: let me know if you have a task for me | 02:19 |
| *** nati_ueno has quit IRC | 02:19 | |
| clarkb | jeblair: well I have new certs in puppetdb:/var/lib/puppet/ssl that I generated on puppetdb and signed by and on puppet master. | 02:19 |
| clarkb | now I am running the command to rebuild the certs in /etc/puppetdb/ssl and have restarted puppetdb but puppet agent --test still doesn't work | 02:20 |
| nibalizer | clarkb: move aside puppetdb.yaml on the msaster | 02:20 |
| nibalizer | and remove reports = puppetdb from puppet.conf on the master | 02:20 |
| nibalizer | and then puppet will run | 02:20 |
| nibalizer | thats 'disabling' puppetdb from the master perspective | 02:20 |
| clarkb | nibalizer: but I want working puppetdb or no puppetdb | 02:21 |
| clarkb | not reporting to it is a bad middle ground | 02:21 |
| jeblair | clarkb: isn't that 'no puppetdb'? | 02:22 |
| fungi | i assume he means just long enough to get the updated cert onto the puppetdb server | 02:22 |
| nibalizer | yep | 02:22 |
| nibalizer | also could puppet cert generate and copy over | 02:22 |
| clarkb | I already have the cert signed and on the puppetdb | 02:22 |
| nibalizer | whats the problem you're having? | 02:22 |
| clarkb | Error 400 on SERVER: Failed to submit 'replace facts' command for ci-puppetmaster.openstack.org to PuppetDB at puppetdb.openstack.org:8081 | 02:23 |
| * clarkb is looking at network connectivity now | 02:23 | |
| *** mattoliverau has joined #openstack-infra | 02:23 | |
| nibalizer | do you have an old CRL on the puppetdb server? | 02:23 |
| clarkb | ah connection refused | 02:23 |
| nibalizer | did you get the stuff in /etc/puppetdb ? | 02:23 |
| nibalizer | did a firewall turn on? | 02:23 |
| clarkb | nibalizer: no I moved /var/lib/puppet/ssl aside and /etc/puppetdb/ssl aside | 02:24 |
| clarkb | 8081 is only listening on the 10 address is that expected? | 02:24 |
| nibalizer | nope | 02:24 |
| nibalizer | what's in jetty.ini ? | 02:24 |
| nibalizer | there should be a listen address in one of those files | 02:24 |
| nibalizer | that address should be the hostname | 02:25 |
| nibalizer | and maybe the 10 address is higher in /etc/hosts than the real address? | 02:25 |
| clarkb | ssl-host = puppetdb.openstack.org | 02:25 |
| *** zz_gondoi is now known as gondoi | 02:25 | |
| nibalizer | host puppetdb.openstack.org on puppetdb? | 02:26 |
| fungi | yeah, didn't we discover that setting an empty listen address maybe 0.0.0.0 caused it to listen on all interface addresses (including ipv6)? | 02:26 |
| nibalizer | oh yea that! | 02:26 |
| nibalizer | do that! | 02:26 |
| clarkb | what do I do? | 02:26 |
| nibalizer | fungi++ | 02:26 |
| nibalizer | ssl-host = 0.0.0.0 | 02:26 |
| clarkb | k | 02:26 |
| fungi | in the configuration on the puppetdb server | 02:26 |
| fungi | i thought that was in the puppet config template for that module now though | 02:26 |
| nibalizer | right but puppet isn't running to set that up | 02:27 |
| nibalizer | this is so broken chinken-and-egg | 02:27 |
| jeblair | clarkb: yeah, jetty.ini.bak.1396923405 has that | 02:27 |
| clarkb | nibalizer: right, puppetlabs should maybe consider fixin that | 02:27 |
| fungi | i wonder what regenerated the jetty.ini | 02:27 |
| clarkb | could be confused puppet agents with the CA having broken? | 02:27 |
| clarkb | that is scary because they should just do nothing | 02:28 |
| *** dkliban has quit IRC | 02:28 | |
| fungi | i have seen puppet agent think it needed to fall back on a locally cached copy of its config before (and find some horribly outdated one which shouldn't have been there or something) | 02:28 |
| fungi | i believe someone mentioned a way to turn that behavior off, but i don't recall the specifics | 02:29 |
| *** changbl has quit IRC | 02:29 | |
| clarkb | :( | 02:29 |
| nibalizer | fungi: probably the 'fixit' script that clarkb ran, or maybe a puppetdb package? | 02:29 |
| nibalizer | clarkb: you ran the puppetdb-ssl-fixplz script right? | 02:29 |
| fungi | oh, maybe scripty-thingy did it | 02:29 |
| clarkb | nibalizer: yes | 02:30 |
| clarkb | that may have done it too | 02:30 |
| fungi | ohai-plz-to-be-fixing | 02:30 |
| nibalizer | fungi: ohai-plz-to-be-fixing --srsly | 02:30 |
| *** Shrews has quit IRC | 02:30 | |
| clarkb | puppet agent is running again on puppet master and puppetdb | 02:30 |
| *** harlowja is now known as harlowja_away | 02:30 | |
| clarkb | I am going to test our process on logstash-worker16 for the remaining nodes | 02:30 |
| clarkb | then when that is done I may ask jeblair and fungi to poke at a couple nodes so that everything looks good | 02:31 |
| fungi | if it helps, i did check out the ca cert and it seems to have used sane options when generated | 02:31 |
| *** Shrews has joined #openstack-infra | 02:31 | |
| clarkb | thanks | 02:31 |
| fungi | so we won't, like, have it expire on us in 30 days or anything annoying like that | 02:31 |
| *** amcrn has joined #openstack-infra | 02:32 | |
| jeblair | clarkb: let me know when you need a hand | 02:34 |
| nibalizer | default is 5 years | 02:35 |
| nibalizer | anything else I can do to help? | 02:35 |
| clarkb | jeblair: I am editing the etherpad as I find problems | 02:35 |
| clarkb | jeblair: will let you know when I think it is ready for generate abuse | 02:35 |
| clarkb | nibalizer: nope, I think we are good now thanks | 02:35 |
| mgagne | clarkb: finally found what I was looking for. They inject the CA and signed certs beforehand using hiera. | 02:37 |
| *** melwitt has quit IRC | 02:38 | |
| *** Ryan_Lane1 has quit IRC | 02:38 | |
| clarkb | jeblair: fungi ok I think the steps there work | 02:39 |
| clarkb | I am going to start with my same set of nodes from the reboot list | 02:40 |
| *** esker has joined #openstack-infra | 02:40 | |
| clarkb | any chance you guys want to do the same with your lists? | 02:40 |
| *** esker has quit IRC | 02:40 | |
| *** harlowja_away is now known as harlowja | 02:40 | |
| jeblair | clarkb: will do | 02:40 |
| fungi | yep | 02:40 |
| jeblair | clarkb: question in etherpad, just to make sure... | 02:41 |
| jeblair | clarkb: also, have you done the cert generate for everything? | 02:41 |
| *** asettle is now known as alex-lunch | 02:42 | |
| clarkb | jeblair: I have not done cert generate for everything you will need to do that before you touch your nodes | 02:43 |
| *** dcramer_ has joined #openstack-infra | 02:43 | |
| clarkb | I kinda want to compare the list we generated against what we end up with in the puppet ssl dir | 02:43 |
| fungi | clarkb: recommended way for making sure this is working? just watch puppet-dashboard? | 02:45 |
| clarkb | fungi: I ran puppet agent --test on the first node I did manually | 02:45 |
| clarkb | but yes dashboard should show last report times | 02:46 |
| clarkb | oh wait, we may not be applying the right perms in /var/lib/ssl | 02:47 |
| clarkb | I think preserving perms was mostly what we wanted but the user needs to be puppet isntead of root | 02:47 |
| clarkb | actually nevermind it seems to work fien with the perms from master | 02:48 |
| jeblair | clarkb, fungi: i just pasted in a python script i'm using | 02:50 |
| *** amotoki has quit IRC | 02:53 | |
| *** amcrn has quit IRC | 02:53 | |
| fungi | okay, my section are rekeyed | 02:54 |
| jeblair | fungi: want to start on mordred's? | 02:55 |
| fungi | yep, just checking puppetboard to make sure they checked in first | 02:55 |
| fungi | oh, of course all the jenkins servers are complaining because i set the jenkins packages on them to held status (so they wouldn't auto-upgrade) | 02:56 |
| clarkb | logstash* are done | 02:58 |
| clarkb | doing elasticsearch* now | 02:58 |
| jeblair | done with my precise hosts; working on centos now | 02:59 |
| fungi | clarkb: you have "ssh node 'sudo /etc/init.d/puppet start'" in there, but didn't that previously cause issues with the mysql module? has that ceased being an issue now? | 03:01 |
| clarkb | elasticsearch* done | 03:01 |
| clarkb | fungi: it should be fine if using the init script iirc | 03:01 |
| clarkb | its only when run directly with sudo eg puppet agent that breaks | 03:01 |
| fungi | oh, good to know | 03:02 |
| *** alex-lunch is now known as asettle | 03:03 | |
| *** maxbit has joined #openstack-infra | 03:05 | |
| *** sweston has joined #openstack-infra | 03:05 | |
| jeblair | https://rhn.redhat.com/errata/RHSA-2014-0376.html | 03:05 |
| clarkb | jeblair: looks like centos 6 has it now too | 03:06 |
| clarkb | I am going to see if I can upgrade git05 now | 03:06 |
| jeblair | clarkb: oh? i just checked and it didn't say... | 03:06 |
| clarkb | unless people want me to do anything with pupept recerting | 03:06 |
| clarkb | jeblair: http://mirror.centos.org/centos/6.5/updates/x86_64/Packages/ has it | 03:07 |
| clarkb | rax may be masking it for us though | 03:07 |
| fungi | okay, got through rekeying all of mordred's section too | 03:07 |
| fungi | i'll go ahead and un-hold jenkins on all the jenkins masters so puppet will stop complaining about the held status | 03:08 |
| jeblair | clarkb: yeah, the [updates] repo is a mirror | 03:08 |
| jeblair | clarkb: uncommenting the #mirrors line in that section should do it | 03:09 |
| * clarkb learns yum | 03:10 | |
| jeblair | i've completed the puppet stuff for all my hosts | 03:10 |
| jeblair | clarkb: uncomment that line, then 'yum update' | 03:11 |
| clarkb | ok | 03:11 |
| * clarkb tries that on git05 | 03:11 | |
| clarkb | jeblair: fungi: do we want to try cycling git nodes in a non impactful way or just reboot them? | 03:11 |
| jeblair | clarkb: just reboot i think | 03:11 |
| clarkb | jeblair: uncomment the line in /etc/yum.repos.d/CentOS-Base? | 03:12 |
| jeblair | clarkb: yep | 03:12 |
| fungi | and after they're all done, gerrit replicate --all | 03:12 |
| fungi | not important for the moment, but tripleo seems to have died again | 03:13 |
| fungi | i'll give #tripleo a quick heads up | 03:13 |
| jeblair | fungi: it's possible that nodepool lost track of the few remaining nodes in the restart | 03:13 |
| clarkb | well that seems to have just broken facter.. | 03:13 |
| jeblair | clarkb: not if you didn't say "yes" | 03:13 |
| fungi | jeblair: good point | 03:14 |
| clarkb | jeblair: ah ok, it bailed without asking me to say yes | 03:14 |
| jeblair | clarkb: those hosts are probably missing the exclude | 03:14 |
| jeblair | fungi: it should clear up after 8 hours, but can you check and maybe nodepool delete them if that's what it looks like? | 03:14 |
| jeblair | fungi: it should think there are a bunch of used nodes in that state for several hours | 03:15 |
| *** zehicle has quit IRC | 03:15 | |
| clarkb | jeblair: yup appears to be the case fixing | 03:15 |
| jeblair | clarkb: i've got git.o.o and the others in that section, you get gitNN, ok? | 03:16 |
| clarkb | ok I can do gitNN | 03:16 |
| clarkb | jeblair: fwiw I still can't get it to update after the facter thing | 03:16 |
| clarkb | I think it is finding the fastest mirror | 03:16 |
| clarkb | rather than the mirror | 03:16 |
| fungi | jeblair: yep, that was it--clearing those now | 03:16 |
| jeblair | clarkb: yeah, that seems to be the case | 03:17 |
| * clarkb fixes facter | 03:17 | |
| jeblair | clarkb: baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/ | 03:19 |
| jeblair | clarkb: that should do it then | 03:20 |
| clarkb | that did it | 03:20 |
| clarkb | working through gitNN now without rebooting | 03:21 |
| * mordred is back - sorry for the delay - anything left I can help on? | 03:21 | |
| *** thuc has joined #openstack-infra | 03:22 | |
| *** thuc_ has joined #openstack-infra | 03:22 | |
| clarkb | mordred: maybe keep any eye on puppetboard to make sure that we recerted all the nodes | 03:23 |
| clarkb | mordred: and look at nodes that aren't reporting in the last 20 minutes | 03:23 |
| *** changbl has joined #openstack-infra | 03:24 | |
| jeblair | clarkb: i had to do 'yum clean all' a few times on one of them because it wasn't downloading the repo data | 03:24 |
| mordred | clarkb: kk | 03:24 |
| jeblair | i'm going to reboot the non-git nodes | 03:25 |
| clarkb | jeblair: ok | 03:25 |
| *** thuc has quit IRC | 03:26 | |
| mordred | all of the -dev servers seem non-responsive- is that a known thing? | 03:27 |
| jeblair | mordred: no | 03:27 |
| clarkb | 16.el6_5.7 is the version we want | 03:27 |
| jeblair | clarkb: correct | 03:27 |
| clarkb | I am double checking that it is installed on the gitNN nodes | 03:27 |
| clarkb | jeblair: you did git.o.o right? | 03:27 |
| jeblair | rpm -q openssl | 03:28 |
| jeblair | yep | 03:28 |
| mordred | clarkb: if I find a server that does not seem to have a cert - do you havea script/process to fix? I'm assuming make on the master and copy to the machine? | 03:28 |
| jeblair | mordred: give me the fqdn i'll do it quick | 03:28 |
| mordred | jeblair: groups.o.o | 03:28 |
| clarkb | git01-git05 are ready | 03:28 |
| fungi | i thought i did groups.o.o | 03:28 |
| mordred | jeblair: groups.openstack.org if you want cut and paste | 03:28 |
| *** wchrisj has quit IRC | 03:29 | |
| jeblair | mordred: err: Could not call generate: A Certificate already exists for groups.openstack.org | 03:29 |
| *** matsuhashi has quit IRC | 03:29 | |
| fungi | but yeah, still looks unreported | 03:29 |
| *** Ryan_Lane has joined #openstack-infra | 03:29 | |
| fungi | along with groups-dev and jenkins-dev | 03:29 |
| *** wchrisj has joined #openstack-infra | 03:29 | |
| jeblair | mordred: where is it missing a cert? | 03:29 |
| clarkb | jeblair: did the tar step get neglected? | 03:30 |
| clarkb | fungi: ^ | 03:30 |
| jeblair | clarkb: want to just reboot all the git servers together? | 03:30 |
| fungi | clarkb: that's what i'm checking, but i don't think so | 03:30 |
| *** blamar has joined #openstack-infra | 03:30 | |
| mordred | jeblair: oh - sorry, I got a cert-related puppet error then when rechecking unreported nodes | 03:30 |
| clarkb | jeblair: yeah I think so | 03:30 |
| mordred | I get this: err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=ci-puppetmaster.openstack.org] | 03:30 |
| jeblair | clarkb: why don't you just do them all | 03:30 |
| mordred | on mirror26 | 03:30 |
| clarkb | jeblair: I was going to for loop a thing that logged in a sudo reboot -h now | 03:30 |
| *** zehicle has joined #openstack-infra | 03:30 | |
| jeblair | clarkb: ++ | 03:31 |
| jeblair | clarkb: (including git.o.o) | 03:31 |
| clarkb | jeblair: ok doing that now | 03:31 |
| fungi | on groups.o.o /var/lib/puppet/ssl/{certs,private_keys,public_keys}/groups.openstack.org.pem are all there with today's date | 03:31 |
| clarkb | er less the -h now | 03:31 |
| clarkb | too much shutdown -h now habits for personal machines | 03:32 |
| clarkb | `sudo reboot` is what I am doing | 03:32 |
| clarkb | lol command reboot not found :/ | 03:32 |
| mordred | also - mirror26 is a centos node too - does it need the yum update you guys just did for git*? | 03:32 |
| mordred | and - zm01 doesn't have puppet installed | 03:32 |
| clarkb | ok all 6 git* nodes are rebooting | 03:33 |
| jeblair | sha1sums of those files don't match the puppetmaster | 03:33 |
| fungi | i'm going to try restarting the puppet agent again on groups.o.o in case i restarted it too quickly after removing/replacing certs | 03:33 |
| fungi | oh? | 03:33 |
| jeblair | mordred: i got mirror26, but it's recent | 03:33 |
| mordred | jeblair: ok. cool | 03:33 |
| * mordred is going to install puppet on zm01 | 03:34 | |
| jeblair | mordred: can you find out why it's not installed | 03:34 |
| mordred | jeblair: sure! | 03:34 |
| jeblair | because clearly it was at some point | 03:34 |
| *** gondoi is now known as zz_gondoi | 03:34 | |
| *** zz_gondoi is now known as gondoi | 03:34 | |
| fungi | how weird. the keys indeed don't match but the certs do. i'll retransfer those | 03:34 |
| clarkb | running replicate --all on review.o.o now | 03:35 |
| *** weshay has quit IRC | 03:35 | |
| *** blamar has quit IRC | 03:35 | |
| jeblair | fungi: the timestamps on the keys on the host were 2 minutes later than the master | 03:35 |
| *** blamar has joined #openstack-infra | 03:35 | |
| clarkb | done | 03:36 |
| * clarkb tries to clone stuff | 03:36 | |
| mordred | jeblair: only think I can see in the root history is the dist-upgrade steps | 03:36 |
| mordred | 2014-04-07 23:36:54 remove puppet-common 2.7.25-1puppetlabs1 <none> | 03:36 |
| jeblair | fungi: i'm guessing something out of sequence caused puppet to make some local certs | 03:36 |
| mordred | from the log | 03:36 |
| mordred | I think dist-upgrade boned it | 03:36 |
| jeblair | neat | 03:36 |
| jeblair | i think zuul-dev may have the same prob | 03:36 |
| mordred | ah - yup - it didn't get the facter pin before the distupgrade | 03:37 |
| mordred | it has facter 2.0 | 03:37 |
| jeblair | sorry, i thought i ran puppet agent --test everywhere | 03:37 |
| clarkb | I am able to clone openstack-infra/config from git.o.o via all 3 clone methods | 03:37 |
| jeblair | actually i'm sure of it, it's right there in my history | 03:37 |
| clarkb | so I think git replicas are good | 03:37 |
| fungi | err: Could not call generate: groups-dev.openstack.org already has a requested certificate; ignoring certificate request | 03:38 |
| fungi | anybody know where those get stashed so i can delete it? | 03:38 |
| clarkb | fungi: /var/lib/puppet/ssl | 03:38 |
| clarkb | fungi: you might also need to puppet cert something something on the puppet master | 03:38 |
| clarkb | fungi: puppet cert list --all to see alisting | 03:38 |
| jeblair | fungi: oh i thought you were just going to recopy from the master? | 03:38 |
| fungi | jeblair: i did for groups.o.o | 03:39 |
| fungi | for groups-dev there was a pending csrt | 03:39 |
| fungi | csr | 03:39 |
| jeblair | fungi: ah ok | 03:39 |
| fungi | certificate_requests was the subpath i was hunting for | 03:39 |
| jeblair | mordred: yep, my fault, sorry | 03:39 |
| mordred | zm01 fixed- | 03:39 |
| mordred | and reporting | 03:39 |
| mordred | woot | 03:39 |
| jeblair | mordred: and yeah, i did the same thing on zuul-dev too | 03:40 |
| mordred | jeblair: want me to fix? or you got it? | 03:40 |
| jeblair | mordred: can you? | 03:40 |
| mordred | jeblair: yup. on it | 03:40 |
| jeblair | your fingers are already warmed up | 03:40 |
| fungi | 'sudo puppet cert generate groups-dev.openstack.org' seems to always create a /var/lib/puppet/ssl/certificate_requests/groups-dev.openstack.org.pem and then complain that it exists | 03:40 |
| fungi | very, very weird | 03:41 |
| mordred | ruby | 03:41 |
| fungi | this is with puppet agent stopped on groups-dev as well | 03:41 |
| clarkb | fungi: is that on the master? | 03:41 |
| fungi | clarkb: yep | 03:41 |
| clarkb | yo uwant to generate the cert on the master | 03:41 |
| fungi | clarkb: yep | 03:41 |
| clarkb | fungi: does puppet cert list --all show a cert pending? | 03:41 |
| fungi | that's what i'm trying | 03:41 |
| clarkb | fungi: if it shows one pending it might be easiset to just sign that cert then copy it the other direction | 03:41 |
| clarkb | I will need to do this with the puppetdb certs tomorrow | 03:41 |
| jeblair | er | 03:42 |
| fungi | clarkb: yeah, it's listed | 03:42 |
| jeblair | don't sign it unless you've verified it's correct | 03:42 |
| clarkb | jeblair: ya that | 03:42 |
| jeblair | lots of random people send us csrs | 03:42 |
| fungi | trying to figure out how to delist it so i can generate a new one | 03:42 |
| clarkb | when I did puppetdb only it and puppet master were listed, was easy to check and sing :) | 03:42 |
| jeblair | fungi: 'puppet cert clean' ? | 03:42 |
| fungi | err: Could not call revoke: Could not find a serial number for groups-dev.openstack.org | 03:43 |
| fungi | that seems to only work on those which have already been signed | 03:43 |
| fungi | checking manpage for viable options | 03:43 |
| *** dkliban has joined #openstack-infra | 03:44 | |
| *** chandan_kumar has quit IRC | 03:44 | |
| fungi | i wonder if there's an additional groups-dev out there somewhere trying to get noticed | 03:44 |
| clarkb | I am going to patch my local(ish) nodes now | 03:45 |
| clarkb | biab | 03:45 |
| *** thuc has joined #openstack-infra | 03:45 | |
| *** blamar has quit IRC | 03:45 | |
| fungi | found! also have to rm /var/lib/puppet/ssl/ca/requests/groups-dev.openstack.org.pem | 03:46 |
| mordred | jeblair: zuul-dev fixed | 03:46 |
| jeblair | mordred: cool thanks | 03:46 |
| mordred | jeblair: did you say you were dealing with cert issue on mirror26? or do you want me to look in to that? | 03:47 |
| fungi | okay, groups-dev all fixed up | 03:47 |
| fungi | jenkins-dev seems to have never checked in | 03:47 |
| jeblair | mordred: i was not, but i will look at mirrir26 now | 03:47 |
| *** thuc_ has quit IRC | 03:48 | |
| *** wenlock_ has joined #openstack-infra | 03:48 | |
| *** timrc is now known as timrc-afk | 03:48 | |
| *** thuc has quit IRC | 03:49 | |
| fungi | okay, puppetboard shows groups and groups-dev rporting in now | 03:49 |
| *** amcrn has joined #openstack-infra | 03:49 | |
| mordred | woot | 03:49 |
| mordred | devstack-launch.slave.openstack.org is not - but I think it's a thing we don't care about, yeah? | 03:50 |
| fungi | puppet agent is unable to start on jenkins-dev | 03:50 |
| jeblair | mordred: i think it doesn't exist anymore, right? | 03:50 |
| mordred | it does exist | 03:50 |
| *** chandan_kumar has joined #openstack-infra | 03:50 | |
| fungi | mordred: i think it still is pending deletion | 03:50 |
| mordred | but I'm not sure why it still exists | 03:50 |
| mordred | fungi: should we fix it or delete it? | 03:50 |
| fungi | mordred: i vote delete | 03:50 |
| jeblair | kill it | 03:50 |
| * mordred on it | 03:50 | |
| fungi | please clear the record from dns too if you have time | 03:51 |
| mordred | fungi: while you're doing certs - you wanna delete it from the cert list? | 03:51 |
| fungi | mordred: it should not be in the new cert list | 03:51 |
| jeblair | hrm, the ca.pem on mirror26 is old | 03:51 |
| clarkb | gah /boot doesn't have enough space on disk to mkinitramfs | 03:51 |
| fungi | clarkb: started from a fresh cert list | 03:51 |
| clarkb | maybe ubuntu sould make /boot bigger by default | 03:51 |
| jeblair | but the rest of the certs are there... | 03:51 |
| clarkb | jeblair: the rm /var/lib/puppet/ssl should've removed it then the first puppet run grab it again | 03:52 |
| clarkb | jeblair: you might want to run that node through the deletion coyp steps just to be safe | 03:52 |
| mordred | jeblair: puppetdb is still reporting it though | 03:52 |
| jeblair | ooooh | 03:53 |
| jeblair | it's because we ran puppet stop && rm -rf | 03:53 |
| jeblair | puppet wasn't running so it short circuited | 03:53 |
| clarkb | jeblair: gah sorry | 03:53 |
| clarkb | my bug | 03:53 |
| jeblair | (puppet doesn't run agent on slaves) | 03:53 |
| fungi | Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass mysql_root_password to Class[Openstack_project::Nodepool] at /opt/config/production/modules/openstack_project/manifests/nodepool.pp:3 on node jenkins-dev.openstack.org | 03:54 |
| clarkb | fungi: I saw that prior to all of this | 03:54 |
| fungi | so we have broken puppetry for jenkins-dev looks like | 03:54 |
| clarkb | fungi: so :/ | 03:54 |
| fungi | which is why i can't start the puppet agent there | 03:54 |
| jeblair | why can't that be detected by the syntax checks? | 03:55 |
| *** chandan_kumar has quit IRC | 03:56 | |
| * jeblair despairs of puppet | 03:56 | |
| clarkb | ok laptop is rebooting back in a bit | 03:56 |
| clarkb | fungi: any idea how to convince wheezy to upgrade to openssl*u5 on rax? | 03:56 |
| clarkb | fungi: I assume same issue there as with our precise nodes | 03:56 |
| jeblair | someone should just yank nodepool from jenkins-dev | 03:56 |
| jeblair | let me know if you want me to do that | 03:56 |
| jeblair | mordred: mirror26 is running | 03:57 |
| *** gondoi is now known as zz_gondoi | 03:57 | |
| clarkb | jeblair: tehre is a change up to fix it on jenkins-dev | 03:57 |
| fungi | clarkb: probably, if you don't see security.debian.org in the sources.list | 03:57 |
| mordred | jeblair: awesome. I'm deleting dns for devstack-launch | 03:57 |
| clarkb | jeblair: would be nice to just get that thorugh | 03:57 |
| jeblair | clarkb: i've reviewed it 5 times, i'm not going to review it again right now :( | 03:57 |
| jeblair | clarkb: but if you want to pick this up tomorrow, we can probably just leave jenkins-dev unreporting | 03:58 |
| mordred | jeblair, fungi: dns record deleted for devstack-launch | 03:58 |
| jeblair | mordred: ack, thx | 03:58 |
| *** wchrisj has quit IRC | 03:59 | |
| *** sabari has joined #openstack-infra | 04:00 | |
| jeblair | i have an early flight tomorrow and have not packed; i should really go prepare for that | 04:01 |
| mordred | jeblair: ++ | 04:01 |
| fungi | jeblair: thanks for all the help--travel well! | 04:01 |
| clarkb | gnite | 04:01 |
| mordred | fungi: for reference, stale nodes in pupeptboard can e removed with "puppet node deactivate devstack-launch.slave.openstack.org" | 04:01 |
| jeblair | i think we've gotten the high-prority stuff done at least | 04:01 |
| jeblair | mordred, clarkb, fungi: thanks and good night | 04:01 |
| clarkb | also xubunty should really tell me what is broken when it wants to phone home | 04:02 |
| clarkb | I have no idea what is broken right now | 04:03 |
| clarkb | but it insists on calling home without giving me any info | 04:03 |
| clarkb | pleia2: ^ | 04:03 |
| fungi | i'm going to get some sleep now, and stop caring about puppet on jenkins-dev for the moment. if anyone else cares about it working tonight, feel free to have a look | 04:03 |
| mgagne | jeblair: you could detect such "syntax errors" by adding unit tests. But I do not believe it to be possible with current openstack-infra setup as it heavily depends on secret data in hiera. This means unit tests (executed w/o hiera) would fail (due to missing hiera entries) while in production it would work as hiera is available. | 04:04 |
| clarkb | fungi: I will probably try to get it working properly tomorrow | 04:04 |
| clarkb | fungi: it should be fine in its current state | 04:04 |
| fungi | clarkb: it's probably just missing a variable passed in through its manifest or something | 04:04 |
| mordred | mgagne: I think we can set defaults in our hiera calls for what to use if there is no hiera | 04:04 |
| mgagne | mordred: true | 04:05 |
| mordred | clarkb: I'm reviewing khai's change to fix nodepool on jenkins-dev right now | 04:05 |
| mgagne | mordred: but missing required (no longer) secret values would now be harder to debug as puppet won't fail anymore and use invalid default value | 04:05 |
| clarkb | fungi: good night and thank you for staying up to help | 04:06 |
| fungi | the change up to fix nodepool on jenkins-dev looks only half done (for example, it only templatizes one of the leav prepare_node_*.sh scripts and adds the key var there, but leaves other prepare scripts out in the cold... might warrant happening in the bottom-level prepare_node.sh or something | 04:06 |
| mordred | mgagne: ++ | 04:06 |
| mordred | yah | 04:06 |
| mgagne | mordred: you can inject those in unit tests but you aren't testing against production config anymore so tests aren't so valid anymore | 04:06 |
| * fungi places his wetware into standby for recharging | 04:07 | |
| *** Ryan_Lane has quit IRC | 04:07 | |
| clarkb | huh I think openvpn might be breaking me | 04:07 |
| clarkb | it is trying to start on boot but can't read a passwrd on non existant stdin | 04:07 |
| clarkb | which is lolzy | 04:08 |
| pleia2 | clarkb: it should tell you if you click on the details thing (which should exist), but honestly I just turn apport off because I don't care when things crash and I don't like it when it sends reports from my day to day system | 04:08 |
| clarkb | pleia2: it just says apport thing | 04:08 |
| clarkb | pleia2: basically telling me apport is going to take my password and collect data as root | 04:08 |
| clarkb | which isn't super useful | 04:08 |
| pleia2 | bah | 04:09 |
| clarkb | maybe to people that don't grok linux but for me I want to see the error | 04:09 |
| pleia2 | yeah | 04:09 |
| clarkb | I think it is related to openvpn though | 04:09 |
| *** maxbit has quit IRC | 04:11 | |
| clarkb | well that wasn't it. I see network manager is complaining about dnsmasq not existing though | 04:12 |
| *** chrisstreeter_ has joined #openstack-infra | 04:12 | |
| clarkb | that is the only error from the most recent boot | 04:14 |
| *** chandan_kumar has joined #openstack-infra | 04:14 | |
| *** sabari has quit IRC | 04:14 | |
| clarkb | guessing that is a race? | 04:14 |
| pleia2 | yeah | 04:15 |
| *** wchrisj_ has joined #openstack-infra | 04:15 | |
| pleia2 | pretty sure there is a bug out for that | 04:16 |
| clarkb | ya looks like xfce starts dnsmasq but network manager starts at boot | 04:16 |
| clarkb | oh nevermind network manager starts dnsmasq | 04:16 |
| clarkb | so network manager must be looking for it before it forks and execs it | 04:17 |
| * clarkb ignores for now as that isn't too broken | 04:17 | |
| *** unicell has quit IRC | 04:18 | |
| *** kashyap` is now known as kashyap | 04:18 | |
| *** ianw has quit IRC | 04:19 | |
| *** wchrisj_ has quit IRC | 04:19 | |
| *** thuc has joined #openstack-infra | 04:19 | |
| *** ianw has joined #openstack-infra | 04:19 | |
| *** matsuhashi has joined #openstack-infra | 04:19 | |
| ianw | does anyone know off-hand how to add color to jenkins comments? | 04:20 |
| ianw | sorry, gerrit, not jenkins | 04:20 |
| clarkb | ianw: if you look in openstack-infra/config/modules/openstack_project/manifests/gerrit.pp you will see what we do to the comment links to color the jenkins results | 04:21 |
| clarkb | in gerrit | 04:21 |
| clarkb | and my irc box is updated. I suppose I will have to reboot it too :( back in a bit | 04:21 |
| *** clarkb has quit IRC | 04:22 | |
| zaro | ianw: i think gerrit comments support html markup. | 04:22 |
| *** wchrisj_ has joined #openstack-infra | 04:23 | |
| *** clarkb has joined #openstack-infra | 04:25 | |
| *** sabari has joined #openstack-infra | 04:27 | |
| clarkb | that was fun | 04:27 |
| clarkb | I suppose I should kick back for now and get ready for tomorrow | 04:27 |
| ianw | clarkb: clearly i'm missing something about it https://review.openstack.org/#/c/50776/ | 04:28 |
| ianw | clarkb: oh, hang on, i see, if i write in that format it will markup for me | 04:29 |
| clarkb | right | 04:29 |
| clarkb | it escapes html so you have to do it in special format that gets formated for you | 04:30 |
| *** markmcclain has joined #openstack-infra | 04:31 | |
| *** Ryan_Lane has joined #openstack-infra | 04:33 | |
| *** Ryan_Lane has joined #openstack-infra | 04:33 | |
| *** wchrisj_ has quit IRC | 04:33 | |
| *** Ryan_Lane1 has joined #openstack-infra | 04:36 | |
| *** Ryan_Lane has quit IRC | 04:36 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 04:36 | |
| *** Ryan_Lane has quit IRC | 04:36 | |
| *** Ryan_Lane has joined #openstack-infra | 04:36 | |
| *** yfried has quit IRC | 04:38 | |
| ianw | clarkb: success! or should I say SUCCESS (https://review.openstack.org/#/c/50776/) :) | 04:39 |
| *** wchrisj has joined #openstack-infra | 04:39 | |
| clarkb | woot | 04:39 |
| *** wchrisj has quit IRC | 04:45 | |
| *** imcsk8_lap has quit IRC | 04:50 | |
| *** imcsk8_lap has joined #openstack-infra | 04:50 | |
| *** Ryan_Lane1 has joined #openstack-infra | 04:52 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Fix for not loading middleware https://review.openstack.org/85693 | 04:52 |
| *** nati_ueno has joined #openstack-infra | 04:53 | |
| *** thuc_ has joined #openstack-infra | 04:54 | |
| SergeyLukjanov | clarkb, evening | 04:55 |
| clarkb | SergeyLukjanov: hello | 04:55 |
| SergeyLukjanov | clarkb, is there any agreement on non-merging nodepool patches now? | 04:55 |
| *** wchrisj has joined #openstack-infra | 04:56 | |
| SergeyLukjanov | (I've read scrollback, but probably missed something) | 04:56 |
| SergeyLukjanov | https://review.openstack.org/#/c/83975/6 | 04:56 |
| clarkb | SergeyLukjanov: it probably just needs someone to babysit, that said we are probably going to be swamped babysitting updates to deal with the openssl vulnerabilities | 04:57 |
| *** thuc has quit IRC | 04:57 | |
| SergeyLukjanov | clarkb, oh, yup, thanks | 04:57 |
| *** thuc_ has quit IRC | 04:58 | |
| *** wchrisj has quit IRC | 05:00 | |
| *** timrc-afk is now known as timrc | 05:02 | |
| *** harlowja is now known as harlowja_away | 05:03 | |
| *** skraynev_afk is now known as skraynev | 05:03 | |
| *** mihgen has joined #openstack-infra | 05:04 | |
| *** dkliban has quit IRC | 05:06 | |
| SergeyLukjanov | clarkb, re openssl, is it re https://www.openssl.org/news/secadv_20140407.txt? | 05:07 |
| *** Ryan_Lane1 has quit IRC | 05:08 | |
| *** amotoki has joined #openstack-infra | 05:08 | |
| clarkb | SergeyLukjanov: yes, so far today we updated openssl on all of our nodes and rebooted them to make sure nothing was sticking around using old openssl | 05:08 |
| *** ildikov_ has quit IRC | 05:08 | |
| clarkb | SergeyLukjanov: then we killed the puppetmaster CA and created a new oen, and recerted all puppet agents | 05:09 |
| *** zhiyan_ is now known as zhiyan | 05:09 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix sftp access on CentOS machines https://review.openstack.org/85924 | 05:11 |
| mordred | clarkb: I just found the silliest thing ^^ | 05:11 |
| *** oomichi has joined #openstack-infra | 05:11 | |
| *** timrc is now known as timrc-afk | 05:12 | |
| *** chrisstreeter_ has quit IRC | 05:12 | |
| *** markmcclain has quit IRC | 05:13 | |
| mordred | SergeyLukjanov: ^^ if you get a sec | 05:18 |
| *** alff has joined #openstack-infra | 05:18 | |
| *** alff_ has joined #openstack-infra | 05:18 | |
| clarkb | mordred: it failed linting I -1'd with comment on how to fix | 05:18 |
| mordred | clarkb: thanks. I have tested it for functionality, btw | 05:19 |
| clarkb | oh good, you run that against one of the git* boxes? | 05:19 |
| SergeyLukjanov | mordred, /me looking | 05:19 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix sftp access on CentOS machines https://review.openstack.org/85924 | 05:19 |
| mordred | rocket ships fixed | 05:20 |
| *** pcrews has quit IRC | 05:20 | |
| SergeyLukjanov | mordred, +2 | 05:22 |
| SergeyLukjanov | mordred, /me not approving due to the probably need to babysit / check after merge | 05:22 |
| mordred | SergeyLukjanov: thanks. I'll babysit - I want it to work for something else | 05:25 |
| *** saper has quit IRC | 05:27 | |
| *** saju_m has joined #openstack-infra | 05:28 | |
| *** Ryan_Lane1 has joined #openstack-infra | 05:28 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix sftp access on CentOS machines https://review.openstack.org/85924 | 05:30 |
| *** yfried has joined #openstack-infra | 05:31 | |
| *** ildikov_ has joined #openstack-infra | 05:32 | |
| *** pcrews has joined #openstack-infra | 05:32 | |
| *** dolphm has quit IRC | 05:33 | |
| *** dolphm has joined #openstack-infra | 05:34 | |
| *** _nadya_ has joined #openstack-infra | 05:37 | |
| *** dolphm has quit IRC | 05:38 | |
| *** dolphm has joined #openstack-infra | 05:40 | |
| *** salv-orlando has joined #openstack-infra | 05:41 | |
| *** _nadya_ has quit IRC | 05:42 | |
| *** jlibosva has joined #openstack-infra | 05:44 | |
| *** salv-orlando has quit IRC | 05:45 | |
| *** maxbit has joined #openstack-infra | 05:49 | |
| *** Clabbe has joined #openstack-infra | 05:50 | |
| *** Ryan_Lane1 has quit IRC | 05:52 | |
| *** salv-orlando has joined #openstack-infra | 05:53 | |
| *** oomichi is now known as ken1ohmichi | 05:58 | |
| *** ominakov has joined #openstack-infra | 05:58 | |
| *** wenlock_ has quit IRC | 06:01 | |
| *** wenlock has quit IRC | 06:02 | |
| *** _nadya_ has joined #openstack-infra | 06:04 | |
| *** thuc has joined #openstack-infra | 06:04 | |
| *** unicell has joined #openstack-infra | 06:05 | |
| *** thuc has quit IRC | 06:09 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add example of comment formatting https://review.openstack.org/85930 | 06:10 |
| *** saper has joined #openstack-infra | 06:12 | |
| mordred | http://puppetdb.openstack.org/report/review.openstack.org/486b614ed63da1b70b9a4726b0aff66e28550a16 | 06:16 |
| mordred | somethign has gone strange on hosts with jeepyb | 06:16 |
| mordred | I have looked at it in person and can't tell what it's trying to do - but it looks like it's lost track of /opt/jeepyb beinga git repo | 06:17 |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Also clone os-cloud-config in prepare_tripleo.sh https://review.openstack.org/85932 | 06:18 |
| openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 06:19 |
| *** salv-orlando has quit IRC | 06:20 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add sphix deps to setup.py https://review.openstack.org/85933 | 06:21 |
| StevenK | ianw: Sphix? :-) | 06:22 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add sphinx deps to setup.py https://review.openstack.org/85933 | 06:23 |
| ianw | StevenK: oops, sorry. pretty sure the patch isn't right, but it's not really very clear how to build the docs if you want to test your change | 06:23 |
| mordred | ianw: you want "tox -evenv python setup.py build_sphinx" | 06:23 |
| ianw | mordred: ah, ok, thanks | 06:28 |
| ianw | i was avoiding tox because it has a hard requirement on > 1.6 which isn't in f19 | 06:28 |
| *** denis_makogon has joined #openstack-infra | 06:29 | |
| *** _nadya_ has quit IRC | 06:32 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add note on ci.openstack.org source https://review.openstack.org/85933 | 06:33 |
| *** zehicle has quit IRC | 06:35 | |
| *** zehicle_at_dell has quit IRC | 06:35 | |
| *** zehicle has joined #openstack-infra | 06:36 | |
| *** Sukhdev has joined #openstack-infra | 06:41 | |
| *** flaper87|afk is now known as flaper87 | 06:44 | |
| *** jamielennox is now known as jamielennox|away | 06:48 | |
| *** Ryan_Lane1 has joined #openstack-infra | 06:50 | |
| *** dizquierdo has joined #openstack-infra | 07:00 | |
| *** reed has joined #openstack-infra | 07:00 | |
| *** yolanda has joined #openstack-infra | 07:00 | |
| *** andreykurilin_ has joined #openstack-infra | 07:02 | |
| *** maxbit has quit IRC | 07:11 | |
| *** saju_m has quit IRC | 07:17 | |
| *** Ryan_Lane1 has quit IRC | 07:18 | |
| *** jcoufal has joined #openstack-infra | 07:23 | |
| *** e0ne has joined #openstack-infra | 07:23 | |
| *** mrda is now known as mrda_away | 07:25 | |
| *** andreykurilin_ has quit IRC | 07:25 | |
| *** saju_m has joined #openstack-infra | 07:25 | |
| *** flaper87 is now known as flaper87|afk | 07:26 | |
| *** jgallard has joined #openstack-infra | 07:27 | |
| *** ttx has quit IRC | 07:32 | |
| *** ttx has joined #openstack-infra | 07:32 | |
| *** ttx has quit IRC | 07:32 | |
| *** ttx has joined #openstack-infra | 07:32 | |
| *** saju_m has quit IRC | 07:32 | |
| *** alff has quit IRC | 07:35 | |
| *** alff_ has quit IRC | 07:35 | |
| *** morganfainberg is now known as morganfainberg_Z | 07:36 | |
| *** Sukhdev has quit IRC | 07:37 | |
| *** jp_at_hp has joined #openstack-infra | 07:40 | |
| *** sabari has quit IRC | 07:43 | |
| *** reed_ has joined #openstack-infra | 07:43 | |
| *** reed has quit IRC | 07:46 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 07:46 |
| *** dutsmoc has joined #openstack-infra | 07:49 | |
| *** ttx has quit IRC | 07:50 | |
| *** ttx has joined #openstack-infra | 07:50 | |
| *** ttx has quit IRC | 07:50 | |
| *** ttx has joined #openstack-infra | 07:50 | |
| *** matsuhashi has quit IRC | 07:53 | |
| *** ttx has quit IRC | 07:53 | |
| *** ttx has joined #openstack-infra | 07:53 | |
| *** ttx has quit IRC | 07:53 | |
| *** ttx has joined #openstack-infra | 07:53 | |
| ttx | yay, key rotation day | 07:54 |
| *** matsuhashi has joined #openstack-infra | 07:54 | |
| *** hashar has joined #openstack-infra | 07:55 | |
| *** e0ne has quit IRC | 07:55 | |
| *** andreaf has joined #openstack-infra | 07:56 | |
| *** e0ne has joined #openstack-infra | 07:56 | |
| *** saju_m has joined #openstack-infra | 07:58 | |
| *** ociuhandu has quit IRC | 07:59 | |
| *** e0ne has quit IRC | 08:00 | |
| *** derekh has joined #openstack-infra | 08:02 | |
| *** saju_m has quit IRC | 08:03 | |
| *** Ryan_Lane1 has joined #openstack-infra | 08:04 | |
| *** Ryan_Lane1 has quit IRC | 08:04 | |
| *** saschpe has joined #openstack-infra | 08:04 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Missing docstrings added https://review.openstack.org/85718 | 08:06 |
| *** jpich has joined #openstack-infra | 08:07 | |
| *** mihgen has quit IRC | 08:09 | |
| *** Ryan_Lane has quit IRC | 08:09 | |
| *** nati_ueno has quit IRC | 08:11 | |
| *** vponomaryov has joined #openstack-infra | 08:14 | |
| *** saju_m has joined #openstack-infra | 08:15 | |
| *** chandan_kumar has quit IRC | 08:15 | |
| SergeyLukjanov | ttx, yup, or several days :) | 08:17 |
| *** nati_ueno has joined #openstack-infra | 08:18 | |
| *** nati_ueno has quit IRC | 08:18 | |
| *** sweston has quit IRC | 08:19 | |
| *** sweston has joined #openstack-infra | 08:20 | |
| *** comstud has quit IRC | 08:20 | |
| *** dutsmoc is now known as comstud | 08:20 | |
| *** che-arne has quit IRC | 08:21 | |
| *** ihrachyshka has joined #openstack-infra | 08:24 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make transifex the only source of translations https://review.openstack.org/84191 | 08:26 |
| *** nprivalova has quit IRC | 08:29 | |
| *** ociuhandu has joined #openstack-infra | 08:31 | |
| *** _nadya_ has joined #openstack-infra | 08:37 | |
| *** sweston has quit IRC | 08:37 | |
| *** sweston has joined #openstack-infra | 08:38 | |
| *** afazekas has joined #openstack-infra | 08:42 | |
| *** jooools has joined #openstack-infra | 08:44 | |
| *** andreaf has quit IRC | 08:45 | |
| *** comstud has quit IRC | 08:48 | |
| *** comstud has joined #openstack-infra | 08:48 | |
| *** alexpilotti has joined #openstack-infra | 08:50 | |
| *** Ryan_Lane has joined #openstack-infra | 08:54 | |
| *** comstud has quit IRC | 08:57 | |
| *** mihgen has joined #openstack-infra | 09:00 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Minor fixes in the installation documentation https://review.openstack.org/85886 | 09:01 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Sort ant builders settings for consistency https://review.openstack.org/82492 | 09:01 |
| *** Ryan_Lane has quit IRC | 09:03 | |
| *** comstud has joined #openstack-infra | 09:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add jobs to publish nodepool docs https://review.openstack.org/84857 | 09:08 |
| *** pcrews_ has joined #openstack-infra | 09:09 | |
| *** pcrews has quit IRC | 09:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Disable pypy jobs in ironic-python-agent https://review.openstack.org/85787 | 09:12 |
| *** e0ne has joined #openstack-infra | 09:13 | |
| *** andreaf has joined #openstack-infra | 09:13 | |
| *** denis_makogon has quit IRC | 09:13 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix some minor typos in the trigger documentation https://review.openstack.org/85885 | 09:14 |
| *** pcrews_ has quit IRC | 09:19 | |
| *** sweston has quit IRC | 09:21 | |
| *** pcrews_ has joined #openstack-infra | 09:21 | |
| *** ihrachyshka has quit IRC | 09:26 | |
| *** comstud has quit IRC | 09:28 | |
| rcarrillocruz | clarkb , jeblair, fungi, mordred : hi guys, any of you still around? | 09:28 |
| *** comstud has joined #openstack-infra | 09:28 | |
| rcarrillocruz | i ran the review.pp manifest in a VM and I notice that it created home folders for your users, is this expected? Otherwise I'd open a bug and look at the manifests to see where those user home folders are created.... | 09:29 |
| *** e0ne_ has joined #openstack-infra | 09:30 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Sort ant builders settings for consistency https://review.openstack.org/82492 | 09:30 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not use obsolete translations https://review.openstack.org/85973 | 09:32 |
| *** e0ne has quit IRC | 09:33 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not generate obsolete translations https://review.openstack.org/85973 | 09:33 |
| *** sweston has joined #openstack-infra | 09:34 | |
| *** comstud has quit IRC | 09:39 | |
| *** saju_m has quit IRC | 09:39 | |
| *** saschpe has quit IRC | 09:39 | |
| *** matsuhashi has quit IRC | 09:39 | |
| *** ttx has quit IRC | 09:39 | |
| *** dolphm has quit IRC | 09:39 | |
| *** Shrews has quit IRC | 09:39 | |
| *** mattoliverau has quit IRC | 09:39 | |
| *** yamahata has quit IRC | 09:39 | |
| *** jhesketh has quit IRC | 09:39 | |
| *** mgagne has quit IRC | 09:39 | |
| *** harlowja_away has quit IRC | 09:39 | |
| *** russellb has quit IRC | 09:39 | |
| *** sahumada has quit IRC | 09:39 | |
| *** zul has quit IRC | 09:39 | |
| *** bogdando has quit IRC | 09:39 | |
| *** annegentle_ has quit IRC | 09:39 | |
| *** markmc has quit IRC | 09:39 | |
| *** adam_g has quit IRC | 09:39 | |
| *** stevebaker has quit IRC | 09:39 | |
| *** jbryce has quit IRC | 09:39 | |
| *** thingee has quit IRC | 09:39 | |
| *** wendar_ has quit IRC | 09:39 | |
| *** Guest21911 has quit IRC | 09:39 | |
| *** pfallenop has quit IRC | 09:39 | |
| *** nibalizer has quit IRC | 09:39 | |
| *** dstanek has quit IRC | 09:39 | |
| *** russell_h has quit IRC | 09:39 | |
| *** jlk has quit IRC | 09:39 | |
| *** rainya- has quit IRC | 09:39 | |
| *** hughsaunders has quit IRC | 09:39 | |
| *** zz_gondoi has quit IRC | 09:39 | |
| *** Steap has quit IRC | 09:39 | |
| *** gaelL has quit IRC | 09:39 | |
| *** dguerri_ has quit IRC | 09:39 | |
| *** dtroyer has quit IRC | 09:39 | |
| *** alaski has quit IRC | 09:39 | |
| *** gmoro has quit IRC | 09:39 | |
| *** jroll has quit IRC | 09:39 | |
| *** EmilienM has quit IRC | 09:39 | |
| *** pvo has quit IRC | 09:39 | |
| *** uvirtbot has quit IRC | 09:39 | |
| *** westmaas has quit IRC | 09:39 | |
| *** tteggel has quit IRC | 09:39 | |
| *** dosaboy has quit IRC | 09:39 | |
| *** dhellmann has quit IRC | 09:39 | |
| *** flaper87|afk has quit IRC | 09:39 | |
| *** nijaba has quit IRC | 09:39 | |
| *** odyi has quit IRC | 09:39 | |
| *** tristanC has quit IRC | 09:39 | |
| *** dangers_away has quit IRC | 09:39 | |
| *** milki has quit IRC | 09:39 | |
| *** maurosr has quit IRC | 09:39 | |
| *** dragondm has quit IRC | 09:39 | |
| *** stevebaker has joined #openstack-infra | 09:39 | |
| *** adam_g has joined #openstack-infra | 09:39 | |
| *** adam_g has quit IRC | 09:39 | |
| *** adam_g has joined #openstack-infra | 09:39 | |
| *** pfallenop has joined #openstack-infra | 09:39 | |
| *** milki has joined #openstack-infra | 09:39 | |
| *** markmc has joined #openstack-infra | 09:39 | |
| *** odyi has joined #openstack-infra | 09:39 | |
| *** flaper87|afk has joined #openstack-infra | 09:39 | |
| *** Steap has joined #openstack-infra | 09:39 | |
| *** bogdando has joined #openstack-infra | 09:39 | |
| *** gmoro has joined #openstack-infra | 09:39 | |
| *** matsuhashi has joined #openstack-infra | 09:39 | |
| *** mattoliverau has joined #openstack-infra | 09:39 | |
| *** EmilienM has joined #openstack-infra | 09:39 | |
| *** tteggel has joined #openstack-infra | 09:39 | |
| *** saju_m has joined #openstack-infra | 09:39 | |
| *** odyi has quit IRC | 09:39 | |
| *** bogdando has quit IRC | 09:39 | |
| *** sweston has quit IRC | 09:39 | |
| *** pcrews_ has quit IRC | 09:39 | |
| *** jcoufal has quit IRC | 09:39 | |
| *** yfried has quit IRC | 09:39 | |
| *** clarkb has quit IRC | 09:39 | |
| *** david-lyle has quit IRC | 09:39 | |
| *** openstackgerrit has quit IRC | 09:39 | |
| *** SumitNaiksatam has quit IRC | 09:39 | |
| *** jamespage has quit IRC | 09:39 | |
| *** zhiyan has quit IRC | 09:39 | |
| *** moted has quit IRC | 09:39 | |
| *** sdake has quit IRC | 09:39 | |
| *** thomasem has quit IRC | 09:39 | |
| *** rfolco has quit IRC | 09:39 | |
| *** miarmak has quit IRC | 09:39 | |
| *** fifieldt has quit IRC | 09:39 | |
| *** kevinbenton has quit IRC | 09:39 | |
| *** slagle has quit IRC | 09:39 | |
| *** flaper87|afk is now known as flaper87 | 09:39 | |
| *** miarmak has joined #openstack-infra | 09:39 | |
| *** zul has joined #openstack-infra | 09:39 | |
| *** openstack has joined #openstack-infra | 09:50 | |
| *** unicell has joined #openstack-infra | 09:50 | |
| *** kiall has joined #openstack-infra | 09:51 | |
| *** soren has joined #openstack-infra | 09:51 | |
| *** che-arne has joined #openstack-infra | 09:52 | |
| *** wayneeseguin has joined #openstack-infra | 09:52 | |
| *** matsuhashi has quit IRC | 09:52 | |
| *** mikal has joined #openstack-infra | 09:52 | |
| *** bnemec has joined #openstack-infra | 09:52 | |
| *** flaper87 has quit IRC | 09:53 | |
| *** flaper87 has joined #openstack-infra | 09:53 | |
| *** slagle has joined #openstack-infra | 09:53 | |
| *** koolhead17 has joined #openstack-infra | 09:53 | |
| *** matsuhashi has joined #openstack-infra | 09:53 | |
| *** jp_at_hp has quit IRC | 09:55 | |
| *** jp_at_hp has joined #openstack-infra | 09:55 | |
| *** adarazs has quit IRC | 09:56 | |
| *** adarazs has joined #openstack-infra | 09:56 | |
| *** jeckersb has quit IRC | 09:56 | |
| *** jeckersb has joined #openstack-infra | 09:56 | |
| *** yfried has quit IRC | 09:56 | |
| *** yfried has joined #openstack-infra | 09:56 | |
| *** SumitNaiksatam has quit IRC | 09:56 | |
| *** SumitNaiksatam has joined #openstack-infra | 09:56 | |
| *** jcoufal has quit IRC | 09:56 | |
| *** jcoufal has joined #openstack-infra | 09:56 | |
| *** dangers_away has quit IRC | 09:56 | |
| *** dangers_away has joined #openstack-infra | 09:56 | |
| *** kmartin has quit IRC | 09:57 | |
| *** kmartin has joined #openstack-infra | 09:57 | |
| *** tteggel has quit IRC | 09:57 | |
| *** tteggel has joined #openstack-infra | 09:57 | |
| *** mkerrin has quit IRC | 09:57 | |
| *** mkerrin has joined #openstack-infra | 09:57 | |
| *** maurosr has quit IRC | 09:57 | |
| *** maurosr has joined #openstack-infra | 09:57 | |
| *** jlibosva has quit IRC | 09:57 | |
| *** jlibosva has joined #openstack-infra | 09:57 | |
| *** che-arne has quit IRC | 09:59 | |
| *** che-arne has joined #openstack-infra | 09:59 | |
| *** Ryan_Lane has joined #openstack-infra | 09:59 | |
| *** persia has quit IRC | 10:00 | |
| *** persia has joined #openstack-infra | 10:01 | |
| *** persia has quit IRC | 10:01 | |
| *** persia has joined #openstack-infra | 10:01 | |
| *** persia is now known as Guest17198 | 10:01 | |
| *** Guest17198 has quit IRC | 10:01 | |
| *** Guest17198 has joined #openstack-infra | 10:01 | |
| *** Guest17198 is now known as persia | 10:02 | |
| *** persia has quit IRC | 10:03 | |
| *** wendar has quit IRC | 10:03 | |
| *** sileht has quit IRC | 10:03 | |
| *** greghaynes has quit IRC | 10:03 | |
| *** stevebaker has quit IRC | 10:03 | |
| *** fungi has quit IRC | 10:03 | |
| *** adam_g has quit IRC | 10:03 | |
| *** sc68cal has quit IRC | 10:03 | |
| *** mrda_away has quit IRC | 10:03 | |
| *** zigo has quit IRC | 10:03 | |
| *** StevenK has quit IRC | 10:03 | |
| *** jdurgin has quit IRC | 10:03 | |
| *** sdague has quit IRC | 10:03 | |
| *** bknudson has quit IRC | 10:03 | |
| *** yongli has quit IRC | 10:03 | |
| *** gothicmindfood has quit IRC | 10:03 | |
| *** ruhe has quit IRC | 10:03 | |
| *** dansmith has quit IRC | 10:03 | |
| *** JoshNang has quit IRC | 10:03 | |
| *** pbelanyi has quit IRC | 10:03 | |
| *** ihrachyshka has joined #openstack-infra | 10:03 | |
| *** adam_g` has joined #openstack-infra | 10:03 | |
| *** persia has joined #openstack-infra | 10:03 | |
| *** stevebaker has joined #openstack-infra | 10:03 | |
| *** dansmith has joined #openstack-infra | 10:03 | |
| *** StevenK has joined #openstack-infra | 10:03 | |
| *** mrda_away has joined #openstack-infra | 10:03 | |
| *** JoshNang has joined #openstack-infra | 10:03 | |
| *** yongli has joined #openstack-infra | 10:03 | |
| *** greghaynes has joined #openstack-infra | 10:03 | |
| *** persia has quit IRC | 10:03 | |
| *** persia has joined #openstack-infra | 10:03 | |
| *** zhiyan is now known as zhiyan_ | 10:03 | |
| *** ruhe has joined #openstack-infra | 10:04 | |
| *** sdague has joined #openstack-infra | 10:04 | |
| *** fungi has joined #openstack-infra | 10:04 | |
| *** gothicmindfood has joined #openstack-infra | 10:04 | |
| *** sc68cal has joined #openstack-infra | 10:04 | |
| *** sileht has joined #openstack-infra | 10:04 | |
| *** wendar has joined #openstack-infra | 10:04 | |
| *** zigo has joined #openstack-infra | 10:04 | |
| *** Ryan_Lane has quit IRC | 10:04 | |
| *** jdurgin has joined #openstack-infra | 10:04 | |
| *** pbelanyi has joined #openstack-infra | 10:04 | |
| sdague | mikal: what's up with turbohipster being drunk? | 10:05 |
| *** Ng has quit IRC | 10:08 | |
| *** jgallard has quit IRC | 10:09 | |
| *** Ng has joined #openstack-infra | 10:09 | |
| *** yassine has joined #openstack-infra | 10:09 | |
| *** pvo has quit IRC | 10:13 | |
| *** jbryce has quit IRC | 10:13 | |
| *** russell_h has quit IRC | 10:13 | |
| *** annegentle has quit IRC | 10:13 | |
| *** dolphm has quit IRC | 10:13 | |
| *** wendar has quit IRC | 10:14 | |
| *** wendar has joined #openstack-infra | 10:14 | |
| *** StevenK has quit IRC | 10:15 | |
| *** ttx has quit IRC | 10:15 | |
| *** russellb has quit IRC | 10:15 | |
| *** rainya has quit IRC | 10:15 | |
| *** jlk has quit IRC | 10:15 | |
| *** gondoi has quit IRC | 10:15 | |
| *** Guest99168 has quit IRC | 10:15 | |
| *** fungi has quit IRC | 10:15 | |
| *** stevebaker has quit IRC | 10:15 | |
| *** adam_g` has quit IRC | 10:15 | |
| *** JayF has quit IRC | 10:15 | |
| *** EmilienM has quit IRC | 10:15 | |
| *** jroll has quit IRC | 10:15 | |
| *** comstud has quit IRC | 10:15 | |
| *** alaski has quit IRC | 10:15 | |
| *** StevenK has joined #openstack-infra | 10:15 | |
| *** reed_ has quit IRC | 10:16 | |
| *** oomichi has joined #openstack-infra | 10:16 | |
| *** mgagne has quit IRC | 10:16 | |
| *** _nadya_ has quit IRC | 10:16 | |
| *** adam_g has joined #openstack-infra | 10:20 | |
| *** adam_g has quit IRC | 10:20 | |
| *** adam_g has joined #openstack-infra | 10:20 | |
| *** stevebaker has joined #openstack-infra | 10:20 | |
| *** ttx has joined #openstack-infra | 10:20 | |
| *** JayF has joined #openstack-infra | 10:21 | |
| *** JayF has quit IRC | 10:21 | |
| *** JayF has joined #openstack-infra | 10:21 | |
| *** EmilienM has joined #openstack-infra | 10:22 | |
| *** NikitaKonovalov has quit IRC | 10:26 | |
| *** DinaBelova has quit IRC | 10:26 | |
| *** sdake_ has quit IRC | 10:26 | |
| *** medieval1 has quit IRC | 10:26 | |
| *** jhesketh has quit IRC | 10:26 | |
| *** bogdando has quit IRC | 10:26 | |
| *** 6JTAANJSJ has quit IRC | 10:26 | |
| *** uvirtbot has quit IRC | 10:26 | |
| *** zhiyan_ has quit IRC | 10:26 | |
| *** jcoufal has quit IRC | 10:26 | |
| *** michchap has quit IRC | 10:26 | |
| *** Shrews has quit IRC | 10:26 | |
| *** kevinbenton has quit IRC | 10:26 | |
| *** gema has quit IRC | 10:26 | |
| *** fifieldt has quit IRC | 10:26 | |
| *** andreaf has quit IRC | 10:26 | |
| *** alexpilotti has quit IRC | 10:26 | |
| *** jooools has quit IRC | 10:26 | |
| *** jp_at_hp has quit IRC | 10:26 | |
| *** ominakov has quit IRC | 10:26 | |
| *** amotoki has quit IRC | 10:26 | |
| *** imcsk8_lap has quit IRC | 10:26 | |
| *** yaguang has quit IRC | 10:26 | |
| *** adalbas has quit IRC | 10:26 | |
| *** ogelbukh has quit IRC | 10:26 | |
| *** eloib_ has quit IRC | 10:26 | |
| *** pabelanger has quit IRC | 10:26 | |
| *** primeministerp has quit IRC | 10:26 | |
| *** kashyap has quit IRC | 10:26 | |
| *** jgriffith has quit IRC | 10:26 | |
| *** locke105 has quit IRC | 10:26 | |
| *** Adri2000 has quit IRC | 10:26 | |
| *** mtreinish has quit IRC | 10:26 | |
| *** katyafervent has quit IRC | 10:26 | |
| *** mburned has quit IRC | 10:26 | |
| *** hdd has quit IRC | 10:26 | |
| *** mika has quit IRC | 10:26 | |
| *** vladan has quit IRC | 10:26 | |
| *** Mithrandir has quit IRC | 10:26 | |
| *** spredzy has quit IRC | 10:26 | |
| *** rcarrillocruz has quit IRC | 10:26 | |
| *** simonmcc_ has quit IRC | 10:26 | |
| *** kevinbenton has joined #openstack-infra | 10:27 | |
| *** imcsk8_lap has joined #openstack-infra | 10:27 | |
| *** gema has joined #openstack-infra | 10:27 | |
| *** mika has joined #openstack-infra | 10:27 | |
| *** mika has joined #openstack-infra | 10:27 | |
| *** simonmcc_ has joined #openstack-infra | 10:27 | |
| *** zehicle has joined #openstack-infra | 10:27 | |
| *** pabelanger has joined #openstack-infra | 10:27 | |
| *** uvirtbot has joined #openstack-infra | 10:27 | |
| *** hdd has joined #openstack-infra | 10:27 | |
| *** ogelbukh has joined #openstack-infra | 10:27 | |
| *** adalbas has joined #openstack-infra | 10:27 | |
| *** katyafervent has joined #openstack-infra | 10:27 | |
| *** mtreinish has joined #openstack-infra | 10:27 | |
| *** bogdando has joined #openstack-infra | 10:27 | |
| *** amotoki has joined #openstack-infra | 10:27 | |
| *** jp_at_hp has joined #openstack-infra | 10:27 | |
| *** rcarrillocruz has joined #openstack-infra | 10:27 | |
| *** sdake_ has joined #openstack-infra | 10:27 | |
| *** spredzy has joined #openstack-infra | 10:27 | |
| *** simonmcc_ has quit IRC | 10:27 | |
| *** simonmcc_ has joined #openstack-infra | 10:27 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:27 | |
| *** michchap has joined #openstack-infra | 10:27 | |
| *** ominakov has joined #openstack-infra | 10:27 | |
| *** jooools has joined #openstack-infra | 10:27 | |
| *** alexpilotti has joined #openstack-infra | 10:27 | |
| *** andreaf has joined #openstack-infra | 10:27 | |
| *** jgriffit1 has joined #openstack-infra | 10:27 | |
| *** Mithrandir has joined #openstack-infra | 10:27 | |
| *** jp_at_hp has quit IRC | 10:27 | |
| *** jp_at_hp has joined #openstack-infra | 10:27 | |
| *** vladan has joined #openstack-infra | 10:27 | |
| *** andreaf has quit IRC | 10:27 | |
| *** andreaf has joined #openstack-infra | 10:27 | |
| *** medieval1 has joined #openstack-infra | 10:27 | |
| *** hdd has quit IRC | 10:27 | |
| *** hdd has joined #openstack-infra | 10:27 | |
| *** jp_at_hp has quit IRC | 10:27 | |
| *** jp_at_hp has joined #openstack-infra | 10:27 | |
| *** jp_at_hp has quit IRC | 10:28 | |
| *** jp_at_hp has joined #openstack-infra | 10:28 | |
| *** Shrews has joined #openstack-infra | 10:28 | |
| *** zhiyan_ has joined #openstack-infra | 10:28 | |
| *** DinaBelova has joined #openstack-infra | 10:28 | |
| mikal | sdague: turbo-hipster had some issues with the grizzly branch being deleted from git. They should be resolved now. | 10:28 |
| *** kashyap has joined #openstack-infra | 10:28 | |
| *** yaguang has joined #openstack-infra | 10:28 | |
| sdague | ok, it just failed me again on master | 10:28 |
| *** jcoufal has joined #openstack-infra | 10:28 | |
| sdague | unless it doesn't do rechecks? | 10:29 |
| *** jhesketh has joined #openstack-infra | 10:29 | |
| *** fifieldt has joined #openstack-infra | 10:29 | |
| sdague | I guess that's an old vote | 10:29 |
| *** fungi has joined #openstack-infra | 10:29 | |
| *** eloib_ has joined #openstack-infra | 10:29 | |
| sdague | mikal: it worth retriggering t-h results on all the -1s? I know I tend to filter out -1s so all the changes from yesterday will be masked for a lot of people | 10:30 |
| *** Adri2000 has joined #openstack-infra | 10:30 | |
| *** Adri2000 has quit IRC | 10:30 | |
| *** Adri2000 has joined #openstack-infra | 10:31 | |
| *** locke105 has joined #openstack-infra | 10:31 | |
| *** Adri2000 is now known as Guest44106 | 10:31 | |
| mikal | sdague: I believe jhesketh has already done that | 10:31 |
| mikal | sdague: recheck is definitely supported | 10:31 |
| mikal | Got a review URL I can take a look at? | 10:31 |
| jhesketh | sdague: yep, should re re-running | 10:31 |
| jhesketh | http://zuul.rcbops.com | 10:32 |
| sdague | mikal: https://review.openstack.org/#/c/85780/ | 10:32 |
| mikal | LOL, that guy | 10:32 |
| mikal | Ahhh, failure was from before jhesketh fixed this. The recheck is still running. | 10:33 |
| *** primeministerp has joined #openstack-infra | 10:34 | |
| *** saju_m has quit IRC | 10:34 | |
| *** saju_m has joined #openstack-infra | 10:35 | |
| *** mburned has joined #openstack-infra | 10:36 | |
| *** reed has joined #openstack-infra | 10:36 | |
| sdague | ok, no prob | 10:36 |
| mikal | Sorry for any inconvenience, and please come again! | 10:36 |
| mikal | jhesketh: we should file a bug to make sure this doesn't happen again | 10:36 |
| *** Ryan_Lane has joined #openstack-infra | 10:36 | |
| *** rcarrillocruz has quit IRC | 10:39 | |
| *** zehicle has quit IRC | 10:39 | |
| *** sdake_ has quit IRC | 10:39 | |
| *** mika has quit IRC | 10:39 | |
| *** gothicmindfood has quit IRC | 10:39 | |
| *** slagle has quit IRC | 10:39 | |
| *** soren has quit IRC | 10:39 | |
| *** bookwar has quit IRC | 10:39 | |
| *** maurosr has quit IRC | 10:39 | |
| *** vponomaryov has quit IRC | 10:39 | |
| *** tteggel has quit IRC | 10:39 | |
| *** aglarendil has quit IRC | 10:39 | |
| *** sdake has quit IRC | 10:39 | |
| *** jamespage has quit IRC | 10:39 | |
| *** antonym has quit IRC | 10:39 | |
| *** david-lyle has quit IRC | 10:39 | |
| *** yfried has quit IRC | 10:39 | |
| *** jeckersb has quit IRC | 10:39 | |
| *** miarmak has quit IRC | 10:39 | |
| *** mattoliverau has quit IRC | 10:39 | |
| *** flaper87 has quit IRC | 10:39 | |
| *** pfallenop has quit IRC | 10:39 | |
| *** ociuhandu has quit IRC | 10:39 | |
| *** hashar has quit IRC | 10:39 | |
| *** dcramer_ has quit IRC | 10:39 | |
| *** msabramo has quit IRC | 10:39 | |
| *** gnuoy has quit IRC | 10:39 | |
| *** MIDENN_ has quit IRC | 10:39 | |
| *** Guest456 has quit IRC | 10:39 | |
| *** GheRivero has quit IRC | 10:39 | |
| *** mwagner_lap has quit IRC | 10:39 | |
| *** asettle has quit IRC | 10:39 | |
| *** jamespage has joined #openstack-infra | 10:39 | |
| *** mika has joined #openstack-infra | 10:39 | |
| *** mika has quit IRC | 10:39 | |
| *** mika has joined #openstack-infra | 10:39 | |
| *** msabramo has joined #openstack-infra | 10:39 | |
| *** jamespage has quit IRC | 10:39 | |
| *** jamespage has joined #openstack-infra | 10:39 | |
| *** slagle has joined #openstack-infra | 10:39 | |
| *** aglarendil has joined #openstack-infra | 10:39 | |
| *** mattoliverau has joined #openstack-infra | 10:40 | |
| *** GheRivero has joined #openstack-infra | 10:40 | |
| *** jeckersb has joined #openstack-infra | 10:40 | |
| *** miarmak has joined #openstack-infra | 10:40 | |
| *** dcramer_ has joined #openstack-infra | 10:40 | |
| *** zehicle has joined #openstack-infra | 10:40 | |
| *** yfried has joined #openstack-infra | 10:40 | |
| *** MIDENN_ has joined #openstack-infra | 10:40 | |
| *** gnuoy has joined #openstack-infra | 10:40 | |
| *** rcarrillocruz has joined #openstack-infra | 10:40 | |
| *** gothicmindfood has joined #openstack-infra | 10:40 | |
| *** soren has joined #openstack-infra | 10:40 | |
| *** asettle has joined #openstack-infra | 10:40 | |
| *** mwagner_lap has joined #openstack-infra | 10:40 | |
| *** sdake_ has joined #openstack-infra | 10:40 | |
| *** sdake has joined #openstack-infra | 10:40 | |
| *** Vivek has joined #openstack-infra | 10:40 | |
| *** antonym has joined #openstack-infra | 10:40 | |
| *** vponomaryov has joined #openstack-infra | 10:40 | |
| *** hashar has joined #openstack-infra | 10:40 | |
| *** jeckersb has quit IRC | 10:40 | |
| *** jeckersb has joined #openstack-infra | 10:40 | |
| *** david-lyle has joined #openstack-infra | 10:40 | |
| *** ociuhandu has joined #openstack-infra | 10:40 | |
| *** dcramer_ has quit IRC | 10:40 | |
| *** dcramer_ has joined #openstack-infra | 10:40 | |
| *** pfallenop has joined #openstack-infra | 10:40 | |
| *** MIDENN_ has quit IRC | 10:40 | |
| *** MIDENN_ has joined #openstack-infra | 10:40 | |
| *** flaper87 has joined #openstack-infra | 10:40 | |
| *** yfried has quit IRC | 10:40 | |
| *** yfried has joined #openstack-infra | 10:40 | |
| *** Vivek is now known as Guest26771 | 10:40 | |
| *** bookwar has joined #openstack-infra | 10:40 | |
| *** tteggel has joined #openstack-infra | 10:40 | |
| *** maurosr has joined #openstack-infra | 10:40 | |
| *** flaper87 has quit IRC | 10:40 | |
| *** flaper87 has joined #openstack-infra | 10:40 | |
| *** maurosr has quit IRC | 10:40 | |
| *** maurosr has joined #openstack-infra | 10:40 | |
| *** flaper87 has quit IRC | 10:40 | |
| *** flaper87 has joined #openstack-infra | 10:40 | |
| *** tteggel has quit IRC | 10:41 | |
| *** tteggel has joined #openstack-infra | 10:41 | |
| *** Ryan_Lane has quit IRC | 10:41 | |
| jhesketh | sdague: there you go, turbo-hipster +1 | 10:42 |
| jhesketh | :-) | 10:42 |
| jhesketh | mikal: do you mean a bug for why it failed? | 10:42 |
| mikal | jhesketh: yeah, a bug to remind us to try for tags as well as branches | 10:42 |
| *** AJaeger has joined #openstack-infra | 10:44 | |
| * mikal goes back to his movie | 10:44 | |
| jhesketh | righto | 10:45 |
| *** matsuhashi has quit IRC | 10:53 | |
| *** matsuhashi has joined #openstack-infra | 10:54 | |
| *** sdake has quit IRC | 10:57 | |
| *** sandywalsh has joined #openstack-infra | 10:58 | |
| *** matsuhashi has quit IRC | 10:58 | |
| *** matsuhashi has joined #openstack-infra | 10:59 | |
| *** ildikov_ has joined #openstack-infra | 11:02 | |
| *** chandan_kumar has joined #openstack-infra | 11:03 | |
| *** hashar has quit IRC | 11:14 | |
| *** julienvey has joined #openstack-infra | 11:21 | |
| *** e0ne_ has quit IRC | 11:22 | |
| *** e0ne has joined #openstack-infra | 11:23 | |
| *** e0ne has quit IRC | 11:27 | |
| *** sdake has joined #openstack-infra | 11:31 | |
| *** yaguang has quit IRC | 11:32 | |
| *** Ryan_Lane has joined #openstack-infra | 11:37 | |
| *** weshay has joined #openstack-infra | 11:41 | |
| *** Ryan_Lane has quit IRC | 11:42 | |
| *** chandan_kumar has quit IRC | 11:44 | |
| *** alaski has joined #openstack-infra | 11:44 | |
| *** jroll has joined #openstack-infra | 11:45 | |
| *** zz_gondoi has joined #openstack-infra | 11:46 | |
| *** dolphm has joined #openstack-infra | 11:46 | |
| *** annegentle has joined #openstack-infra | 11:46 | |
| *** zz_gondoi is now known as gondoi | 11:46 | |
| *** russellb has joined #openstack-infra | 11:46 | |
| *** jbryce has joined #openstack-infra | 11:46 | |
| *** russell_h has joined #openstack-infra | 11:46 | |
| *** salv-orlando has joined #openstack-infra | 11:47 | |
| *** mgagne has joined #openstack-infra | 11:48 | |
| *** lillie has joined #openstack-infra | 11:48 | |
| *** pvo has joined #openstack-infra | 11:48 | |
| *** lillie is now known as Guest19269 | 11:49 | |
| *** jlk has joined #openstack-infra | 11:52 | |
| *** jlk has joined #openstack-infra | 11:52 | |
| *** fifieldt has quit IRC | 11:52 | |
| *** comstud has joined #openstack-infra | 11:52 | |
| *** e0ne has joined #openstack-infra | 11:53 | |
| *** jgallard has joined #openstack-infra | 11:55 | |
| *** e0ne has quit IRC | 11:58 | |
| *** pdmars has joined #openstack-infra | 11:59 | |
| *** mbacchi has joined #openstack-infra | 12:00 | |
| *** markmcclain has joined #openstack-infra | 12:00 | |
| *** chandan_kumar has joined #openstack-infra | 12:02 | |
| *** ttx has quit IRC | 12:03 | |
| *** ttx has joined #openstack-infra | 12:03 | |
| *** salv-orlando has quit IRC | 12:07 | |
| *** pcrews__ has joined #openstack-infra | 12:07 | |
| *** sdake has quit IRC | 12:09 | |
| *** pcrews_ has quit IRC | 12:10 | |
| *** che-arne has quit IRC | 12:13 | |
| *** pcm has joined #openstack-infra | 12:14 | |
| *** hdd has quit IRC | 12:14 | |
| *** simonmcc_ has quit IRC | 12:16 | |
| *** gondoi has quit IRC | 12:17 | |
| *** sc68cal has quit IRC | 12:17 | |
| *** wendar has quit IRC | 12:17 | |
| *** david-lyle has quit IRC | 12:17 | |
| *** jbryce has quit IRC | 12:17 | |
| *** annegentle has quit IRC | 12:17 | |
| *** simonmcc_ has joined #openstack-infra | 12:17 | |
| *** zigo has quit IRC | 12:17 | |
| *** persia has quit IRC | 12:17 | |
| *** Guest19269 has quit IRC | 12:17 | |
| *** pvo has quit IRC | 12:17 | |
| *** russell_h has quit IRC | 12:17 | |
| *** pabelanger has quit IRC | 12:17 | |
| *** wendar has joined #openstack-infra | 12:17 | |
| *** persia has joined #openstack-infra | 12:18 | |
| *** matsuhashi has quit IRC | 12:18 | |
| *** persia has quit IRC | 12:18 | |
| *** persia has joined #openstack-infra | 12:18 | |
| *** zigo has joined #openstack-infra | 12:18 | |
| *** comstud has quit IRC | 12:18 | |
| *** dolphm has quit IRC | 12:18 | |
| *** AJaeger has quit IRC | 12:18 | |
| *** jlk has quit IRC | 12:18 | |
| *** russellb has quit IRC | 12:18 | |
| *** matsuhashi has joined #openstack-infra | 12:18 | |
| *** adam_g has quit IRC | 12:18 | |
| *** dkranz has joined #openstack-infra | 12:19 | |
| *** salv-orlando has joined #openstack-infra | 12:19 | |
| *** jroll has quit IRC | 12:19 | |
| *** alaski has quit IRC | 12:19 | |
| *** stevebaker has quit IRC | 12:19 | |
| *** StevenK has quit IRC | 12:19 | |
| *** sileht has quit IRC | 12:19 | |
| *** greghaynes has quit IRC | 12:19 | |
| *** mgagne has quit IRC | 12:19 | |
| *** JayF has quit IRC | 12:19 | |
| *** jdurgin has quit IRC | 12:19 | |
| *** matsuhashi has quit IRC | 12:19 | |
| *** matsuhashi has joined #openstack-infra | 12:19 | |
| *** StevenK has joined #openstack-infra | 12:20 | |
| *** JayF has joined #openstack-infra | 12:20 | |
| *** stevebaker has joined #openstack-infra | 12:20 | |
| *** comstud has joined #openstack-infra | 12:20 | |
| *** alaski has joined #openstack-infra | 12:20 | |
| *** jlk has joined #openstack-infra | 12:20 | |
| *** mkoderer has joined #openstack-infra | 12:20 | |
| *** jlk has quit IRC | 12:20 | |
| *** jlk has joined #openstack-infra | 12:20 | |
| *** AJaeger has joined #openstack-infra | 12:21 | |
| *** AJaeger has joined #openstack-infra | 12:21 | |
| *** mgagne has joined #openstack-infra | 12:21 | |
| *** greghaynes has joined #openstack-infra | 12:21 | |
| *** dolphm has joined #openstack-infra | 12:21 | |
| *** adam_g has joined #openstack-infra | 12:21 | |
| *** adam_g has quit IRC | 12:21 | |
| *** adam_g has joined #openstack-infra | 12:21 | |
| *** sc68cal has joined #openstack-infra | 12:21 | |
| *** jroll has joined #openstack-infra | 12:21 | |
| *** russell_h has joined #openstack-infra | 12:21 | |
| *** jdurgin has joined #openstack-infra | 12:21 | |
| *** annegentle has joined #openstack-infra | 12:21 | |
| *** russellb has joined #openstack-infra | 12:21 | |
| *** jbryce has joined #openstack-infra | 12:21 | |
| *** sileht has joined #openstack-infra | 12:22 | |
| *** lillie- has joined #openstack-infra | 12:22 | |
| *** pabelanger has joined #openstack-infra | 12:22 | |
| *** pvo has joined #openstack-infra | 12:23 | |
| *** e0ne has joined #openstack-infra | 12:23 | |
| *** gondoi has joined #openstack-infra | 12:23 | |
| *** freyes has quit IRC | 12:23 | |
| *** _nadya_ has joined #openstack-infra | 12:23 | |
| *** freyes has joined #openstack-infra | 12:26 | |
| *** yfried_ has joined #openstack-infra | 12:26 | |
| *** zhiyan_ is now known as zhiyan | 12:27 | |
| *** ihrachyshka has quit IRC | 12:27 | |
| *** yfried has quit IRC | 12:27 | |
| *** e0ne has quit IRC | 12:28 | |
| *** dcramer_ has quit IRC | 12:30 | |
| *** matsuhashi has quit IRC | 12:30 | |
| *** aysyd has joined #openstack-infra | 12:31 | |
| *** salv-orlando has quit IRC | 12:32 | |
| *** changbl has quit IRC | 12:32 | |
| pcm | sdague: ping | 12:34 |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** bknudson has joined #openstack-infra | 12:37 | |
| *** eharney has joined #openstack-infra | 12:37 | |
| *** Ryan_Lane has joined #openstack-infra | 12:38 | |
| *** Ryan_Lane has joined #openstack-infra | 12:38 | |
| *** miqui has joined #openstack-infra | 12:38 | |
| *** jroll has quit IRC | 12:38 | |
| *** jroll has joined #openstack-infra | 12:39 | |
| NobodyCam | good morning Ironic | 12:39 |
| *** e0ne has joined #openstack-infra | 12:39 | |
| *** blamar has joined #openstack-infra | 12:40 | |
| sdague | pcm: pong | 12:40 |
| *** Ryan_Lane has quit IRC | 12:42 | |
| *** freyes has quit IRC | 12:43 | |
| *** heyongli has joined #openstack-infra | 12:44 | |
| *** yamahata has joined #openstack-infra | 12:46 | |
| *** yfried__ has joined #openstack-infra | 12:48 | |
| *** maxbit has joined #openstack-infra | 12:48 | |
| *** ameade has joined #openstack-infra | 12:49 | |
| *** lillie- is now known as lillie | 12:49 | |
| *** lillie is now known as Guest37432 | 12:50 | |
| NobodyCam | doh .... good morning infra :-p | 12:51 |
| *** thuc has joined #openstack-infra | 12:51 | |
| *** yfried_ has quit IRC | 12:52 | |
| *** thuc has quit IRC | 12:52 | |
| *** thuc_ has joined #openstack-infra | 12:52 | |
| *** thuc has joined #openstack-infra | 12:54 | |
| *** dkranz has quit IRC | 12:55 | |
| *** dstanek has quit IRC | 12:55 | |
| *** dstanek has joined #openstack-infra | 12:56 | |
| *** pblaho has joined #openstack-infra | 12:57 | |
| SergeyLukjanov | NobodyCam, morning ;) | 12:57 |
| *** thuc_ has quit IRC | 12:57 | |
| *** fifieldt has joined #openstack-infra | 12:57 | |
| NobodyCam | morning SergeyLukjanov :) | 12:57 |
| *** maxbit has quit IRC | 12:58 | |
| pcm | Now that Juno is open, I'd like to bring up https://bugs.launchpad.net/neutron/+bug/1282855 for consideration of httmock in test-requirements. | 12:58 |
| uvirtbot | Launchpad bug 1282855 in neutron "Add httmock to test-requirements.txt and update requests -> 2.1.0" [Undecided,In progress] | 12:58 |
| pcm | sdague: ^^ | 12:59 |
| *** _nadya_ has quit IRC | 13:00 | |
| *** signed8bit has joined #openstack-infra | 13:01 | |
| sdague | pcm: could you start that with an ML thread on the list. I feel we've got a lot of different mocking libraries, and we should figure out if we can converge | 13:01 |
| sdague | because if test code is different in every project that inhibits contributors | 13:01 |
| SergeyLukjanov | fungi, clarkb, jeblair, mordred, gerrit bot is dead... | 13:01 |
| pcm | I did send out an e-mail on the ML a few days ago. | 13:01 |
| pcm | 4/4/2014 actually. | 13:02 |
| *** dprince has joined #openstack-infra | 13:02 | |
| pcm | sdague: http://lists.openstack.org/pipermail/openstack-dev/2014-April/031901.html | 13:05 |
| sdague | ok but it's a neutron only tag | 13:06 |
| *** jaypipes has joined #openstack-infra | 13:06 | |
| pcm | on the ML? I have [infra][neutron]. Or do you mean the bug? | 13:06 |
| sdague | the ML post | 13:06 |
| *** timrc-afk is now known as timrc | 13:06 | |
| pcm | Not sure how to do this. Can you please advise? | 13:07 |
| pcm | (as in not sure how to get a proposal for a new package presented - been floundering for several months now) | 13:08 |
| sdague | pcm: well how about figuring out who's using httpretty at this point, because the 2 seem to largely overlap in function, and we're trying to not just make the requirements explode | 13:11 |
| *** andreykurilin_ has joined #openstack-infra | 13:14 | |
| *** _nadya_ has joined #openstack-infra | 13:14 | |
| pcm | sdague: ok. Once found, what is the next step? | 13:15 |
| *** miqui has quit IRC | 13:15 | |
| sdague | pcm: get an agreement to move to this instead | 13:15 |
| *** miqui has joined #openstack-infra | 13:16 | |
| sdague | if you are proposing a new requirement that's largely duplicative of an existing requirement, the burden has to be on the proposer to get the duplicative requirement deprecated across openstack | 13:16 |
| sdague | otherwise requirements grow without bound as everyone brings in their favorite libraries, and the projects diverge | 13:17 |
| pcm | ok. Thanks for the info. | 13:17 |
| *** unicell has quit IRC | 13:17 | |
| *** unicell has joined #openstack-infra | 13:18 | |
| *** unicell has quit IRC | 13:19 | |
| *** lcostantino has joined #openstack-infra | 13:19 | |
| dhellmann | SergeyLukjanov: freenode restarted a bunch of servers overnight because of the OpenSSL bug | 13:19 |
| *** unicell has joined #openstack-infra | 13:19 | |
| pcm | Will either spend the effort to push this, or give up and try to make httpretty work. | 13:19 |
| SergeyLukjanov | dhellmann, it was working about 4 hours ago | 13:20 |
| *** unicell has quit IRC | 13:20 | |
| dhellmann | SergeyLukjanov: ok, I saw a message about restarts on twitter but I don't know what time that was sent | 13:20 |
| SergeyLukjanov | dhellmann, anyway, someone should kick gerrit bot | 13:21 |
| SergeyLukjanov | I mean foot kick :) | 13:21 |
| fungi | okay, what's splodey today? | 13:21 |
| dhellmann | SergeyLukjanov: heh | 13:21 |
| fungi | any fallout from all our updates last night? | 13:21 |
| *** pcargnel has joined #openstack-infra | 13:21 | |
| SergeyLukjanov | fungi, morning | 13:21 |
| SergeyLukjanov | fungi, everything looks ok | 13:22 |
| *** homeless has joined #openstack-infra | 13:22 | |
| fungi | SergeyLukjanov: thanks--that's excellent news. maybe i'll get some work done then ;) | 13:22 |
| SergeyLukjanov | fungi, gerrit bot is slacking again | 13:22 |
| fungi | oh, just saw that--fixing now | 13:22 |
| *** signed8bit has quit IRC | 13:22 | |
| SergeyLukjanov | fungi, thx | 13:23 |
| *** signed8bit has joined #openstack-infra | 13:23 | |
| *** pcargnel has quit IRC | 13:23 | |
| SergeyLukjanov | fungi, probably we should make it fail-fast and run under the "forever" | 13:24 |
| *** openstackgerrit has joined #openstack-infra | 13:24 | |
| fungi | openstackgerrit has returned | 13:24 |
| SergeyLukjanov | yay! | 13:24 |
| fungi | openstackstatus (statusbot) and openstack (meetbot) are still here, so i think all is well in ircdom | 13:25 |
| *** mfer has joined #openstack-infra | 13:26 | |
| *** fbo is now known as fbo_away | 13:27 | |
| SergeyLukjanov | ok | 13:27 |
| SergeyLukjanov | sdague, are you ok with https://review.openstack.org/#/c/84840/? | 13:28 |
| *** _nadya_ has quit IRC | 13:28 | |
| *** dcramer_ has joined #openstack-infra | 13:29 | |
| *** pcm has left #openstack-infra | 13:30 | |
| sdague | SergeyLukjanov: I think there are more things that should be there | 13:30 |
| *** skraynev is now known as skraynev_afk | 13:31 | |
| *** dizquierdo is now known as dizquierdo_afk | 13:32 | |
| *** dims has joined #openstack-infra | 13:33 | |
| *** mrmartin has joined #openstack-infra | 13:34 | |
| sdague | fungi: man, gerrit is so slow again | 13:34 |
| sdague | did we every figure out if there was a way to instrument that? | 13:35 |
| fungi | slow to respond to api calls? | 13:35 |
| *** miqui_ has joined #openstack-infra | 13:36 | |
| *** miqui has quit IRC | 13:37 | |
| *** Ryan_Lane has joined #openstack-infra | 13:39 | |
| *** julim has joined #openstack-infra | 13:39 | |
| anteaya | mattoliverau: looking for your email regarding your ballot, I don't see anything. I am concluding you have sorted out the answer your needed on your own. | 13:39 |
| *** thomasem has joined #openstack-infra | 13:41 | |
| *** nkinder has quit IRC | 13:41 | |
| sdague | fungi: git review takes > 30s | 13:41 |
| sdague | to push a 10 line patch | 13:41 |
| *** oomichi has quit IRC | 13:42 | |
| *** jcoufal has quit IRC | 13:42 | |
| *** zehicle has quit IRC | 13:43 | |
| *** zehicle_at_dell has quit IRC | 13:43 | |
| *** thuc_ has joined #openstack-infra | 13:43 | |
| *** Ryan_Lane has quit IRC | 13:43 | |
| fungi | the jvm for gerrit is running system load up to ~3 | 13:43 |
| fungi | though the server has 8vcpus, so that should be plenty low | 13:44 |
| *** jcoufal has joined #openstack-infra | 13:44 | |
| sdague | fungi: ok, is that something which is monitored in cacti? hard to kmow if that's good or bad | 13:44 |
| *** ihrachyshka has joined #openstack-infra | 13:44 | |
| sdague | what all did you end up having to revoke for infra based on the heartbug? | 13:44 |
| *** dkranz has joined #openstack-infra | 13:46 | |
| *** thuc has quit IRC | 13:46 | |
| *** alaski has quit IRC | 13:46 | |
| *** alaski has joined #openstack-infra | 13:47 | |
| *** thuc_ has quit IRC | 13:47 | |
| *** hashar has joined #openstack-infra | 13:47 | |
| *** andreykurilin_ has quit IRC | 13:47 | |
| fungi | it is monitored by cacti, but i can't look at the graphs for it right now because http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-2708.html | 13:48 |
| uvirtbot | fungi: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem. When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-2708) | 13:48 |
| fungi | last night we got through rekeying the puppet relationships for all machines after upgrading packages everywhere and rebooting for good measure. still plenty of things to revoke/regenerate/reissue/replace though | 13:49 |
| fungi | but the big scare was that we had an openssl-based socket listening on the puppetmaster, so keys to the kingdom were top priority | 13:50 |
| *** alexpilotti has quit IRC | 13:51 | |
| fungi | the other stuff we can update piecemeal with minimal impact to the community (still a priority, but doesn't need to happen in one shot or during maintenance windows) | 13:51 |
| *** freyes_ has joined #openstack-infra | 13:52 | |
| sdague | fungi: gotcha. I'm trying to sort out if openvpn is actually impacted by this | 13:52 |
| *** vhoward- has left #openstack-infra | 13:52 | |
| fungi | mainly want to regenerate ssh host keys, get x.509 certs for services reissued based on fresh keys, reset passwords for everything, and so on | 13:52 |
| fungi | sdague: funny, i got asked the same thing last night by a friend who uses it to virtually network devices for his customers (manages smart climate control systems in residences and office/industrial buildings) | 13:54 |
| fungi | from what i know of openvpn it uses openssl for its crypto primitives but doesn't *actually* do ssl/tls with it | 13:54 |
| sdague | yeh, well my exposure changes if openvpn is or is not affected, like puppet. As my puppet happens only over openvpn links | 13:54 |
| sdague | because it was simpler to just build a shared L2 that way, and not punch extra holes in my firewall | 13:55 |
| fungi | also, i suspect if you follow their recommendation to use a tls key like they recommend, it never actually gets to the handshake because openvpn discards datagrams which don't have the right secret on them | 13:56 |
| *** pcrews__ has quit IRC | 13:56 | |
| fungi | looking into it myself out of curiosity now, even though i don't really use openvpn for anything currently | 13:56 |
| fungi | though i used to manage large customer vpn aggregators based on it, so have a pretty strong understanding of its inner workings | 13:57 |
| *** che-arne has joined #openstack-infra | 13:57 | |
| *** markmcclain has quit IRC | 13:58 | |
| *** prad has joined #openstack-infra | 13:59 | |
| *** zns has joined #openstack-infra | 14:00 | |
| *** hashar has quit IRC | 14:00 | |
| fungi | lots of news articles out there which mention that openvpn uses openssl, but no statement from the openvpn devs on actual viability of the exploit against it and if so what configuration options expose the behavior | 14:00 |
| *** zns has quit IRC | 14:01 | |
| *** zns has joined #openstack-infra | 14:01 | |
| openstackgerrit | Eric Windisch proposed a change to openstack-infra/config: Make docker py26/py27 gates voting https://review.openstack.org/86042 | 14:02 |
| *** msabramo has quit IRC | 14:02 | |
| *** jgriffit1 has quit IRC | 14:03 | |
| *** jgriffit1 has joined #openstack-infra | 14:05 | |
| fungi | sdague: as i suspected, "Using the tls-auth option should protect against this vulnerability | 14:06 |
| fungi | (assuming that your tls-auth key is not known to the attacker)." | 14:06 |
| fungi | http://sourceforge.net/p/openvpn/mailman/openvpn-devel/?viewmonth=201404 | 14:06 |
| *** malini_afk is now known as malini | 14:07 | |
| *** jergerber has joined #openstack-infra | 14:08 | |
| *** pdmars has quit IRC | 14:08 | |
| *** pdmars has joined #openstack-infra | 14:08 | |
| sdague | fungi: ok, good to know, however I didn't set that | 14:08 |
| fungi | oh | 14:09 |
| fungi | yeah, i always used those and baked them into the auto-installed client config for good measure | 14:10 |
| *** miqui_ is now known as miqui | 14:10 | |
| fungi | since the howtos on configuring openvpn (back to the early days) recommended it as a belt-and-braces measure against ssl-layer vulnerabilities | 14:10 |
| fungi | though worth noting, it's a shared key amongst all clients to that endpoint (sort of similar to an ipsec xauth group key but at a lower level and of course entirely different protocol) so if you have a malicious client with a copy of that key you're still vulnerable to it at that point | 14:13 |
| fungi | merely reduces your exposure to those who know that key | 14:13 |
| sdague | yep | 14:14 |
| sdague | so you guys did host key resets on everything as well? | 14:15 |
| fungi | that's still in progress | 14:15 |
| *** chandan_kumar has quit IRC | 14:16 | |
| *** rcleere has joined #openstack-infra | 14:16 | |
| fungi | i'm waffling on regenerating the api host key for review.openstack.org since i suspect that will be disruptive to the entire development community with long-lived repercussions | 14:16 |
| sdague | well thinking this through, isn't that the only vulnerable system | 14:17 |
| sdague | because you'd need a valid connection for the heartbeat | 14:18 |
| sdague | or am I thinking about that wrong | 14:18 |
| fungi | no, the way this vulnerability works is by exploiting an uninitialized struct in the response packet during the handshake | 14:19 |
| fungi | reminiscent of the old days when you could read system memory remotely by sending large icmp echo packets and looking at the padding in the echo replies | 14:19 |
| *** heyongli has quit IRC | 14:20 | |
| fungi | the response packet is related to feature option negotiation for tls heartbeat support, but doesn't actually use the heartbeat traffic | 14:20 |
| *** che-arne has quit IRC | 14:20 | |
| fungi | also, the ssh api wouldn't be directly vulnerable to this because it's not tls, nor is it probably even using openssl inside the gerrit jvm (though i suppose it could be at some level, not in a vulnerable manner) | 14:21 |
| *** jgrimm has joined #openstack-infra | 14:22 | |
| fungi | the reason to regenerate the host key for that service is purely because it's cached in memory on the host and thus could have been read out by exploiting the vulnerable https service running there before we upgraded it | 14:22 |
| *** thuc has joined #openstack-infra | 14:24 | |
| *** freyes_ is now known as freyes | 14:24 | |
| fungi | or, in an even more convoluted scenario, someone reading key material off the puppetmaster could have somehow coerced themselves access to review.o.o at a level where they were able to retrieve its host key from the filesystem or something | 14:25 |
| *** dkliban has joined #openstack-infra | 14:26 | |
| sdague | but the exposure is only within the process memory right? | 14:26 |
| sdague | because the vmm protects you from reading actual system memory, as that's a segmentation fault | 14:27 |
| *** pblaho has quit IRC | 14:27 | |
| *** david-lyle has joined #openstack-infra | 14:28 | |
| *** thuc has quit IRC | 14:29 | |
| *** nkinder has joined #openstack-infra | 14:30 | |
| fungi | hopefully yes, though you're trusting malloc'd segments to be zeroed (which is expected, but depends on platform) | 14:31 |
| *** msabramo has joined #openstack-infra | 14:33 | |
| fungi | on modern linux i expect it's probably safe, which is why the top priority concerns we have are secret data which could be used by/pass through the process linking openssl and doing tls socket handling | 14:34 |
| dhellmann | fungi: you mentioned regnerating ssh host keys, do we need to worry about individual user keys, too? | 14:36 |
| *** e0ne has quit IRC | 14:39 | |
| fungi | dhellmann: i wouldn't unless you were keeping them somewhere accessible to a vulnerable service (a web-based password safe for example) | 14:39 |
| *** e0ne has joined #openstack-infra | 14:39 | |
| *** Ryan_Lane has joined #openstack-infra | 14:39 | |
| fungi | rsa authentication is good about you not having to disclose your secret key to services authenticating you | 14:40 |
| dhellmann | fungi: thanks | 14:40 |
| * dhellmann trusts that the fungi answering questions in this room right now is the one he knows, and the answers can therefore be trusted #security | 14:41 | |
| *** saju_m has quit IRC | 14:41 | |
| sdague | it's turtles all the way down | 14:41 |
| dhellmann | sdague: indeed | 14:42 |
| *** salv-orlando has joined #openstack-infra | 14:42 | |
| *** e0ne has quit IRC | 14:43 | |
| *** sandywalsh_ has joined #openstack-infra | 14:43 | |
| *** Ryan_Lane has quit IRC | 14:44 | |
| *** thingee has quit IRC | 14:45 | |
| *** jlibosva has quit IRC | 14:45 | |
| *** mkerrin has quit IRC | 14:45 | |
| *** dhellmann has quit IRC | 14:45 | |
| *** juice has quit IRC | 14:45 | |
| *** jd__ has quit IRC | 14:45 | |
| *** gilliard has quit IRC | 14:45 | |
| *** dizquierdo_afk has quit IRC | 14:45 | |
| *** dangers_away has quit IRC | 14:45 | |
| *** akscram has quit IRC | 14:45 | |
| *** plomakin_ has quit IRC | 14:45 | |
| *** sirushti has quit IRC | 14:45 | |
| *** sahumada has quit IRC | 14:45 | |
| *** nibalizer has quit IRC | 14:45 | |
| *** gaelL has quit IRC | 14:45 | |
| *** dosaboy has quit IRC | 14:45 | |
| *** gmoro has quit IRC | 14:45 | |
| *** mihgen has quit IRC | 14:45 | |
| *** afazekas has quit IRC | 14:45 | |
| *** yolanda has quit IRC | 14:45 | |
| *** ianw has quit IRC | 14:45 | |
| *** jeblair has quit IRC | 14:45 | |
| *** marktraceur has quit IRC | 14:45 | |
| *** marun has quit IRC | 14:45 | |
| *** krotscheck has quit IRC | 14:45 | |
| *** ryanpetrello has quit IRC | 14:45 | |
| *** YorikSar has quit IRC | 14:45 | |
| *** mordred has quit IRC | 14:45 | |
| *** dteselki- has quit IRC | 14:45 | |
| *** jamielennox|away has quit IRC | 14:45 | |
| *** BobBallA1ay has quit IRC | 14:45 | |
| *** proffalken has quit IRC | 14:45 | |
| *** dachary has quit IRC | 14:45 | |
| *** timrc has quit IRC | 14:45 | |
| *** jeremyb has quit IRC | 14:45 | |
| *** skraynev_afk has quit IRC | 14:45 | |
| *** SergeyLukjanov has quit IRC | 14:45 | |
| *** SpamapS has quit IRC | 14:45 | |
| *** spiffxp has quit IRC | 14:45 | |
| *** jesusaurus has quit IRC | 14:45 | |
| *** guitarzan has quit IRC | 14:45 | |
| *** blkperl has quit IRC | 14:45 | |
| *** vipul has quit IRC | 14:45 | |
| *** morgabra has quit IRC | 14:45 | |
| *** wenlock has joined #openstack-infra | 14:46 | |
| *** dangers_away has joined #openstack-infra | 14:48 | |
| *** guitarzan has joined #openstack-infra | 14:48 | |
| *** gmoro has joined #openstack-infra | 14:48 | |
| *** blkperl has joined #openstack-infra | 14:49 | |
| *** proffalken has joined #openstack-infra | 14:49 | |
| *** dosaboy has joined #openstack-infra | 14:49 | |
| *** sirushti has joined #openstack-infra | 14:49 | |
| *** krotscheck has joined #openstack-infra | 14:49 | |
| *** skraynev_afk has joined #openstack-infra | 14:49 | |
| *** wenlock has quit IRC | 14:49 | |
| *** wenlock has joined #openstack-infra | 14:49 | |
| *** jeremyb has joined #openstack-infra | 14:49 | |
| *** YorikSar has joined #openstack-infra | 14:49 | |
| *** jamielennox|away has joined #openstack-infra | 14:49 | |
| *** dangers_away has quit IRC | 14:49 | |
| *** dangers_away has joined #openstack-infra | 14:49 | |
| *** akscram has joined #openstack-infra | 14:49 | |
| *** yolanda has joined #openstack-infra | 14:49 | |
| *** jamielennox|away has quit IRC | 14:49 | |
| *** jamielennox|away has joined #openstack-infra | 14:49 | |
| *** dizquierdo has joined #openstack-infra | 14:49 | |
| *** ryanpetrello has joined #openstack-infra | 14:49 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:49 | |
| *** spiffxp has joined #openstack-infra | 14:49 | |
| *** afazekas has joined #openstack-infra | 14:49 | |
| *** jlibosva has joined #openstack-infra | 14:49 | |
| *** timrc has joined #openstack-infra | 14:49 | |
| *** HenryG has quit IRC | 14:49 | |
| *** sahumada has joined #openstack-infra | 14:49 | |
| *** sandywalsh_ has quit IRC | 14:49 | |
| *** dhellmann has joined #openstack-infra | 14:49 | |
| *** vipul has joined #openstack-infra | 14:49 | |
| *** plomakin has joined #openstack-infra | 14:49 | |
| *** BobBallAway has joined #openstack-infra | 14:49 | |
| *** dachary has joined #openstack-infra | 14:50 | |
| *** jd__ has joined #openstack-infra | 14:50 | |
| *** morgabra has joined #openstack-infra | 14:50 | |
| *** dteselkin_ has joined #openstack-infra | 14:50 | |
| *** juice has joined #openstack-infra | 14:50 | |
| *** jesusaurus has joined #openstack-infra | 14:50 | |
| *** skolekonov has joined #openstack-infra | 14:50 | |
| *** SpamapS has joined #openstack-infra | 14:50 | |
| *** jeblair has joined #openstack-infra | 14:50 | |
| *** marktraceur has joined #openstack-infra | 14:50 | |
| *** ianw has joined #openstack-infra | 14:51 | |
| *** thedodd has joined #openstack-infra | 14:51 | |
| *** mordred has joined #openstack-infra | 14:51 | |
| *** gaelL has joined #openstack-infra | 14:52 | |
| *** nibalizer has joined #openstack-infra | 14:53 | |
| *** esker has joined #openstack-infra | 14:54 | |
| *** thingee has joined #openstack-infra | 14:55 | |
| *** mkerrin has joined #openstack-infra | 14:55 | |
| *** skolekonov has quit IRC | 14:58 | |
| *** thuc has joined #openstack-infra | 14:59 | |
| *** HenryG has joined #openstack-infra | 14:59 | |
| *** e0ne has joined #openstack-infra | 14:59 | |
| *** thuc_ has joined #openstack-infra | 15:00 | |
| *** wchrisj has joined #openstack-infra | 15:00 | |
| *** sabari2 has joined #openstack-infra | 15:02 | |
| *** gilliard has joined #openstack-infra | 15:02 | |
| *** gokrokve has joined #openstack-infra | 15:03 | |
| *** e0ne_ has joined #openstack-infra | 15:03 | |
| *** russell_h has quit IRC | 15:03 | |
| *** russell_h has joined #openstack-infra | 15:03 | |
| *** thuc has quit IRC | 15:03 | |
| *** hashar has joined #openstack-infra | 15:05 | |
| *** pcrews has joined #openstack-infra | 15:05 | |
| *** e0ne has quit IRC | 15:06 | |
| mordred | fungi: I would not expect malloc'd segments to be zeroed | 15:06 |
| mordred | fungi: one of the big differences between new in C++ and malloc in C is that new _Does_ zero and malloc does not - it's up to the programmer in C to remember to memset(buff, 0, buffsize) | 15:07 |
| jd__ | indeed, that'd be calloc() | 15:07 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities https://review.openstack.org/85565 | 15:08 |
| *** yfried has joined #openstack-infra | 15:09 | |
| *** yfried__ has quit IRC | 15:09 | |
| *** guitarzan has quit IRC | 15:09 | |
| *** mihgen has joined #openstack-infra | 15:09 | |
| *** dcramer_ has quit IRC | 15:09 | |
| *** wchrisj has quit IRC | 15:09 | |
| *** jcoufal has quit IRC | 15:10 | |
| *** ArxCruz has joined #openstack-infra | 15:10 | |
| *** pdmars has quit IRC | 15:11 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities https://review.openstack.org/85565 | 15:11 |
| *** sabari2 has quit IRC | 15:11 | |
| *** branen has quit IRC | 15:11 | |
| *** pdmars has joined #openstack-infra | 15:12 | |
| *** dangers_away is now known as dangers | 15:13 | |
| *** yfried has quit IRC | 15:13 | |
| *** dcramer_ has joined #openstack-infra | 15:15 | |
| fungi | mordred: i should have said s/expect/hope/, up to the caller to zero blocks allocated and deallocated after all | 15:15 |
| fungi | but as sdague points out vmm should also hopefully prevent leaking between processes even if the author forgot to do so | 15:16 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/86057 | 15:17 |
| *** branen has joined #openstack-infra | 15:17 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/86057 | 15:18 |
| *** maxbit has joined #openstack-infra | 15:19 | |
| mordred | fungi: ++ | 15:19 |
| *** andreaf has quit IRC | 15:20 | |
| *** dcramer_ has quit IRC | 15:20 | |
| *** jlibosva has quit IRC | 15:21 | |
| fungi | it's considered a security bug in the kernel if it leaks abandoned memory contents from your process into a new process's allocation, just better to be sure anyway if you can | 15:22 |
| sdague | fungi: also, wouldn't an attempt to exploit this lead to seeing seg faults of processes | 15:22 |
| sdague | at least at a blanket attack | 15:22 |
| sdague | which the kernel typically logs in dmesg | 15:22 |
| *** marun has joined #openstack-infra | 15:22 | |
| *** dstanek has quit IRC | 15:23 | |
| *** changbl has joined #openstack-infra | 15:23 | |
| fungi | sdague: i don't think this would segfault anything. you're reading 64k of surrounding process memory, but it's allocated to that process | 15:23 |
| sdague | fungi: I thought the point was you could read any 64k | 15:23 |
| *** alff has joined #openstack-infra | 15:24 | |
| *** alff_ has joined #openstack-infra | 15:24 | |
| fungi | sdague: if it's been leaked into the that process's memory, then yes. the point that there are circumstances where that *might* happen, and the researchers who announced this said they read passwords and other document material from memory remotely in their tests, but i've also seen a lot of skeptical analysis of those claims | 15:25 |
| sdague | well, if it was a web based system, like running owncloud, sure | 15:25 |
| fungi | right now it's zomg-do-whatever-you-can panic, while the more detailed exploit pocs are waiting in the wings to be published | 15:26 |
| sdague | fungi: gotcha | 15:26 |
| fungi | i suspect it's an overreaction, but better safe than sorry | 15:26 |
| sdague | this is because the cloudfire folks leaked early before all the patches? | 15:26 |
| dims | hallyn, jamespage, zul, clarkb, jogo - just noticed ibvirt 1.2.2+ from UCA on precise - kicking off a review for it - https://review.openstack.org/#/c/86057/ fyi | 15:27 |
| jamespage | dims, ack | 15:27 |
| fungi | sdague: that's the rumor i heard. rh was aware and had apparently given other major distros a heads up yesterday morning, but then it all exploded a few hours later | 15:27 |
| *** mrodden has joined #openstack-infra | 15:28 | |
| mordred | I thought they disclosed after it was announced - and it was the fact taht the distros werent' included on the embargo | 15:28 |
| sdague | jamespage: what's the eta on the .18 libvirt precise package? | 15:28 |
| mordred | either way - everything about it was craptastic | 15:28 |
| sdague | http://packages.ubuntu.com/precise/libvirt-bin | 15:28 |
| zul | dims: okies | 15:29 |
| *** dcramer_ has joined #openstack-infra | 15:29 | |
| sdague | because we're hot patching from our own copy of that package in devstack-gate now | 15:29 |
| sdague | and I'd like to stop doing that | 15:29 |
| sdague | (if we don't hot patch something like 5% of all tempest runs fail, which means we almost can't land patches) | 15:29 |
| fungi | mordred: that was the first thing i heard, then later one of the other not-rh distros piped up and said that rh had started to warn the others, but that the announcement came mere hours on the heels of that because someone leaked it | 15:29 |
| fungi | so i'm guessing they meant to disclose later this week or early next | 15:30 |
| *** denis_makogon_ has joined #openstack-infra | 15:30 | |
| fungi | though i've heard nothing to substantiate that yet, pure speculation on my part | 15:30 |
| mordred | fungi: gah | 15:31 |
| nibalizer | :( | 15:31 |
| jamespage | sdague, lemme check with hallyn | 15:31 |
| fungi | we're all still very much in the rumor and misinformation phase of this situation after all | 15:31 |
| mordred | dims: is this the build-libvirt-python-from-pip release? | 15:32 |
| *** ArxCruz has quit IRC | 15:32 | |
| *** gyee has joined #openstack-infra | 15:32 | |
| *** ArxCruz has joined #openstack-infra | 15:33 | |
| dims | mordred, not sure saw it here - http://javacruft.wordpress.com/2014/04/03/openstack-icehouse-rc1/, we have been bugging UCA team for 1.2.x on precise for a while now and looks like we have it now | 15:33 |
| sdague | fungi: can I get your assisstance with the right test for - https://review.openstack.org/#/c/84829/7/test-functions.sh - as I'm clearly not quite thinking about this in the same way as you and jeblair | 15:33 |
| fungi | sdague: sure, momentarily... just peeled off of a conference call | 15:34 |
| sdague | yep, no problem | 15:34 |
| *** ominakov has quit IRC | 15:37 | |
| *** esker has quit IRC | 15:38 | |
| *** msabramo has quit IRC | 15:40 | |
| *** zns has quit IRC | 15:41 | |
| *** homeless has quit IRC | 15:41 | |
| fungi | sdague: okay, so basically the way the current unit tests in d-g are structured is that you describe a change which is being tested (project name and target branch) with the changes ahead of it in a dependent queue. then you set the branch-affecting variables you would set in a particular job for that change, and see what would have been checked out for various projects which were a part of the | 15:41 |
| fungi | integration test (some of which may correspond to changes ahead in the queue, others possibly not) | 15:41 |
| *** signed8bit has quit IRC | 15:42 | |
| *** homeless has joined #openstack-infra | 15:42 | |
| *** signed8bit has joined #openstack-infra | 15:42 | |
| *** che-arne has joined #openstack-infra | 15:42 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Make ironic-undercloud and f20-overcloud jobs non-voting https://review.openstack.org/85468 | 15:42 |
| bnemec | fungi: pleia2: ^Sorry! | 15:42 |
| bnemec | I promise not to change that again. :-) | 15:43 |
| fungi | sdague: so for example, you want to test that a nova stable/havana change gets an integration job run with master tempest when you set the project override for tempest to master | 15:43 |
| bnemec | Even if those jobs start passing they obviously have some stability issues so they can stay non-voting for a while. | 15:43 |
| sdague | fungi: yep | 15:43 |
| fungi | sdague: and you may want to confirm that the zuul ref on master for a devstack-gate change ahead of it gets checked out | 15:43 |
| *** markwash has joined #openstack-infra | 15:43 | |
| fungi | sdague: and possibly that a neutron change on stable/havana ahead of it causes the zuul ref for that to get checked out rather than the stable/havana branch tip | 15:44 |
| fungi | and so on | 15:44 |
| sdague | ok, sure | 15:44 |
| *** denis_makogon_ is now known as denis_makogon | 15:45 | |
| *** msabramo has joined #openstack-infra | 15:46 | |
| *** freyes has quit IRC | 15:46 | |
| fungi | i'm thinking you may want a total of two tests there... one making sure that you can test a change for tempest master against nova stable/havana, and another making sure that you can test a change for nova stabe/havana against tempest master | 15:46 |
| sdague | fungi: ok | 15:47 |
| fungi | and in the first, use the global override, while in the second use the project-specific override or whatever makes sense | 15:47 |
| *** jcoufal has joined #openstack-infra | 15:48 | |
| *** zns has joined #openstack-infra | 15:49 | |
| *** jdurgin has quit IRC | 15:53 | |
| sdague | fungi: ok, so that leads me to something like this - https://etherpad.openstack.org/p/zuul-tests | 15:57 |
| sdague | which doesn't pass | 15:57 |
| *** jdurgin has joined #openstack-infra | 15:57 | |
| jamespage | sdague, I've marked it verification done - its had enough testing IMHO - so it should release soonish | 15:58 |
| sdague | jamespage: great | 15:59 |
| *** julienvey has quit IRC | 15:59 | |
| jamespage | dims, also note that we have a new qemu version coming into icehouse UCA today | 15:59 |
| *** ArxCruz has quit IRC | 15:59 | |
| sdague | fungi: is there an alphabet issue? | 15:59 |
| *** ihrachyshka_ has joined #openstack-infra | 16:00 | |
| *** ihrachyshka has quit IRC | 16:00 | |
| fungi | alphabet soup | 16:00 |
| *** morgabra has quit IRC | 16:00 | |
| *** morgabra has joined #openstack-infra | 16:00 | |
| sdague | I think I've got a better version of the test, however, it means that python-glanceclient seems to be detected at the wrong branch point | 16:01 |
| sdague | and I wonder if that's an alphabet issue | 16:01 |
| *** yfried has joined #openstack-infra | 16:01 | |
| *** alff_ has quit IRC | 16:01 | |
| *** alff has quit IRC | 16:01 | |
| *** matrohon has quit IRC | 16:02 | |
| *** lcostantino has quit IRC | 16:04 | |
| *** blamar has quit IRC | 16:04 | |
| sdague | fungi: any thoughts? | 16:04 |
| sdague | I'm about out of ideas, and my zuul matrix in my head is clearly at it's limits | 16:05 |
| *** julim has quit IRC | 16:05 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/85816 | 16:05 |
| fungi | sdague: it's a pyramid | 16:05 |
| fungi | sdague: so for the project corresponding to ZA, you'll also have it appearing in ZB, ZC and ZD | 16:05 |
| fungi | sdague: and then for the project corresponding to ZB, you'll also have it appearing in ZC and ZD | 16:05 |
| fungi | and so on | 16:05 |
| sdague | so I have to manually push ZC onto all places? | 16:06 |
| sdague | regardless of branch? | 16:07 |
| fungi | sdague: so if the projects/branches whose changes are being tested are distinct, you'll have four with a ZD ref, three with a ZC ref, two with a ZB ref and one with a ZA ref | 16:07 |
| fungi | sdague: test_grenade_backward() is a good demonstrative example of the scenario setup | 16:07 |
| fungi | since it has 5 changes in the queue plus incorporates checkouts of other changes not in the queue | 16:08 |
| sdague | so is ZB irrespective of branch? | 16:08 |
| sdague | I guess in my mind I was trying to create the zuul queue | 16:08 |
| sdague | which this is not actually representing | 16:08 |
| fungi | well, it's representing the merge commits which zuul calculates for the change being tested and the other changes it's possibly being tested with ahead of it | 16:08 |
| fungi | and it's roject+branch-specific | 16:09 |
| fungi | because zc may be getting tested with a nova master change in zb and a nova stable change in za, so there will be zc refs on nova master and nova stable | 16:10 |
| fungi | grenade, being our best current example of multi-branch testing, wants to be sure that you're testing an upgrade from za to zb when you're running it on a change at zc | 16:10 |
| *** hogepodge has joined #openstack-infra | 16:11 | |
| fungi | note that in test_grenade_backward() we have both... | 16:11 |
| fungi | TEST_ZUUL_REFS[keystone]+=' refs/zuul/stable/havana/ZD' | 16:11 |
| fungi | TEST_ZUUL_REFS[keystone]+=' refs/zuul/master/ZD' | 16:11 |
| *** sdake has joined #openstack-infra | 16:12 | |
| fungi | because (and also for ZE) | 16:12 |
| fungi | er, s/because // | 16:12 |
| *** mihgen has quit IRC | 16:13 | |
| fungi | because zc is a keystone stable/havana change and zd is a keystone master change | 16:13 |
| fungi | and ze needs to be tested with those | 16:14 |
| *** e0ne_ has quit IRC | 16:14 | |
| *** derekh has quit IRC | 16:14 | |
| *** e0ne has joined #openstack-infra | 16:14 | |
| *** ociuhandu_ has joined #openstack-infra | 16:15 | |
| *** moted has joined #openstack-infra | 16:17 | |
| *** markmcclain has joined #openstack-infra | 16:17 | |
| *** ociuhandu_ has quit IRC | 16:17 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Support Matrix Tie Parent Plugin in wrappers https://review.openstack.org/85642 | 16:18 |
| *** jgallard has quit IRC | 16:18 | |
| fungi | logstash job queue splodeyness | 16:18 |
| *** e0ne has quit IRC | 16:19 | |
| *** ihrachyshka_ has quit IRC | 16:19 | |
| fungi | cluster status is yellow | 16:20 |
| *** krtaylor has quit IRC | 16:21 | |
| *** hogepodge has quit IRC | 16:21 | |
| fungi | several unassigned shards and more being replicated at the moment | 16:22 |
| msabramo | hashar: thoughts on https://review.openstack.org/84778 ? | 16:22 |
| fungi | the es cluster and all the workers look like they're still online | 16:22 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow for generic branch overrides https://review.openstack.org/84829 | 16:24 |
| sdague | fungi: so how about that? | 16:24 |
| *** jcoufal has quit IRC | 16:25 | |
| *** Sukhdev has joined #openstack-infra | 16:26 | |
| hashar | msabramo: gotta leave sorry :/ | 16:29 |
| *** blamar has joined #openstack-infra | 16:29 | |
| *** zns has quit IRC | 16:29 | |
| hashar | msabramo: haven't reviewed that patch yet :-] | 16:30 |
| *** hashar has quit IRC | 16:30 | |
| *** alff_ has joined #openstack-infra | 16:30 | |
| *** alff has joined #openstack-infra | 16:30 | |
| *** markmc has quit IRC | 16:31 | |
| *** blamar has quit IRC | 16:32 | |
| *** blamar has joined #openstack-infra | 16:32 | |
| msabramo | mgagne: Wondering if you have any thoughts on https://review.openstack.org/84778 | 16:34 |
| fungi | sdague: lookin | 16:34 |
| mgagne | msabramo: reading | 16:34 |
| *** timrc is now known as timrc-afk | 16:34 | |
| *** dizquierdo has quit IRC | 16:34 | |
| *** dstanek has joined #openstack-infra | 16:34 | |
| *** saju_m has joined #openstack-infra | 16:34 | |
| *** jcoufal has joined #openstack-infra | 16:35 | |
| *** krtaylor has joined #openstack-infra | 16:39 | |
| *** sabari has joined #openstack-infra | 16:40 | |
| *** MIDENN_ has quit IRC | 16:40 | |
| *** mdenny has joined #openstack-infra | 16:40 | |
| *** ildikov_ has quit IRC | 16:41 | |
| *** MarkAtwood has joined #openstack-infra | 16:42 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 16:42 |
| *** yassine has quit IRC | 16:42 | |
| *** bhuvan has joined #openstack-infra | 16:45 | |
| *** MarkAtwood has quit IRC | 16:46 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: update config variable to OVERRIDE_foo_PROJECT_BRANCH https://review.openstack.org/86090 | 16:48 |
| sdague | fungi: also, with the var change, I need that ^^^^^ | 16:48 |
| fungi | ahh, right | 16:50 |
| *** jroll has quit IRC | 16:50 | |
| *** jroll has joined #openstack-infra | 16:51 | |
| annegentle | ttx: or anyone: is there a rollup location in launchpad to get a count of all blueprints implemented across icehouse for ceilometer cinder glance horizon keystone neutron nova swift trove | 16:52 |
| annegentle | ttx: seems like the /openstack project only contains -infra and docs | 16:53 |
| annegentle | ttx: I can count by hand I guess | 16:53 |
| *** MarkAtwood has joined #openstack-infra | 16:54 | |
| annegentle | fungi: or clarkb any ideas for a blueprint count across 9 projects? ^^ | 16:54 |
| fungi | annegentle: combine nine queries? (that's what i'd probably end up doing) | 16:55 |
| fungi | then again, i'm a brute | 16:55 |
| annegentle | fungi: ha. brute. | 16:55 |
| annegentle | fungi: yeah 36+ browser tabs was the best I came up with | 16:56 |
| *** jgrimm has quit IRC | 16:56 | |
| fungi | (i tend to use brute-force solutions to infrequent problems because they can end up taking less time that devising a more elegant solution) | 16:56 |
| fungi | s/that/than/ | 16:56 |
| annegentle | ayup | 16:57 |
| annegentle | totally get that | 16:57 |
| fungi | er go http://xkcd.com/1205/ | 16:57 |
| fungi | er, ergo | 16:58 |
| annegentle | fungi: ttx probably does this task often tho | 16:58 |
| annegentle | but it's pycon week. | 16:58 |
| koolhead17 | stustus.openstack.com | 16:58 |
| annegentle | that should be on that chart LOL | 16:58 |
| koolhead17 | might have it | 16:58 |
| koolhead17 | status | 16:58 |
| fungi | oh, mining http://status.openstack.org/release/ | 16:59 |
| fungi | good suggestion koolhead17 | 16:59 |
| koolhead17 | annegentle: ^^ | 16:59 |
| *** krotscheck has quit IRC | 17:00 | |
| *** mriedem has joined #openstack-infra | 17:00 | |
| fungi | how quickly i seem to forget that exists | 17:00 |
| *** krotscheck has joined #openstack-infra | 17:00 | |
| koolhead17 | fungi: your getting old :P | 17:00 |
| fungi | tell me about it | 17:00 |
| ttx | annegentle: yes, what fungi said | 17:00 |
| ttx | http://status.openstack.org/release/ | 17:01 |
| fungi | to be perfectly fair, that was koolhead17's suggestion, because i'm dense | 17:01 |
| *** bhuvan_ has joined #openstack-infra | 17:01 | |
| annegentle | koolhead17: thanks! | 17:02 |
| *** julim has joined #openstack-infra | 17:02 | |
| clarkb | morning | 17:02 |
| *** andreykurilin_ has joined #openstack-infra | 17:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: update config variable to OVERRIDE_foo_PROJECT_BRANCH https://review.openstack.org/86090 | 17:02 |
| marun | fungi: is it possible for me to be personally notified if the neutron functional job fails? | 17:02 |
| *** _nadya_ has joined #openstack-infra | 17:03 | |
| *** amcrn has quit IRC | 17:04 | |
| *** bhuvan has quit IRC | 17:04 | |
| *** mrmartin has quit IRC | 17:04 | |
| *** rfolco has quit IRC | 17:05 | |
| *** fifieldt has quit IRC | 17:05 | |
| *** ihrachyshka has joined #openstack-infra | 17:05 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat auth bug 1304544 https://review.openstack.org/86093 | 17:06 |
| uvirtbot | Launchpad bug 1304544 in heat "test_server_cfn_init fails in setup with Unauthorized" [Undecided,New] https://launchpad.net/bugs/1304544 | 17:06 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Only run jobs with neutron on neutron changes https://review.openstack.org/84840 | 17:07 |
| *** amotoki has quit IRC | 17:07 | |
| fungi | marun: i think what you'll want to do is make sure you have gerrit set to watch the projects it runs on and send you e-mail for all comments, then filter the message bodies looking for the job name/failure pattern and stick those messages somewhere you'll notice | 17:07 |
| fungi | marun: i do something similar to separate different sorts of gerrit e-mails into different buckets so i can keep tabs on things better | 17:08 |
| *** rfolco has joined #openstack-infra | 17:09 | |
| *** timrc-afk is now known as timrc | 17:09 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 17:09 |
| *** jgrimm has joined #openstack-infra | 17:09 | |
| *** jgrimm has quit IRC | 17:10 | |
| *** jgrimm has joined #openstack-infra | 17:11 | |
| *** andreykurilin_ has quit IRC | 17:11 | |
| *** harlowja has joined #openstack-infra | 17:16 | |
| mtreinish | fungi, clarkb: any thoughts on getting this pushed through?: https://review.openstack.org/#/c/85431/ | 17:18 |
| *** SumitNaiksatam has quit IRC | 17:18 | |
| clarkb | mtreinish: this week is going to be a bad week | 17:19 |
| mtreinish | clarkb: ok, np it can wait then | 17:20 |
| mtreinish | clarkb: what's this week? | 17:21 |
| *** jpich has quit IRC | 17:21 | |
| clarkb | mtreinish: heartbleed | 17:21 |
| mtreinish | ahh | 17:21 |
| *** _nadya_ has quit IRC | 17:22 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:22 | |
| dstufft | dhellmann: FWIW as I udnerstand it the namespace bug is basically unresolvable in pip | 17:23 |
| *** Ajaeger1 has joined #openstack-infra | 17:24 | |
| *** sdague has quit IRC | 17:27 | |
| *** sdague has joined #openstack-infra | 17:29 | |
| *** Sukhdev has quit IRC | 17:30 | |
| *** e0ne has joined #openstack-infra | 17:31 | |
| *** jerryz has joined #openstack-infra | 17:32 | |
| dims | jamespage, fyi, ran into this -r/--arch requirement in euca command line https://eucalyptus.atlassian.net/browse/DOC-807 http://logs.openstack.org/57/86057/2/check/check-grenade-dsvm/f459067/logs/old/devstacklog.txt.gz | 17:32 |
| dims | jamespage, thanks for the heads up about qemu version | 17:32 |
| marun | fungi: danke | 17:33 |
| *** amcrn has joined #openstack-infra | 17:35 | |
| *** Ryan_Lane has joined #openstack-infra | 17:35 | |
| Ajaeger1 | clarkb: the translation jobs are working fine with your patch as far as I could see when I checked earlier today. Could you review https://review.openstack.org/#/c/85973/ as followup, please? | 17:35 |
| jerryz | HI, anybody knows whether i can use DependentPipelineManager on Check pipeline? What if only two changes from two different projects tested together can pass? | 17:35 |
| clarkb | jerryz: you can. it may not always work as expected though | 17:36 |
| clarkb | Ajaeger1: right now, still dealing with heartbleed fallout | 17:36 |
| clarkb | Ajaeger1: infra is probably going to fall behind this week especially with people at pycon | 17:36 |
| fungi | Ajaeger1: also, what is an obsolete translation entry? | 17:36 |
| *** ianw has quit IRC | 17:36 | |
| Ajaeger1 | clarkb: I see. | 17:36 |
| Ajaeger1 | fungi: those are the entries that caused us troubles. These are commented out entries. | 17:37 |
| Ajaeger1 | https://review.openstack.org/#/c/85958/ for one patch where obsolete entries were removed | 17:37 |
| fungi | Ajaeger1: aha, so entries where there is no msgid because the original text is no longer present in the software | 17:38 |
| Ajaeger1 | fungi: exactly | 17:39 |
| Ajaeger1 | fungi: thanks for the review | 17:40 |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/reviewday: Quit smoking. https://review.openstack.org/86104 | 17:41 |
| *** fbo_away has quit IRC | 17:42 | |
| *** fbo_away has joined #openstack-infra | 17:43 | |
| *** bhuvan_ has quit IRC | 17:43 | |
| *** fbo_away is now known as fbo | 17:43 | |
| jerryz | clarkb: thanks. In my scenario, the two changes heavily depend on each other, they can not pass on one's own, only Z=C1+C2 can pass. will DependentPiplelineManager still apply? | 17:43 |
| *** SumitNaiksatam has joined #openstack-infra | 17:43 | |
| clarkb | jerryz: pipeline managers apply to all changes though | 17:43 |
| fungi | jerryz: what you're really wanting is cross-project dependencies, i suspect... a feature we've been discussing for a while | 17:46 |
| fungi | jerryz: unless you're talking about changes on the same project and branch, in which case just make them depend on each other in gerrit and zuul will handle that case automatically | 17:46 |
| *** jcoufal has quit IRC | 17:46 | |
| *** MarkAtwood has quit IRC | 17:47 | |
| jerryz | fungi: cross-project dependency it is | 17:47 |
| fungi | jerryz: that's definitely a feature we want, but we don't currently have a way to support it | 17:48 |
| *** _nadya_ has joined #openstack-infra | 17:48 | |
| jerryz | fungi: what do you do know? like manually force one change to merge and test the other? | 17:48 |
| fungi | jerryz: though there has been a fair amount of design work around it, no implementation work has been started yet as far as i know | 17:48 |
| fungi | jerryz: we make sure changes are implemented in pieces so they can be correctly serialized | 17:49 |
| *** esker has joined #openstack-infra | 17:49 | |
| *** Sukhdev has joined #openstack-infra | 17:50 | |
| *** e0ne has quit IRC | 17:50 | |
| fungi | jerryz: even when we eventually implement cross-project dependencies, we still will require them to be able to be tested and landed in series. allowing intertwined commits where neither can work unless both are merged supports very bad design habits | 17:50 |
| *** resker has joined #openstack-infra | 17:50 | |
| *** e0ne has joined #openstack-infra | 17:51 | |
| jerryz | fungi: a seamless switch-over ? | 17:51 |
| fungi | jerryz: right, you need to be able to land support in one project, then land the code which uses it in a different project, support proper deprecation of features and configuration, et cetera | 17:52 |
| jerryz | fungi: got it. i will see if we can serialize in my scenario. | 17:52 |
| fungi | jerryz: otherwise you break people doing continuous deployment | 17:52 |
| *** esker has quit IRC | 17:53 | |
| fungi | jerryz: just like you wouldn't want to land multiple changes in one project where the first breaks the software and then the next one fixes it. proper testing already prevents that... now we're just talking about how to spread the serialization properly between multiple projects | 17:53 |
| *** hogepodge has joined #openstack-infra | 17:55 | |
| *** e0ne has quit IRC | 17:55 | |
| *** resker has quit IRC | 17:55 | |
| *** Guest44106 is now known as Adri2000 | 17:55 | |
| *** gokrokve has quit IRC | 17:55 | |
| dhellmann | dstufft: can it be resolved somewhere else, like setuptools, or is this a case of allowing multiple ways to do something causing fundamental incompatibilities? | 17:56 |
| *** anteaya has quit IRC | 17:56 | |
| dstufft | dhellmann: I'm not super knowledgeable about this issue, it long predates my involvement, but my understanding is there are multiple ways which are incompatible. It might be possible for setuptools to resolve those incompatibilities I dunno | 17:56 |
| dstufft | or they might just be fundamentally incompat | 17:56 |
| dhellmann | dstufft: ok | 17:57 |
| *** e0ne has joined #openstack-infra | 17:57 | |
| *** zns has joined #openstack-infra | 17:58 | |
| *** jamielennox|away is now known as jamielennox | 17:58 | |
| *** zns has quit IRC | 17:58 | |
| *** melwitt has joined #openstack-infra | 17:58 | |
| *** zns has joined #openstack-infra | 17:59 | |
| *** mspreitz has joined #openstack-infra | 17:59 | |
| dhellmann | dstufft: it's not really clear how much this affects us with sdague's devstack changes, which is part of why I wanted to start the conversation | 17:59 |
| sdague | yeh, now that we are punting on -e for oslo libs, devstack will stop being odd for folks | 18:00 |
| dhellmann | sdague: I think we'll have a period where existing systems need to be cleaned up, but yeah, I hope that change helps a lot | 18:03 |
| *** esker has joined #openstack-infra | 18:03 | |
| *** thedodd has quit IRC | 18:03 | |
| *** MarkAtwood has joined #openstack-infra | 18:04 | |
| *** che-arne has quit IRC | 18:04 | |
| *** thedodd has joined #openstack-infra | 18:06 | |
| *** sandywalsh_ has joined #openstack-infra | 18:07 | |
| *** sandywalsh has quit IRC | 18:09 | |
| *** wchrisj has joined #openstack-infra | 18:10 | |
| *** zns has quit IRC | 18:11 | |
| *** freyes has joined #openstack-infra | 18:12 | |
| *** thuc has joined #openstack-infra | 18:13 | |
| *** zns has joined #openstack-infra | 18:13 | |
| *** bhuvan has joined #openstack-infra | 18:15 | |
| *** thuc_ has quit IRC | 18:16 | |
| *** thuc has quit IRC | 18:17 | |
| *** dcramer_ has quit IRC | 18:19 | |
| *** UtahDave has joined #openstack-infra | 18:20 | |
| sdague | yeh, I left the clean on olso at least | 18:20 |
| sdague | we could add the same clean logic for other packages | 18:20 |
| *** dcramer_ has joined #openstack-infra | 18:21 | |
| *** mdenny has quit IRC | 18:23 | |
| *** mdenny has joined #openstack-infra | 18:23 | |
| *** hogepodge has quit IRC | 18:26 | |
| *** _nadya_ has quit IRC | 18:26 | |
| *** mspreitz_ has joined #openstack-infra | 18:28 | |
| *** mspreitz has quit IRC | 18:29 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 18:29 | |
| *** SumitNaiksatam has quit IRC | 18:29 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 18:29 | |
| *** mdenny has quit IRC | 18:30 | |
| *** mdenny has joined #openstack-infra | 18:30 | |
| *** mspreitz has joined #openstack-infra | 18:31 | |
| *** mspreitz_ has quit IRC | 18:32 | |
| *** mspreitz_ has joined #openstack-infra | 18:34 | |
| *** dcramer_ has quit IRC | 18:35 | |
| *** persia has quit IRC | 18:35 | |
| *** mspreitz has quit IRC | 18:36 | |
| *** mspreitz_ is now known as mspreitz | 18:36 | |
| *** 23LAAAIJP has joined #openstack-infra | 18:36 | |
| *** dcramer_ has joined #openstack-infra | 18:37 | |
| *** 23LAAAIJP has quit IRC | 18:37 | |
| *** 23LAAAIJP has joined #openstack-infra | 18:37 | |
| *** 23LAAAIJP is now known as persia | 18:37 | |
| *** thuc_ has joined #openstack-infra | 18:37 | |
| *** atiwari has joined #openstack-infra | 18:39 | |
| *** saju_m has quit IRC | 18:44 | |
| *** yolanda has quit IRC | 18:47 | |
| *** thuc has joined #openstack-infra | 18:50 | |
| *** thedodd has quit IRC | 18:50 | |
| *** ildikov_ has joined #openstack-infra | 18:51 | |
| *** thedodd has joined #openstack-infra | 18:52 | |
| *** thuc_ has quit IRC | 18:53 | |
| *** thuc has quit IRC | 18:54 | |
| pabelanger | fungi, do you have access to the rackspace mycloud interface? I'd be curious to see how many of your images are in a building (0%) / error state. | 18:56 |
| *** dkehn_ has joined #openstack-infra | 18:56 | |
| *** yolanda has joined #openstack-infra | 18:56 | |
| *** weshay has quit IRC | 18:57 | |
| *** ihrachyshka has quit IRC | 18:57 | |
| fungi | pabelanger: i do, but don't have time to jump in there and look just now | 18:58 |
| clarkb | pabelanger: and three is the the open question of whether nor not it is currently safe to log in to it | 18:58 |
| pabelanger | fungi, clarkb: understood | 18:58 |
| fungi | clarkb: well, this would be the jenkins tenant, which nodepool is constantly authenticating to | 18:58 |
| sdague | clarkb or jeblair: https://review.openstack.org/#/c/84829/ the branch override piece would be nice to land if we could, unit tests seem solid now | 18:59 |
| fungi | so that ship has probably sailed (but maybe not if their dashboard and api endpoint are individually vulnerable in separate ways) | 18:59 |
| clarkb | can any rackspace folks address that? if we log in to the web ui and/or talk to openstack api endpoints at rackspace are we 1. vulnerable to credential leakage and 2. if not because patching has happened are we vulnerable to mitm because ssl certs haven't been swapped out? | 18:59 |
| clarkb | phschwartz: ^ I think you mentioned you are at rax | 18:59 |
| fungi | keeping tabs on https://community.rackspace.com/general/f/34/t/3596 too | 19:00 |
| fungi | also, meeting time | 19:00 |
| clarkb | and https://status.rackspace.com/ but none of them address the problem direclty as it pertains to me logging in and doing stuff | 19:00 |
| phschwartz | clarkb: I can pass the question along as I am not sure of the answer | 19:00 |
| *** derekh has joined #openstack-infra | 19:00 | |
| clarkb | phschwartz: thanks | 19:01 |
| *** weshay has joined #openstack-infra | 19:02 | |
| clarkb | phschwartz: I suppose there is a third option where we were never vulnerable because heartbeat wasn't used or older openssl was used. But concrete info would be handy | 19:02 |
| *** ianw has joined #openstack-infra | 19:02 | |
| *** yolanda has quit IRC | 19:04 | |
| *** esker has quit IRC | 19:07 | |
| *** maxbit has quit IRC | 19:10 | |
| *** gokrokve has joined #openstack-infra | 19:13 | |
| phschwartz | clarkb | 19:13 |
| clarkb | phschwartz: o/ | 19:14 |
| phschwartz | clarkb: It looks like it is option 3. Our F5's are running a version of openssl that is not affected. | 19:15 |
| phschwartz | all of the openstack api's are behind LBs, but other api's in our infra are still being verified to be behind LBs or that they do not have an affected openssl version. | 19:15 |
| clarkb | phschwartz: what about web dashboard? | 19:16 |
| *** cody-somerville has quit IRC | 19:16 | |
| *** dangers is now known as dangers_away | 19:16 | |
| phschwartz | Sorry don't know which dashboard your talking about. There are a lot that we have. mind passing a url (or are you talking about the mycloud.rackspace.com) | 19:17 |
| *** weshay has quit IRC | 19:17 | |
| clarkb | phschwartz: ya whatever it is I get when I log in to spin up a server | 19:18 |
| clarkb | or add a dns record | 19:18 |
| *** ihrachyshka has joined #openstack-infra | 19:18 | |
| fungi | phschwartz: "https://mycloud.rackspace.com/" | 19:18 |
| phschwartz | Those were tested and don't appear to have the vuln either (actually looks like a majority of our things don't have heartbeat enabled) | 19:20 |
| clarkb | phschwartz: great, thank you for the info | 19:23 |
| phschwartz | clarkb: np | 19:23 |
| *** jcoufal has joined #openstack-infra | 19:24 | |
| *** david_lyle_ has joined #openstack-infra | 19:26 | |
| *** rcleere has quit IRC | 19:26 | |
| *** gyee has quit IRC | 19:28 | |
| *** dklyle has joined #openstack-infra | 19:28 | |
| mordred | clarkb: for HP: https://community.hpcloud.com/status/incident/2533 | 19:29 |
| *** stevebaker has quit IRC | 19:30 | |
| *** stevebaker has joined #openstack-infra | 19:30 | |
| *** david-lyle has quit IRC | 19:30 | |
| clarkb | mordred: thanks | 19:31 |
| *** dklyle has quit IRC | 19:31 | |
| clarkb | so I think we are good to update things via rax and hpcloud api/dashboards | 19:31 |
| clarkb | fungi: ^ | 19:31 |
| *** dklyle has joined #openstack-infra | 19:31 | |
| fungi | clarkb: agreed | 19:31 |
| *** mrmartin has joined #openstack-infra | 19:32 | |
| fungi | though i may need to get an early dinner right after the meeting and start beating on it as soon as i get back from that | 19:32 |
| *** david_lyle_ has quit IRC | 19:32 | |
| clarkb | same here | 19:32 |
| clarkb | I am running on fumes | 19:33 |
| *** ildikov has joined #openstack-infra | 19:33 | |
| fungi | last night was sorta late for me ;) | 19:33 |
| *** jswarren has joined #openstack-infra | 19:35 | |
| *** malini is now known as malini_afk | 19:36 | |
| *** thedodd has quit IRC | 19:37 | |
| phschwartz | what time is the meeting? | 19:38 |
| phschwartz | I want to get more involved and I figure that is one of the steps | 19:38 |
| clarkb | phschwartz: now, sorry I should've mentioned it earlier | 19:38 |
| *** thedodd has joined #openstack-infra | 19:38 | |
| clarkb | phschwartz: we are just wrapping up though so we can get back to dealing with heartbleed, but feel free to join | 19:39 |
| clarkb | and a log will be posted at http://eavesdrop.openstack.org | 19:39 |
| phschwartz | ah, I joined, but will look at the log and will make sure to be there next week | 19:39 |
| phschwartz | so the meeting is at 3 est? I thought it was at 4. my mistake defn | 19:40 |
| *** ociuhandu has quit IRC | 19:40 | |
| *** dklyle is now known as david-lyle | 19:40 | |
| clarkb | phschwartz: it is scheduled for 1900 UTC so depending on DST it moves | 19:41 |
| phschwartz | Thats what got me. lol | 19:41 |
| *** anteaya has joined #openstack-infra | 19:41 | |
| *** jeremyb has quit IRC | 19:41 | |
| *** jeremyb has joined #openstack-infra | 19:41 | |
| fungi | phschwartz: pretty much all openstack team meetings are scheduled based on utc, since we've got people participating worldwide | 19:43 |
| *** thedodd has quit IRC | 19:43 | |
| *** Ryan_Lane1 has joined #openstack-infra | 19:43 | |
| *** markmc has joined #openstack-infra | 19:44 | |
| clarkb | fungi: mordred: I am going to run really quickly and grab a sandwich then get back to the fun | 19:44 |
| fungi | clarkb: okay, thanks | 19:45 |
| *** alff__ has joined #openstack-infra | 19:47 | |
| *** zns has quit IRC | 19:47 | |
| *** Ryan_Lane1 has quit IRC | 19:48 | |
| *** alff has quit IRC | 19:49 | |
| *** Sukhdev has quit IRC | 19:50 | |
| *** mspreitz has quit IRC | 19:50 | |
| *** zns has joined #openstack-infra | 19:51 | |
| ianw | is there anything useful I can do to help the devstack/fedora job we mentioned? | 19:51 |
| *** zns has quit IRC | 19:52 | |
| fungi | ianw: since the tripleo cloud already has fedora nodes, you might talk to pleia2 about the possibility of having us run a simple devstack job there as a proving ground, perhaps in the check-tripleo or experimental-tripleo pipeline initially while rh is working out the quota issues on the second tripleo region they've built | 19:56 |
| *** zns has joined #openstack-infra | 19:57 | |
| ianw | fungi: ok, will do | 19:57 |
| jogo | dims: https://bugs.launchpad.net/nova/+bug/1304107 | 19:58 |
| uvirtbot | Launchpad bug 1304107 in qemu "Libvirt error launching instance - Device 'virtio-net-pci' could not be initialized" [High,Fix released] | 19:58 |
| jogo | ohh it was fixed since last night, looks like it was libvirt | 19:58 |
| fungi | jogo: yep, dims and jamespage were discussing earlier today in here | 19:59 |
| jogo | neato that was quick | 19:59 |
| anteaya | ianw: she is in transit today to pycon, but you can leave a message for her or email | 20:00 |
| fungi | jogo: at least i think that's why they were talking about libvirt 1.2.2 in uca for precise, but maybe that was an unrelated coincidence | 20:01 |
| Ajaeger1 | fungi, anteaya I'm considering to take a graph or two from the presentations at http://docs.openstack.org/infra/publications. Is that ok? What kind of credit do I need to give? | 20:01 |
| fungi | Ajaeger1: i think there should be a license in that repository/branch | 20:02 |
| *** derekh has quit IRC | 20:02 | |
| Ajaeger1 | fungi: not visible on the website, let me check the repo... | 20:03 |
| *** andreykurilin_ has joined #openstack-infra | 20:03 | |
| fungi | Ajaeger1: yeah, i'm not finding it either | 20:03 |
| fungi | that may be a major oversight on our part | 20:03 |
| Ajaeger1 | fungi: can't find it in the repo either - just checked two recent branches | 20:05 |
| fungi | Ajaeger1: i think the intent was that it should be apache 2 or cc-by 3 but i'm not positive | 20:05 |
| fungi | Ajaeger1: please open a bug on that against openstack-ci, since we definitely should correct it asap | 20:05 |
| anteaya | since the point of having them up is for people to use them | 20:06 |
| fungi | Ajaeger1: i think it's safe to work on the assumption that it would use the same licensing as other content on docs.o.o (so apache or cc-by) but we as authors need to have a conversation confirming that so we can correct it properly i think | 20:06 |
| Ajaeger1 | fungi: ok, will add a slide at the end and reference where I've taken the graphs from. Thanks. | 20:09 |
| * fungi disappears to find early dinner, and then get back to the fun | 20:09 | |
| Ajaeger1 | fungi: enjoy! | 20:09 |
| Ajaeger1 | bug reported at https://bugs.launchpad.net/openstack-ci/+bug/1304636 in case any of you wants to give it some quick comments | 20:10 |
| uvirtbot | Launchpad bug 1304636 in openstack-ci "Copyright information missing for publications" [Undecided,New] | 20:10 |
| dims | jogo, take a quick peek at the uca/precise here - https://review.openstack.org/#/c/86057/ - one bug popped out of it - https://bugs.launchpad.net/devstack/+bug/1304571. jamespage mentioned that a fresh qemu will hit the archive today. | 20:14 |
| uvirtbot | Launchpad bug 1304571 in devstack "euca-bundle-image requires a new architecture parameter" [Undecided,In progress] | 20:14 |
| *** jp_at_hp1 has joined #openstack-infra | 20:16 | |
| *** ociuhandu has joined #openstack-infra | 20:17 | |
| *** jp_at_hp has quit IRC | 20:17 | |
| jogo | that is caused by a new version of euca2ools? | 20:19 |
| jogo | sdague: re https://review.openstack.org/85496 | 20:19 |
| ianw | fungi: so just chatting with dprince, he's not opposed to the idea of using ooo cloud for a smaller devstack testing job, but his preference would probably be to be using RAX | 20:19 |
| jogo | how do I drop hacking for g-r? | 20:20 |
| ianw | fungi: if we're running a smaller job just for devstack changes, can it run in the single provider of RAX? | 20:20 |
| *** stevebaker has quit IRC | 20:20 | |
| *** stevebaker has joined #openstack-infra | 20:20 | |
| jogo | clarkb: ^ | 20:21 |
| *** mdenny has quit IRC | 20:21 | |
| *** mdenny has joined #openstack-infra | 20:21 | |
| *** mrmartin has quit IRC | 20:23 | |
| *** sweston has joined #openstack-infra | 20:23 | |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/reviewstats: Fix capitalization of my gerrit username https://review.openstack.org/86142 | 20:24 |
| clarkb | ianw: as long as it doesn't gate | 20:26 |
| clarkb | ianw: the gate piece is sort of the distinguishing criteria | 20:26 |
| ianw | clarkb: yeah, i mean we wouldn't want it gating for some time until it proved it stability anyway | 20:27 |
| sdague | jogo: oh, you need the ability to punch through the mirror | 20:29 |
| sdague | after tc meeting I can ponder | 20:30 |
| *** dcramer_ has quit IRC | 20:30 | |
| *** dprince has quit IRC | 20:31 | |
| *** e0ne has quit IRC | 20:31 | |
| *** e0ne has joined #openstack-infra | 20:32 | |
| ianw | clarkb / fungi : so in terms of getting this done (sorry for the neophyte questions) the first thing would be a separate jenkins job in the job-builder? | 20:32 |
| clarkb | ianw: first thing would be a nodepool image and flavor for fedora nodes | 20:33 |
| clarkb | ianw: then a JJB job | 20:33 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for grenade g-api service start failure bug 1304652 https://review.openstack.org/86145 | 20:34 |
| uvirtbot | Launchpad bug 1304652 in grenade "g-api did not start" [Undecided,New] https://launchpad.net/bugs/1304652 | 20:34 |
| jogo | sdague: sounds good | 20:35 |
| *** e0ne has quit IRC | 20:35 | |
| ianw | clarkb: how do people usually develop on nodepool? it's not clear to me how to setup a sandbox | 20:35 |
| *** dcramer_ has joined #openstack-infra | 20:35 | |
| clarkb | ianw: if you have credentials for a cloud you can set one up. However it is probably sufficient to just run the steps manually on said clouds base image | 20:36 |
| clarkb | ianw: eg you don't need to test nodeppol as much as you have to test the image builds | 20:36 |
| ianw | clarkb: ok, cool, i'll look into that | 20:37 |
| ianw | clarkb: when i've verified that, i'll probably start a thread on openstack-infra, sound good? | 20:38 |
| clarkb | yup | 20:38 |
| *** vhoward has joined #openstack-infra | 20:39 | |
| *** freyes has quit IRC | 20:40 | |
| *** jp_at_hp1 has quit IRC | 20:41 | |
| *** gyee has joined #openstack-infra | 20:41 | |
| *** jp_at_hp has joined #openstack-infra | 20:41 | |
| *** stevebaker has quit IRC | 20:42 | |
| *** stevebaker has joined #openstack-infra | 20:42 | |
| *** Ajaeger1 has quit IRC | 20:43 | |
| *** dcramer_ has quit IRC | 20:44 | |
| *** dcramer_ has joined #openstack-infra | 20:46 | |
| *** eharney has quit IRC | 20:49 | |
| *** hogepodge has joined #openstack-infra | 20:49 | |
| *** ihrachyshka has quit IRC | 20:54 | |
| *** e0ne has joined #openstack-infra | 20:54 | |
| *** e0ne has quit IRC | 20:56 | |
| *** e0ne has joined #openstack-infra | 20:56 | |
| *** westmaas has quit IRC | 20:57 | |
| *** mrda_away is now known as mrda | 20:57 | |
| *** david-lyle is now known as david-lyle_afk | 20:57 | |
| *** mrda is now known as Guest52851 | 20:58 | |
| *** ihrachyshka has joined #openstack-infra | 20:58 | |
| *** e0ne has quit IRC | 20:59 | |
| *** Guest52851 is now known as mrda | 21:02 | |
| *** ruhe2 has joined #openstack-infra | 21:03 | |
| *** zns has quit IRC | 21:03 | |
| *** dkliban has quit IRC | 21:03 | |
| *** ruhe2 has left #openstack-infra | 21:04 | |
| *** aysyd has quit IRC | 21:08 | |
| *** ihrachyshka has quit IRC | 21:08 | |
| sdague | clarkb: got a few minutes for https://review.openstack.org/#/c/84829/ ? we've only got about 7 days until we'd be setting the tempest branch, so I'd really like to be able to get this exercised before then | 21:09 |
| clarkb | sdague: I feel like if I start making exceptions to do stuff that isn't #heartbleed that is a bad thing | 21:11 |
| sdague | clarkb: ok, fair | 21:11 |
| jogo | sdague: so now that TC is over how do take hacking out of g-r | 21:11 |
| sdague | well, as soon as you come up for air, it would be appreciated | 21:11 |
| sdague | jogo: the global requirements job itself can punch the mirror | 21:12 |
| *** jpeeler has quit IRC | 21:12 | |
| sdague | so we should figure out what it needs to do to do that | 21:12 |
| *** jp_at_hp has quit IRC | 21:13 | |
| jogo | sdague: so that will allow us to pull packages from pypi.python.org? | 21:14 |
| asselin | Hi, what version of Jenkins is openstack-infra using? | 21:18 |
| sdague | jogo: that's my thought | 21:18 |
| clarkb | asselin: it should tell you if you visit a jenkins server | 21:18 |
| *** salv-orlando_ has joined #openstack-infra | 21:19 | |
| jogo | clarkb: ^ does that make sense | 21:19 |
| sdague | because I don't think you actually want to require pyflakes upgrade on all the projects before new hacking | 21:19 |
| sdague | which is what's going to happen | 21:19 |
| jogo | sdague: clarkb had a different idea. but I am with you on this one sdague | 21:19 |
| jogo | sdague: well only projects that pin pyflakes | 21:19 |
| jogo | which is trove and oslo-incubator | 21:19 |
| asselin | clarkb, thanks got it | 21:19 |
| jogo | and patches up to fix both | 21:20 |
| clarkb | right no one should pin it directly | 21:20 |
| jogo | clarkb: but people do | 21:20 |
| clarkb | instead should rely on hacking | 21:20 |
| sdague | jogo: so they specify it? | 21:20 |
| jogo | sdague: https://review.openstack.org/#/c/67138/ | 21:20 |
| sdague | wait, if everyone's supposed to rely on hacking, why are these in the list at all? | 21:20 |
| jogo | https://review.openstack.org/#/c/86153/ | 21:20 |
| clarkb | StevenK: hey, re heartbleed any idea of whether or not launchpad is safe now? | 21:20 |
| jogo | sdague: because hacking needs them ;) | 21:20 |
| clarkb | StevenK: or any idea where I should check? #launchpad on freenode? | 21:21 |
| sdague | yeh, but I'd much rather remove them from g-r, let hacking punch the mirror, then not allow people to specify them | 21:21 |
| jogo | sdague: ++ | 21:21 |
| *** salv-orlando has quit IRC | 21:21 | |
| *** salv-orlando_ is now known as salv-orlando | 21:21 | |
| sdague | though is there a transitive dep mirror problem? | 21:21 |
| clarkb | https://identi.ca/launchpadstatus is quiet | 21:22 |
| sdague | clarkb: I guess we don't have a way to say "put this in the mirror, but don't let anyone use it" | 21:22 |
| clarkb | sdague: nope, which is why I am a proponent of decoupling the mirror from enforcement | 21:22 |
| *** denis_makogon has quit IRC | 21:22 | |
| sdague | yeh, I agree | 21:22 |
| jogo | so moving forward what does that mean for hacking? | 21:26 |
| dhellmann | fungi: did you have a chance to look at the updates to https://review.openstack.org/#/c/85487/ ? | 21:26 |
| mordred | jogo, sdague: we _USED_ to need the specific pins per project - I imagine those are just hysterical raisins | 21:27 |
| *** dkranz has quit IRC | 21:27 | |
| sdague | mordred: they used to use pep8 directly | 21:28 |
| *** mfer has quit IRC | 21:28 | |
| sdague | everyone is going through hacking now | 21:28 |
| *** ameade has quit IRC | 21:28 | |
| mordred | yah | 21:28 |
| sdague | which I think is a better place to control those versions | 21:28 |
| mordred | it is | 21:28 |
| mordred | that's the idea at least | 21:28 |
| sdague | I think trove and oslo-incubator are the only exceptions, and I think those are easily fixed | 21:29 |
| dhellmann | sdague: yeah, I don't see us calling pyflakes directly in the incubator | 21:29 |
| sdague | mordred: so I guess the question here is how to get the mirror to have these packages even if not in g-r | 21:30 |
| sdague | from where we currently stand | 21:30 |
| *** reed has quit IRC | 21:30 | |
| sdague | thats where my understanding of infra falls down | 21:30 |
| clarkb | sdague: you cannot | 21:31 |
| clarkb | you could possibly do a supplemental list, but currently we use the mirror for enforcement | 21:31 |
| SlickNik | sdague: trove's using hacking as well. There's this patch in flight to address the issue that jogo just brought up https://review.openstack.org/#/c/67138/3/test-requirements.txt | 21:32 |
| sdague | clarkb: supplemental list would be fine | 21:32 |
| sdague | clarkb: got an entry point for me tot try to propose something? | 21:32 |
| *** dachary has quit IRC | 21:32 | |
| clarkb | sdague: the mirror jobs in openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config | 21:33 |
| clarkb | sdague: they should pass a reqs file to the run mirror script, if you add another file that should work | 21:33 |
| *** dachary has joined #openstack-infra | 21:35 | |
| SlickNik | sdague / jogo: Trove should be good once that patch merges; correct? | 21:35 |
| sdague | clarkb: ok, I'm trying to figure where they pass a file, I just see them passing a git tree | 21:36 |
| jogo | SlickNik: yup | 21:36 |
| clarkb | sdague: oh it may just be a tree | 21:37 |
| clarkb | sdague: then it looks for files in that tree? | 21:37 |
| *** jooools has quit IRC | 21:37 | |
| sdague | clarkb: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/pypi-mirror.yaml | 21:38 |
| sdague | that's where I'm dead ending | 21:38 |
| *** pdmars has quit IRC | 21:39 | |
| clarkb | sdague: ok, I think it will look for files that look like requirements. You may need to add anothe rproject to the list | 21:39 |
| fungi | dhellmann: i started looking at it earlier before you had jobs passing. need to revisit but i think as long as you have some mechanical solution to ensuring parity/symmetry then my concerns from the previous iteration (about not being able to adequately review its sanity) are covered | 21:39 |
| sdague | clarkb: I suppose we could always add mirror-requirements.txt to g-r | 21:40 |
| *** dizquierdo has joined #openstack-infra | 21:40 | |
| dhellmann | fungi: yeah, the latest version has a little tool for doing that check -- you have to have all of openstack checked out to use the tool, but I did :-) | 21:40 |
| sdague | maye that would pick that up? | 21:40 |
| fungi | dhellmann: yep, saw, just didn't finish covering yet what with all the other distractions, but i think this is probably safe until we find better routes | 21:41 |
| *** sweston has quit IRC | 21:43 | |
| *** david_lyle_ has joined #openstack-infra | 21:44 | |
| *** david_lyle_ is now known as david-lyle | 21:44 | |
| *** david-lyle is now known as david_lyle | 21:44 | |
| *** russellb has quit IRC | 21:44 | |
| *** thomasem has quit IRC | 21:45 | |
| *** russellb has joined #openstack-infra | 21:45 | |
| *** david_lyle_ has joined #openstack-infra | 21:46 | |
| *** reed has joined #openstack-infra | 21:46 | |
| *** david-lyle_afk has quit IRC | 21:47 | |
| *** dcramer_ has quit IRC | 21:48 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: add mirror requirements list https://review.openstack.org/86171 | 21:48 |
| *** dcramer_ has joined #openstack-infra | 21:49 | |
| *** david_lyle has quit IRC | 21:50 | |
| *** mriedem has left #openstack-infra | 21:52 | |
| *** ihrachyshka has joined #openstack-infra | 21:52 | |
| *** e0ne has joined #openstack-infra | 21:54 | |
| *** hogepodge has quit IRC | 21:55 | |
| *** e0ne has quit IRC | 21:56 | |
| *** e0ne has joined #openstack-infra | 21:56 | |
| *** andreykurilin_ has quit IRC | 21:58 | |
| *** eharney has joined #openstack-infra | 21:59 | |
| *** andreykurilin_ has joined #openstack-infra | 21:59 | |
| *** markmcclain has quit IRC | 22:00 | |
| *** markmcclain has joined #openstack-infra | 22:01 | |
| *** e0ne has quit IRC | 22:01 | |
| *** markmcclain has quit IRC | 22:02 | |
| *** gondoi is now known as zz_gondoi | 22:02 | |
| *** markmcclain has joined #openstack-infra | 22:02 | |
| *** russellb has quit IRC | 22:03 | |
| *** markmcclain has quit IRC | 22:03 | |
| *** markmcclain has joined #openstack-infra | 22:03 | |
| *** russellb has joined #openstack-infra | 22:03 | |
| *** gokrokve_ has joined #openstack-infra | 22:04 | |
| SergeyLukjanov | sdague, probably I just should sleep a bit, but I have a crazy idea | 22:04 |
| sdague | SergeyLukjanov: I like crazy :) | 22:04 |
| SergeyLukjanov | sdague, mirror-requirements == [pip install global-requirements then pip freeze minus global-requirements] | 22:05 |
| SergeyLukjanov | I mean all transient requirements | 22:05 |
| *** hogepodge has joined #openstack-infra | 22:06 | |
| sdague | SergeyLukjanov: yeh, that could be | 22:06 |
| SergeyLukjanov | sdague, on the other side, why do we need list of them is we'll install them anyway | 22:06 |
| *** dtroyer has quit IRC | 22:07 | |
| SergeyLukjanov | sdague, oh, I see your explanation about versions | 22:07 |
| *** andreykurilin_ has quit IRC | 22:07 | |
| *** gokrokve has quit IRC | 22:07 | |
| *** dcramer_ has quit IRC | 22:10 | |
| *** dtroyer has joined #openstack-infra | 22:10 | |
| *** dizquierdo has quit IRC | 22:12 | |
| mattoliverau | Morning | 22:12 |
| *** markmc has quit IRC | 22:13 | |
| *** andreaf has joined #openstack-infra | 22:13 | |
| *** dstanek has quit IRC | 22:13 | |
| anteaya | mattoliverau: morning | 22:15 |
| anteaya | mattoliverau: I'm trusting you got your ballot question answered? | 22:15 |
| anteaya | I didn't see an email in my inbox from you | 22:15 |
| mattoliverau | anteaya: sorry, not yet, thanks to the heartbleed openssl bug, yesterday became a bit of a race to patch all the servers, and then got sidetracked will other work that was then behind. I think apart of the problem was I, for some reason was dropped from the dev list. Re-subscribed the other day. So getting email again I wonder if I missed the ballot email. | 22:19 |
| *** Sukhdev has joined #openstack-infra | 22:20 | |
| andreaf | sdague: ping | 22:22 |
| sdague | andreaf: what's up? I'm about to drop for a few | 22:23 |
| andreaf | sdague: nothing urgent - I made the devstack change for domain_name here when you have a sec https://review.openstack.org/#/c/86023/ | 22:23 |
| *** sabari has quit IRC | 22:24 | |
| *** sabari has joined #openstack-infra | 22:24 | |
| sdague | andreaf: is there an extra _ in the 2nd change line? | 22:25 |
| sdague | at the end of the var | 22:25 |
| sdague | I think I see another issue actually | 22:25 |
| andreaf | sdague: I'll remove the _ thanks | 22:26 |
| sdague | I think there are 2 small issues | 22:26 |
| sdague | I just put it in the review | 22:26 |
| sdague | then I'm +2 | 22:26 |
| sdague | ok, off for a bit | 22:26 |
| andreaf | sdague: thanks | 22:27 |
| mgagne | I think I broke gerrit, got 500 after submitting my review =( | 22:28 |
| mgagne | now it's back | 22:29 |
| *** jergerber has quit IRC | 22:30 | |
| *** bhuvan has quit IRC | 22:31 | |
| mattoliverau | anteaya: email sent :) | 22:32 |
| *** markmcclain has quit IRC | 22:33 | |
| *** markmcclain has joined #openstack-infra | 22:33 | |
| clarkb | mgagne: sorry for the turbulence | 22:35 |
| *** adalbas has quit IRC | 22:35 | |
| mgagne | clarkb: =) | 22:36 |
| *** prad has quit IRC | 22:37 | |
| fungi | mgagne: *i* broke gerrit | 22:38 |
| fungi | (by stopping it to reset database account passwords) | 22:38 |
| *** gokrokve_ has quit IRC | 22:38 | |
| fungi | awesome of you to notice me trying to be sneaky ;) | 22:39 |
| *** homeless has quit IRC | 22:39 | |
| *** nkinder has quit IRC | 22:40 | |
| fungi | today's disruption brought to you by the letters h and i, and by the number heartbleed | 22:40 |
| *** gokrokve has joined #openstack-infra | 22:43 | |
| *** Ryan_Lane1 has joined #openstack-infra | 22:46 | |
| *** andreaf has quit IRC | 22:46 | |
| *** andreaf has joined #openstack-infra | 22:47 | |
| * mgagne watches fungi | 22:47 | |
| *** adalbas has joined #openstack-infra | 22:47 | |
| fungi | mgagne: do your worst. my government was watching me long before you even thought of it | 22:48 |
| * fungi makes obscene gestures at the nsa | 22:49 | |
| *** atiwari has quit IRC | 22:49 | |
| *** dims has quit IRC | 22:49 | |
| *** thuc has joined #openstack-infra | 22:50 | |
| *** msabramo has quit IRC | 22:50 | |
| *** Ryan_Lane1 has quit IRC | 22:50 | |
| *** melwitt has quit IRC | 22:52 | |
| *** david_lyle_ has quit IRC | 22:55 | |
| *** thuc has quit IRC | 22:55 | |
| *** e0ne has joined #openstack-infra | 22:55 | |
| *** yamahata has quit IRC | 22:58 | |
| jhesketh | Morning | 22:58 |
| *** e0ne has quit IRC | 22:58 | |
| StevenK | clarkb: I daresay it is, but either there, or #canonical-sysadmin | 22:58 |
| *** changbl has quit IRC | 23:00 | |
| anteaya | mattoliverau: your status with the -dev ml does not effect the status of the ballot | 23:01 |
| *** zehicle_at_dell has joined #openstack-infra | 23:02 | |
| * jhesketh catches up on meeting minutes | 23:04 | |
| jhesketh | the infra meeting is now 5am for me :-( | 23:04 |
| StevenK | The TripleO meeting used to be at 3am for me, but I'm not sure what it is now with daylight savings | 23:05 |
| anteaya | jhesketh: :( | 23:05 |
| jhesketh | lets just move to a flat world with 1 timezone ;-) | 23:05 |
| anteaya | was brief today, fungi chaired and wanted to get back to fixing all the things | 23:06 |
| *** msabramo has joined #openstack-infra | 23:06 | |
| anteaya | I had rebooted my weechat server and it didn't allow me ssh access until the last few minutes of the meeting | 23:06 |
| anteaya | jhesketh: sounds exciting | 23:06 |
| anteaya | got anyplace in mind? | 23:06 |
| *** MarkAtwood has quit IRC | 23:07 | |
| *** MarkAtwood has joined #openstack-infra | 23:07 | |
| *** jcoufal has quit IRC | 23:08 | |
| fungi | i am notorious for wanting to fix things | 23:08 |
| anteaya | you are | 23:09 |
| *** dkranz has joined #openstack-infra | 23:09 | |
| jhesketh | heh, nice | 23:11 |
| phschwartz | fungi: blasphemy | 23:16 |
| *** wenlock has quit IRC | 23:16 | |
| phschwartz | hacky work arounds are all that are allowed ;) | 23:16 |
| fungi | phschwartz: after all, the universe perpetually tends toward an ever increasing state of entropy? | 23:18 |
| *** mriedem1 has joined #openstack-infra | 23:19 | |
| phschwartz | fungi: See, someone who sees things my way | 23:22 |
| *** ihrachyshka_ has joined #openstack-infra | 23:29 | |
| *** thuc has joined #openstack-infra | 23:32 | |
| *** ihrachyshka has quit IRC | 23:32 | |
| *** thuc_ has joined #openstack-infra | 23:33 | |
| *** msabramo1 has joined #openstack-infra | 23:35 | |
| *** nkinder has joined #openstack-infra | 23:36 | |
| *** thuc has quit IRC | 23:36 | |
| *** msabramo has quit IRC | 23:37 | |
| *** rlandy has quit IRC | 23:38 | |
| *** markmcclain has quit IRC | 23:40 | |
| *** markmcclain has joined #openstack-infra | 23:40 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add example of comment formatting https://review.openstack.org/85930 | 23:41 |
| *** UtahDave has quit IRC | 23:42 | |
| *** flaper87 is now known as flaper87|afk | 23:43 | |
| *** dstanek has joined #openstack-infra | 23:44 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add note on ci.openstack.org source https://review.openstack.org/85933 | 23:45 |
| *** mriedem1 has quit IRC | 23:47 | |
| openstackgerrit | Sean Dague proposed a change to openstack-dev/pbr: make pbr use hacking directly https://review.openstack.org/86189 | 23:48 |
| *** dteselkin_ has quit IRC | 23:49 | |
| *** dteselkin_ has joined #openstack-infra | 23:49 | |
| mordred | sdague: does that work ^^ ? | 23:51 |
| mordred | sdague: we did the other thing before because hacking depends on pbr and circular dependency | 23:51 |
| sdague | mordred: tox -e pep8 did | 23:51 |
| mordred | cool | 23:52 |
| mordred | I support the idea | 23:52 |
| sdague | so beyond that, I make no guaruntees | 23:52 |
| sdague | that, however, is what was blocking this - https://review.openstack.org/#/c/86171/ | 23:52 |
| *** gokrokve has quit IRC | 23:52 | |
| *** e0ne has joined #openstack-infra | 23:55 | |
| *** derekh has joined #openstack-infra | 23:56 | |
| *** yamahata has joined #openstack-infra | 23:58 | |
| *** mrda is now known as mdavies | 23:58 | |
| *** mdavies is now known as mrda | 23:59 | |
| *** msabramo1 has quit IRC | 23:59 | |
| *** e0ne has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!