| anteaya | mattoliverau: yeah, thought it was a really small deer for the longest time, the colour was right but the size was wrong | 00:00 |
|---|---|---|
| anteaya | and the behaviour was all wrong for a deer | 00:00 |
| zaro | jhesketh: anyways don't list to me, we should just wait to hear from jeblair. i wouldn't be surprised if he waves his wand and POOF! | 00:00 |
| anteaya | I could see its chest but not its head until it moved | 00:00 |
| devananda | adam_g: fwiw, there's an E-R query for 1326754 but it doesn't seem to be commenting on any failures in ironic's queues | 00:01 |
| mattoliverau | anteaya: amazing, you must've been in the right place at the right time! | 00:01 |
| jhesketh | zaro: yeah, it would be quite a re-write, but yeah, curious to see what jeblair knows | 00:02 |
| adam_g | devananda, hmm. has the ironic gate been getting any E-R comments? i added one a while ago and still haven't seen it comment, but i'm not sure the bug exists anymore (the NodeLocked thing) | 00:03 |
| devananda | adam_g: no, none at all. I pointed that out to clarkb earlier but didn't get any traction | 00:03 |
| devananda | adam_g: i've searched my inbox for all comments on all ironic patch sets in the last 7 days and didn't see a single one from E-R | 00:04 |
| anteaya | mattoliverau: yeah, I happened upon it, I think we both were surprised but the wolf was really curious and not afraid, it held its ground | 00:04 |
| anteaya | it was about 80 metres away | 00:04 |
| anteaya | second time I ever saw a wolf | 00:05 |
| adam_g | devananda, weird, its certainly being found. its tracking @ http://status.openstack.org/elastic-recheck/ but not commenting. | 00:06 |
| mattoliverau | anteaya: scary, I've never seen one outsde a zoo, there isn't many over this way :P | 00:07 |
| anteaya | devananda: there are still some unclassified failures for ironic here: http://status.openstack.org/elastic-recheck/data/uncategorized.html | 00:07 |
| anteaya | mattoliverau: I wasn't scared until after it moved off | 00:07 |
| anteaya | I knew it was curious and had a feeling it was going to try to circle behind me | 00:07 |
| anteaya | dogs and wolves do that | 00:07 |
| anteaya | the circle behind you and follow you | 00:07 |
| anteaya | they | 00:07 |
| *** lakshmiS has quit IRC | 00:08 | |
| anteaya | mattoliverau: how was your weekend? | 00:09 |
| mattoliverau | anteaya: umm, <sarcasm> that isn't scary at all </sarcasm> .. and I bet also don't run or they'll chase you! | 00:09 |
| *** lcostantino has quit IRC | 00:09 | |
| anteaya | I turned and walked toward the house and found my father, who came out with me and we walked to where the wolf had been | 00:10 |
| clarkb | devananda: its not a ironic specific problem... | 00:10 |
| clarkb | devananda: we are working onit | 00:10 |
| anteaya | no, that is bears, for wolves, run | 00:10 |
| clarkb | devananda: there just isn't anything to do for ironic right now | 00:10 |
| fungi | mattoliverau: but you get dingoes instead | 00:10 |
| anteaya | you just stand there, they will attack anyway if they are going to attack | 00:10 |
| *** zhiyan_ is now known as zhiyan | 00:10 | |
| mattoliverau | anteaya: mine was great, the wife and I spend the long weekend in Tasmania, so we explored Hobart, drove to a move historic town, drove up mount wellington and met up with jhesketh and his lovely wife for dinner! | 00:10 |
| mattoliverau | fungi: yeah, but they only eat babies :P | 00:11 |
| *** reed has quit IRC | 00:11 | |
| fungi | touché | 00:11 |
| anteaya | mattoliverau: sounds like an awesome weekend | 00:11 |
| clarkb | fungi: I am going to start spinnnig up those new nodes shortly | 00:11 |
| fungi | clarkb: sounds good. i'm mostly not around fwiw | 00:11 |
| mattoliverau | anteaya: it was, it's a beautiful part of the world :) jhesketh is a lucky man to work from there! | 00:11 |
| devananda | clarkb: what projects does E-R watch and post comments on? is there a defined list somewhere? | 00:12 |
| anteaya | jhesketh: I hear you are a lucky man | 00:12 |
| devananda | clarkb: recheckwatchbot.yaml? | 00:12 |
| jhesketh | anteaya: Indeed I am. mattoliverau is learning to visit more often ;-) | 00:13 |
| anteaya | devananda: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries | 00:13 |
| clarkb | devananda: it listens to everything | 00:13 |
| anteaya | devananda: those are the queries for all the bugs e-r knows about | 00:13 |
| clarkb | fungi: should be fine, just adding nodes | 00:13 |
| anteaya | devananda: if there is a query there, e-r knows about it, if not, it doesn't | 00:13 |
| devananda | anteaya: yes, and there are several in there that match ironic bugs, but we're not seeing *any* comemnts from E-R *ever*, and i'm trying to figure out why | 00:13 |
| *** mmaglana has quit IRC | 00:13 | |
| anteaya | jhesketh: :D | 00:13 |
| clarkb | devananda: because we are behind | 00:13 |
| devananda | clarkb: i get that. sorry if i'm just being dense, but why would e-r comment on nova but not ironic? | 00:14 |
| devananda | if it's that far behind that that's preventing it from commenting | 00:14 |
| clarkb | devananda: it isn't commenting on nova | 00:15 |
| devananda | clarkb: it is ... | 00:15 |
| clarkb | and if it does it just pure luck | 00:15 |
| *** melwitt has quit IRC | 00:15 | |
| devananda | heh | 00:15 |
| clarkb | nova has more changes by an order of magnitude | 00:15 |
| clarkb | therefore chances of things being indexed for random nova change are higher | 00:15 |
| clarkb | arg is_openstack_project isn't actually checking if things are an openstack project | 00:17 |
| clarkb | jogo: mtreinish ^ | 00:18 |
| clarkb | devananda: return "tempest-dsvm-full" in self.comment maybe part of the problem too | 00:18 |
| devananda | yep, that looks likely | 00:19 |
| jerryz | fungi clarkb: i have a nodepool that doesn't mark nodes as used if a job runs on them. does jenkins support two nodepool subscribe to zmq publisher? | 00:19 |
| jesusaurus | clarkb: what does is_openstack_project do? | 00:20 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add four new logstash-worker nodes. https://review.openstack.org/98925 | 00:20 |
| clarkb | fungi: jhesketh SergeyLukjanov ^ please don't merge that (I will do it when I have DNS records added) but reviews would be great | 00:20 |
| clarkb | jesusaurus: it determines if e-r should check bugs for that project | 00:20 |
| devananda | jesusaurus: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/elasticRecheck.py#n312 | 00:21 |
| jesusaurus | ahh | 00:21 |
| clarkb | jerryz: nodepool can listen to two jenkins | 00:21 |
| devananda | clarkb: thanks. I think this is the reason. | 00:21 |
| clarkb | jesusaurus: we have 8 | 00:21 |
| clarkb | er jerryz ^ | 00:21 |
| clarkb | devananda: well right now it wouldn' | 00:21 |
| clarkb | t work regardless | 00:21 |
| clarkb | devananda: I am adding capacity now to combat backlog | 00:21 |
| jhesketh | clarkb: noted | 00:21 |
| devananda | clarkb: that's fair. but it never worked, even during periods with no backlog | 00:22 |
| *** weshay has joined #openstack-infra | 00:22 | |
| devananda | clarkb: filing a bug so this gets fixed at some point | 00:22 |
| jerryz | clarkb: yeah, but can one jenkins support two nodepools? | 00:22 |
| *** gokrokve has quit IRC | 00:24 | |
| clarkb | jerryz: oh for that probably not | 00:26 |
| clarkb | jerryz: actually maybe | 00:26 |
| clarkb | since nodepool should just ignore cases where it doesn't knwo about a node | 00:26 |
| *** matsuhashi has joined #openstack-infra | 00:27 | |
| clarkb | similar to hwo we have long lived slaves | 00:27 |
| clarkb | nodepool should treat things the same if they came from another nodepool | 00:27 |
| jerryz | clarkb: ok | 00:27 |
| jerryz | clarkb: then my issue with nodepool not marking node as used is not caused by this | 00:27 |
| clarkb | probably not | 00:28 |
| clarkb | is your nodepool getting the build started event from zmq? | 00:28 |
| clarkb | it should log that in the debug log | 00:28 |
| jerryz | clarkb: i will take a look. if no log, i will trigger a job | 00:29 |
| devananda | jogo, mtreinish: https://bugs.launchpad.net/openstack-ci/+bug/1328318 | 00:29 |
| uvirtbot | Launchpad bug 1328318 in openstack-ci "elastic-recheck not matching ironic test runs" [Undecided,New] | 00:29 |
| openstackgerrit | A change was merged to openstack/requirements: Add bash8 to global-requirements https://review.openstack.org/97625 | 00:31 |
| *** jaypipes has quit IRC | 00:32 | |
| *** ArxCruz_ has joined #openstack-infra | 00:32 | |
| *** ArxCruz has quit IRC | 00:33 | |
| jerryz | clarkb: i didn't see any log saying build started | 00:36 |
| *** chuckC has quit IRC | 00:37 | |
| clarkb | jerryz: so I would check your zmq event publish connectivity | 00:37 |
| jerryz | clarkb: netstat shows there is connection | 00:37 |
| clarkb | make sure the nodepool is talking to it and that that job is configured to emit events or that the jenkins server is configured to emit them globally | 00:37 |
| *** marcoemorais has quit IRC | 00:38 | |
| *** marcoemorais has joined #openstack-infra | 00:38 | |
| *** mmaglana has joined #openstack-infra | 00:43 | |
| jerryz | clarkb: how to restart zmq publisher, just restart jenkins? i see there are 40 connections with CLOSE_WAIT status left by a nodepool for test, which i already stopped. | 00:43 |
| clarkb | jerryz: yes though you might be able to get away with changing the port in the global config, running a job, then changing the port back | 00:45 |
| *** aconrad_ has quit IRC | 00:45 | |
| clarkb | I believe that will create new objects in the jvm | 00:45 |
| devananda | anteaya: regarding http://status.openstack.org/elastic-recheck/data/uncategorized.html#gate-ironic-python26 | 00:49 |
| devananda | anteaya: my perhaps flawed understanding is that logstash doesn't index py* jobs and therefor E-R queries for them aren't helpful. Am I wrong? | 00:49 |
| clarkb | devananda: it does index the console logs for those jobs | 00:50 |
| *** chuckC has joined #openstack-infra | 00:50 | |
| devananda | clarkb: interesting. my logstash queries aren't matching them ... i'll keep trying | 00:50 |
| *** yamahata has quit IRC | 00:52 | |
| clarkb | fungi: jhesketh SergeyLukjanov ok dns records just created | 00:54 |
| clarkb | I am going to self approve now | 00:55 |
| jhesketh | clarkb: noted | 00:55 |
| * jhesketh waits for the world to break | 00:55 | |
| clarkb | jhesketh: :P it should not cause anything to break | 00:55 |
| clarkb | worst case we have derpy indexers and don't index things | 00:55 |
| jhesketh | heh, all good, just teasing | 00:56 |
| clarkb | now back to why qemu-nbd and mount are not doing what I expect them to | 00:56 |
| clarkb | I just want to mount a partition in a qcow2 :( | 00:56 |
| clarkb | hrm maybe my qcow2 isn't good | 00:57 |
| StevenK | clarkb: What does file say? | 00:58 |
| clarkb | file reports the correct thing | 00:58 |
| clarkb | but qemu-nbd -c /dev/nbd1 pathto.qcow2 && mount /dev/nbd1 /someplace fails demanding I specify a fs type | 00:58 |
| clarkb | if I specify ext* it barfs | 00:58 |
| clarkb | and fdisk on /dev/nbd1 throws up | 00:59 |
| StevenK | clarkb: /dev/nbd1p<num> | 00:59 |
| clarkb | StevenK: that doesn't exist even after a partprobe | 00:59 |
| StevenK | clarkb: Just to show it does work (for me at least): http://paste.openstack.org/show/83462/ | 01:00 |
| clarkb | hrm | 01:00 |
| clarkb | you are trusty yes? | 01:00 |
| StevenK | This is Saucy | 01:00 |
| clarkb | ok this is precise | 01:00 |
| StevenK | Haha | 01:00 |
| clarkb | I mean things should still work | 01:00 |
| clarkb | but maybe nodepool will need to be upgraded too... | 01:01 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/elastic-recheck: Broaden 1326312 to match py26/py27 failures https://review.openstack.org/98929 | 01:01 |
| StevenK | clarkb: Give me a few, I shall try on a Precise server I have around | 01:01 |
| clarkb | lol I use nbd0 and that gives me a p1 | 01:01 |
| clarkb | guessing someting has used nbd1 and hates me | 01:02 |
| devananda | after a bug fix lands upstream, how long should we wait to remove the e-r query? | 01:02 |
| StevenK | Oh, what -c did you use for qemu-nbd? | 01:02 |
| clarkb | StevenK: first time was /nbd1 | 01:02 |
| clarkb | er /dev/nbd1 | 01:02 |
| devananda | anteaya: ^ should categorize all of those ironic py26/27 failures -- judging by the timestamps and looking at a few, I think they're all the same bug | 01:02 |
| clarkb | but then I switched to nbd0 and happy now | 01:03 |
| clarkb | devananda: usually we wait ~2 weeks to see te full period of indexed logs not return any hits | 01:03 |
| StevenK | clarkb: Strange, you might be right that something is using nbd1 | 01:03 |
| devananda | clarkb: ack | 01:04 |
| anteaya | devananda: this was going to be my suggestion: message:"FAIL: ironic.nova.tests.scheduler.test_ironic_host_manager.IronicHostManagerChangedNodesTestCase.test_get_all_host_states" AND tags:"console" AND project:"openstack/ironic" AND build_queue:"gate" | 01:04 |
| *** michchap has joined #openstack-infra | 01:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add four new logstash-worker nodes. https://review.openstack.org/98925 | 01:06 |
| *** aconrad_ has joined #openstack-infra | 01:07 | |
| jerryz | clarkb: after i restarted jenkins, zmq publisher works again | 01:09 |
| *** arnaud has joined #openstack-infra | 01:10 | |
| anteaya | devananda: and I agree they are all the same bug, it seems to have occured on June 4th and then been fixed | 01:10 |
| jerryz | clarkb: wonder why tcp sockets piled up and didn't get closed | 01:10 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/elastic-recheck: Broaden 1286635 https://review.openstack.org/98931 | 01:13 |
| devananda | anteaya: that ^ should start catching a few more of the network failure related issues | 01:14 |
| clarkb | jerryz: probably a zmq bug | 01:15 |
| *** gema has quit IRC | 01:17 | |
| *** jhesketh has quit IRC | 01:17 | |
| jerryz | clarkb: though i haven't been able to reproduce it. maybe because my nodepool for test got restarted constantly | 01:18 |
| *** sweston has joined #openstack-infra | 01:20 | |
| *** yaguang has joined #openstack-infra | 01:20 | |
| anteaya | devananda: ah okay, I get results from that query | 01:22 |
| *** ramashri has quit IRC | 01:22 | |
| anteaya | I don't know enough to review queries yet so let's see what the reviewers say | 01:22 |
| *** gema has joined #openstack-infra | 01:25 | |
| mikal | anteaya: do you understand what Sean Dague means by "work unit testing" in his email about nova-network? | 01:25 |
| mikal | Sorry, work unit logging | 01:25 |
| *** godp1301 has joined #openstack-infra | 01:25 | |
| *** marcoemorais has quit IRC | 01:26 | |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Fixed KEY_DIR for the trove-buildimage-mysql job https://review.openstack.org/98933 | 01:26 |
| clarkb | mikal: I think he is saying that distinct things like create ip attach ip etc aren't logged | 01:26 |
| clarkb | so it isn't traceable | 01:26 |
| clarkb | mikal: though I don;t actually know what a nova network unit of work is | 01:27 |
| *** godp1301 has quit IRC | 01:27 | |
| clarkb | think of it is a trace though | 01:27 |
| *** adalbas has joined #openstack-infra | 01:27 | |
| *** adalbas has quit IRC | 01:27 | |
| *** godp1301 has joined #openstack-infra | 01:27 | |
| *** marun has quit IRC | 01:28 | |
| anteaya | mikal: I'm blanking on context can I get a link to the email? | 01:28 |
| *** aconrad_ has quit IRC | 01:28 | |
| anteaya | mikal: ah okay | 01:28 |
| anteaya | mikal: I think what is happening is that nova-network is lacking logging for unit tests | 01:29 |
| anteaya | mikal: so when nova-network unit tests fail, there is no logging, or not enough logging to debug the unit test failures for nova-network | 01:29 |
| anteaya | which is understandable since it was frozen and everyone expected it to be deprecated | 01:29 |
| anteaya | but now we are using it in our testing and nova-network unit test failures that can't be debugged is allowing gate races to creep in | 01:30 |
| fungi | anteaya: not unit tests, but under operation of the server. when tempest tests try to create too many instances in devstack it stops being able to reach some of them reliably. i think it got narrowed down to the default network size in nova being a 256 so when tempest tried to make more than that many instances... | 01:30 |
| anteaya | oh and what clark said | 01:30 |
| anteaya | great, I'm glad you know more fungi | 01:31 |
| mtreinish | mikal: yeah look at the ssh timeout bugs we've been trying to get to the bottom of for the last few days | 01:31 |
| *** arnaud has quit IRC | 01:31 | |
| anteaya | yes at one point, sdague mentioned he thought nova-network was leaking | 01:31 |
| anteaya | or might be | 01:31 |
| mtreinish | it feels like a floating ip issue but the n-net logs don't have enough details about what happening so we can't figure out where it went wrong | 01:31 |
| jesusaurus | can i get some reviews on https://review.openstack.org/#/c/98561/ and https://review.openstack.org/#/c/98549/ | 01:32 |
| *** Ryan_Lane1 has quit IRC | 01:33 | |
| SlickNik | fungi / clarkb: https://review.openstack.org/98933 => small fix for a mistake I made while setting up the trove-image-build job. Would appreciate a look when you guys get a chance. Thanks! | 01:33 |
| *** nosnos has joined #openstack-infra | 01:33 | |
| jerryz | clarkb: also after jenkins restart, the first build event will be lost because the connection is not there yet, after this event, the connection is established and works from then on | 01:35 |
| *** godp1301 has quit IRC | 01:35 | |
| mikal | Ok, so if I went through the driver interface and added info logs to each of the nova-network events, that would help? | 01:35 |
| *** nati_uen_ has quit IRC | 01:36 | |
| ianw | jesusaurus: maybe it's obvious for the people you really want to review this, but the change doesn't describe *why* you want to do that... | 01:36 |
| jesusaurus | ianw: which change are you referring to? | 01:37 |
| *** yamahata has joined #openstack-infra | 01:37 | |
| ianw | jesusaurus: https://review.openstack.org/#/c/98561/ | 01:37 |
| *** jhesketh has joined #openstack-infra | 01:37 | |
| *** rickerc has quit IRC | 01:38 | |
| mtreinish | mikal: it'd probably be a pretty good start. Definitely better than what's there now. | 01:38 |
| *** rkukura has quit IRC | 01:38 | |
| *** prad has quit IRC | 01:38 | |
| mikal | mtreinish: yeah, it might indicate where we need more specific stuff. | 01:39 |
| mikal | mtreinish: ok, I will take a look at that in a minute, it shouldn't be very painful to do | 01:39 |
| clarkb | mordred: it looks like we can't easily do the db and user setup without having the server being declared in the same puppet process | 01:39 |
| clarkb | mordred: we could rip out the entire DB stack into slave_db and not just the users and db stuff | 01:40 |
| clarkb | mordred: so I think I am leaning that direction. we *should* be able to tell puppet to only listen on socket files and not do tcp with the DBs then have a second post install action update that for final image build | 01:41 |
| *** penguinRaider has quit IRC | 01:41 | |
| harlowja | so a quick question for someone with more experience with testr than me, if i want to say see all the log output that a single cinder tests creates (assume the test does not fail) using testr is there anyway to have testr (or cinder) absorb all the log files and hide them away (somewhere?) | 01:42 |
| clarkb | harlowja: yes they should do that today in the subunit log file | 01:43 |
| harlowja | seems like it squirrels away the logs (or cinder just sends them all the /dev/null), both could be possible | 01:43 |
| clarkb | harlowja: look in .testrepository/$NUMBEROFTESTRUN | 01:43 |
| harlowja | kk, ya, so i get a brief overview there in those files | 01:43 |
| clarkb | sdague: fungi jhesketh ok 4 new logstash workers are up | 01:43 |
| mtreinish | clarkb: does cinder have the logging fixture add the logs to the subunit stream? | 01:43 |
| jhesketh | nice stuff clarkb | 01:43 |
| clarkb | we have 59 worker processes (short one but I need to do dinenr and stuff so will debug tomorrow) | 01:43 |
| harlowja | so there is the following, https://github.com/openstack/cinder/blob/master/cinder/test.py#L124 | 01:44 |
| harlowja | but fake logger seems to just absorb them all | 01:44 |
| clarkb | fungi: jhesketh sdague I think we may also want to restart the logstash gearman client at some point because the geard there seems "slow" I think it may have leaked | 01:44 |
| clarkb | mtreinish: it should | 01:44 |
| clarkb | harlowja: yes it should write them to .testrepository/$NUMBEROFTESTRUN | 01:44 |
| clarkb | which is the subunit log | 01:44 |
| mtreinish | clarkb, harlowja: it doesn't look like it does | 01:45 |
| harlowja | ya, its very skimpy on the logs in those files | 01:45 |
| harlowja | let me see if i can find what nova is doing, and see if that works for this case | 01:46 |
| mtreinish | harlowja: you need something like: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/test.py#n294 | 01:46 |
| mtreinish | or you could start using oslotest which has that in the base test class | 01:46 |
| harlowja | mtreinish ya, nova is doing something different than that, https://github.com/openstack/nova/blob/master/nova/test.py#L254 | 01:46 |
| harlowja | let me try one of these, see what can showup | 01:47 |
| harlowja | thx | 01:47 |
| mtreinish | harlowja: that's because of game nova's playing to catching sytax errors in the log messages | 01:47 |
| *** weshay has quit IRC | 01:47 | |
| harlowja | those nova people and there games :-P | 01:48 |
| * harlowja not sure if i'd buy that game at toys'r'us | 01:49 | |
| mtreinish | harlowja: it's to ensure that all the log messages at any level get compiled in the unit tests, but only those at info or above get added to the subunit stream by the fixture | 01:50 |
| harlowja | interesting | 01:51 |
| clarkb | mordred: ok I am trying it that way. I am moving all stuff into slave_db.pp then normal puppet runs run that code and in_chroot doesn't. This allows us to explicitly run it from post-install.d | 01:53 |
| *** rickerc has joined #openstack-infra | 01:53 | |
| *** rkukura has joined #openstack-infra | 01:53 | |
| clarkb | mordred: we'll see if this works. I am also hacking skip-networking int mysql conf.d into things before puppet runs | 01:53 |
| clarkb | and will have to do something similar to postgres | 01:53 |
| *** gokrokve has joined #openstack-infra | 01:56 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Fix nodepool instance hostnames https://review.openstack.org/98561 | 01:56 |
| harlowja | mtreinish got it to work, thx. much nicer | 01:57 |
| * harlowja wonders how the cinder folks are doing without this | 01:57 | |
| clarkb | harlowja: you should propose that change to cinder | 01:57 |
| harlowja | def | 01:57 |
| *** sweston has quit IRC | 02:01 | |
| *** timrc-afk is now known as timrc | 02:03 | |
| *** rickerc has quit IRC | 02:07 | |
| *** amcrn has quit IRC | 02:08 | |
| *** rkukura has quit IRC | 02:08 | |
| *** sweston has joined #openstack-infra | 02:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixed KEY_DIR for the trove-buildimage-mysql job https://review.openstack.org/98933 | 02:09 |
| *** rkukura has joined #openstack-infra | 02:10 | |
| *** sweston has quit IRC | 02:13 | |
| *** rickerc has joined #openstack-infra | 02:13 | |
| *** zz_gondoi is now known as gondoi | 02:14 | |
| *** gokrokve has quit IRC | 02:18 | |
| *** trinaths has joined #openstack-infra | 02:19 | |
| *** reed has joined #openstack-infra | 02:19 | |
| *** nati_ueno has joined #openstack-infra | 02:23 | |
| harlowja | mtreinish https://review.openstack.org/#/c/98937/ (if u want to comment) | 02:23 |
| *** ramashri has joined #openstack-infra | 02:24 | |
| *** penguinRaider has joined #openstack-infra | 02:24 | |
| *** [1]trinaths has joined #openstack-infra | 02:25 | |
| *** reed has quit IRC | 02:25 | |
| *** trinaths has quit IRC | 02:28 | |
| *** [1]trinaths is now known as trinaths | 02:28 | |
| *** doug-fish has left #openstack-infra | 02:30 | |
| *** arnaud has joined #openstack-infra | 02:33 | |
| *** godp1301 has joined #openstack-infra | 02:34 | |
| *** trinaths has quit IRC | 02:36 | |
| *** godp1301 has quit IRC | 02:37 | |
| lifeless | are we allowed to +A things now? | 02:38 |
| lifeless | sdague: btw you might like to put #tripleo in your channels | 02:38 |
| *** penguinRaider has quit IRC | 02:41 | |
| *** ramashri has quit IRC | 02:45 | |
| *** primeministerp has quit IRC | 02:46 | |
| *** rwsu has quit IRC | 02:52 | |
| mordred_phone | clarkb: nod | 02:59 |
| *** signed8bit has quit IRC | 03:00 | |
| *** praneshp has quit IRC | 03:03 | |
| clarkb | it doesnt work | 03:03 |
| clarkb | even in post install no services :( | 03:03 |
| *** crc32 has quit IRC | 03:05 | |
| *** nati_ueno has quit IRC | 03:10 | |
| *** david-lyle has joined #openstack-infra | 03:11 | |
| *** lbragstad has quit IRC | 03:11 | |
| *** prad has joined #openstack-infra | 03:12 | |
| *** baohua has joined #openstack-infra | 03:17 | |
| *** zhiyan is now known as zhiyan_ | 03:20 | |
| *** zhiyan_ is now known as zhiyan | 03:26 | |
| boris-42 | clarkb mordred hm I am facing strange issue with global update requirements | 03:27 |
| boris-42 | clarkb mordred https://review.openstack.org/#/c/96833/4/test-requirements.txt it says that sphinx is not ok here | 03:28 |
| boris-42 | clarkb mordred http://logs.openstack.org/33/96833/4/check/gate-rally-requirements/8e66a49/console.html | 03:28 |
| boris-42 | and in global requirements https://github.com/openstack/requirements/blob/master/global-requirements.txt#L169 | 03:28 |
| fungi | boris-42: https://review.openstack.org/97893 will fix it, but i think we've just been short on other core reviewers so i'll go ahead and approve it | 03:35 |
| boris-42 | fungi ah ok, it's not rally issue in my project huh | 03:35 |
| fungi | boris-42: no, the job itself was enforcing in the wrong place | 03:36 |
| *** talluri has joined #openstack-infra | 03:37 | |
| *** arnaud has quit IRC | 03:42 | |
| *** nosnos has quit IRC | 03:43 | |
| boris-42 | fungi thanks | 03:43 |
| *** CaptTofu_ has joined #openstack-infra | 03:50 | |
| *** pcrews_ has quit IRC | 03:52 | |
| *** Longgeek has joined #openstack-infra | 03:52 | |
| *** dims_ has quit IRC | 03:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Checks if requirements overlap between head and branch reqs" https://review.openstack.org/97893 | 03:53 |
| *** basha has joined #openstack-infra | 03:54 | |
| clarkb | wait now all those broken sphinx reqs can get through | 03:55 |
| clarkb | I didnt realize that was the "fix" | 03:55 |
| boris-42 | clarkb are they broken? | 03:55 |
| clarkb | yes | 03:55 |
| boris-42 | clarkb oh | 03:55 |
| clarkb | they dont overlap | 03:56 |
| *** basha has quit IRC | 03:57 | |
| *** Longgeek has quit IRC | 03:57 | |
| fungi | clarkb: the breakage was felt on the front end when the reqs change went in, which was where that was supposed to have gotten enforced | 03:57 |
| clarkb | sure but at least it prevented broken | 03:58 |
| fungi | that change also needs to be reverted | 03:58 |
| * fungi checks to see if anyone also did that part | 03:59 | |
| *** jamespage has quit IRC | 04:00 | |
| *** gnuoy has quit IRC | 04:00 | |
| *** jamespage has joined #openstack-infra | 04:00 | |
| *** jamespage has joined #openstack-infra | 04:00 | |
| *** gnuoy has joined #openstack-infra | 04:00 | |
| *** sweston has joined #openstack-infra | 04:00 | |
| adam_g | anyone for a quickie? seems ironic's devstack job no longer likes to run on HP. https://review.openstack.org/#/c/98886/ | 04:01 |
| fungi | i think i'll implement dhellmann's suggestion from https://review.openstack.org/88631 which apparently got overlooked when the change was approved | 04:02 |
| *** MarkAtwood has joined #openstack-infra | 04:02 | |
| *** MarkAtwood has quit IRC | 04:02 | |
| *** basha has joined #openstack-infra | 04:07 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack/requirements: Fix sphinx requirement to add overlap https://review.openstack.org/98947 | 04:07 |
| fungi | clarkb: that ^ ? | 04:08 |
| *** basha has quit IRC | 04:08 | |
| fungi | i couldn't find any indication of an existing change along those lines. though also possible that it's closing the barn door after the horse has left | 04:09 |
| notmyname | fungi: what was the usb microscope you used at the summit keysigning? is it good enough eg to look at stuff in pond water or at cheek cells? | 04:09 |
| fungi | notmyname: that one i brought because it was low enough magnification i could use it for macro range. i have a much higher magnification one which will. hold on and i'll get you the model number | 04:10 |
| notmyname | fungi: cool, thanks! | 04:10 |
| clarkb | fungi: +2 thanks | 04:10 |
| fungi | notmyname: the celestron model i brought with me does 150x and is good enough for examining solder joints for surface-mount electronics. the better one i have goes to 220x magnification, which i got at http://www.adafruit.com/products/636 (a steal at us$80) | 04:17 |
| *** om has joined #openstack-infra | 04:17 | |
| notmyname | fungi: thanks :-) | 04:18 |
| fungi | notmyname: though keep in mind that you need ~500x to see, say, human blood cells | 04:18 |
| notmyname | fungi: got it. I'm looking for something for my kids (who am I kidding? I think it will be really cool) | 04:19 |
| fungi | at that price it's a great starter. i can stream off it at 640x480 using the uvc/v4l drivers in debian no problem | 04:20 |
| fungi | i haven't tried the snapshot function though. might require other drivers | 04:21 |
| fungi | though uvccapture may just work on it | 04:21 |
| *** lcheng has joined #openstack-infra | 04:21 | |
| *** yfried has quit IRC | 04:21 | |
| *** nosnos has joined #openstack-infra | 04:24 | |
| *** gokrokve has joined #openstack-infra | 04:25 | |
| *** om has quit IRC | 04:25 | |
| fungi | i'm tempted to upgrade to http://www.adafruit.com/products/1061 (doesn't support the higher-res stills, but goes to 300x magnification) | 04:29 |
| *** penguinRaider has joined #openstack-infra | 04:31 | |
| *** david-lyle has quit IRC | 04:33 | |
| *** CaptTofu_ has quit IRC | 04:34 | |
| *** CaptTofu_ has joined #openstack-infra | 04:34 | |
| *** marcoemorais has joined #openstack-infra | 04:34 | |
| StevenK | Damn it, I left Zuul Status open on my laptop and it OOM-killed Firefox | 04:36 |
| *** marcoemorais1 has joined #openstack-infra | 04:36 | |
| fungi | StevenK: yeah, does the same to me if i leave it overnight | 04:37 |
| *** gokrokve has quit IRC | 04:37 | |
| fungi | i blame javascript | 04:37 |
| fungi | for all the world's troubles | 04:37 |
| clarkb | huh. zuul is pretty well behaved for me | 04:38 |
| *** lbragstad has joined #openstack-infra | 04:38 | |
| clarkb | gerrit gives it a hard time though | 04:38 |
| StevenK | I think Firefox allocates new RAM everytime it gets more JSON, and doesn't free anything ever | 04:38 |
| *** marcoemorais has quit IRC | 04:39 | |
| *** om has joined #openstack-infra | 04:39 | |
| *** CaptTofu_ has quit IRC | 04:39 | |
| lifeless | StevenK: buy more ram | 04:41 |
| fungi | ...weekly | 04:41 |
| fungi | the firefox, it hungers | 04:41 |
| fungi | it must be fed with ram sacrifices | 04:42 |
| StevenK | lifeless: And install it into the laptop how? Gaffa tape it to the side? | 04:42 |
| StevenK | It has 16GiB, it physically doesn't support more. | 04:42 |
| StevenK | I left Zuul open in Firefox while I went into the office on Friday, and got home ~ 8 hours later too: | 04:43 |
| StevenK | 3610 steven 23 3 19.0g 17g 28m R 69.5 73.5 163:42.52 firefox | 04:43 |
| *** trinaths has joined #openstack-infra | 04:43 | |
| StevenK | (On my desktop, which has 24GiB) | 04:43 |
| *** rlandy has joined #openstack-infra | 04:45 | |
| *** basha has joined #openstack-infra | 04:46 | |
| *** dims_ has joined #openstack-infra | 04:46 | |
| *** dims_ has quit IRC | 04:50 | |
| *** ildikov has quit IRC | 04:51 | |
| *** ArxCruz_ has quit IRC | 04:52 | |
| lifeless | StevenK: beowulf | 04:53 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Ignore newline in bp/bug search in commit message https://review.openstack.org/98324 | 04:53 |
| StevenK | lifeless: If Firefox needs a beowulf to not OOM, I have missed a few memos | 04:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Add devstack-f20 to hpcloud https://review.openstack.org/98024 | 04:55 |
| *** om has quit IRC | 04:55 | |
| lifeless | wouldn't help anyhow, beowulf is process based IIRC | 04:57 |
| *** gokrokve has joined #openstack-infra | 04:58 | |
| fungi | yeah, beowulf clusters usually used a distributed scheduler like mosix or message passing between remote processes with pvm or mpich | 04:58 |
| *** gokrokve has quit IRC | 04:58 | |
| * fungi has no idea why he remembers this after ~15 years | 04:59 | |
| StevenK | Haha | 04:59 |
| *** harlowja is now known as harlowja_away | 04:59 | |
| * fungi still has an "extreme linux" install cd somewhere in his lab... held onto it for reasons of nostalgia | 05:00 | |
| *** wenlock has joined #openstack-infra | 05:00 | |
| clarkb | ugh mpi | 05:01 |
| clarkb | I remember teaching grad students how to compile their mpi | 05:01 |
| fungi | mpi was a lot more tightly-coupled than pvm, for sure | 05:01 |
| *** arnaud has joined #openstack-infra | 05:02 | |
| *** om has joined #openstack-infra | 05:10 | |
| *** lbragstad has quit IRC | 05:13 | |
| *** ildikov has joined #openstack-infra | 05:15 | |
| *** greghaynes has joined #openstack-infra | 05:15 | |
| *** lbragstad has joined #openstack-infra | 05:15 | |
| *** wenlock has quit IRC | 05:17 | |
| *** doude has quit IRC | 05:22 | |
| *** yfried has joined #openstack-infra | 05:24 | |
| *** rlandy_ has joined #openstack-infra | 05:24 | |
| *** rlandy has quit IRC | 05:26 | |
| *** devkulkarni has joined #openstack-infra | 05:28 | |
| *** gyee has quit IRC | 05:38 | |
| *** arnaud__ has joined #openstack-infra | 05:46 | |
| *** dims_ has joined #openstack-infra | 05:47 | |
| *** TheSheep has joined #openstack-infra | 05:47 | |
| *** TheSheep has quit IRC | 05:48 | |
| *** Longgeek has joined #openstack-infra | 05:49 | |
| *** Longgeek has quit IRC | 05:49 | |
| *** rdopiera has joined #openstack-infra | 05:50 | |
| *** rdopiera has quit IRC | 05:50 | |
| *** Longgeek has joined #openstack-infra | 05:50 | |
| *** rdopieralski has joined #openstack-infra | 05:51 | |
| *** rdopieralski has quit IRC | 05:51 | |
| *** Longgeek has quit IRC | 05:51 | |
| *** rdopieralski has joined #openstack-infra | 05:51 | |
| *** Longgeek has joined #openstack-infra | 05:51 | |
| *** dims_ has quit IRC | 05:51 | |
| *** gondoi is now known as zz_gondoi | 05:52 | |
| *** adrian_otto has joined #openstack-infra | 05:57 | |
| *** wenlock has joined #openstack-infra | 06:02 | |
| *** Longgeek has quit IRC | 06:04 | |
| *** Longgeek has joined #openstack-infra | 06:04 | |
| *** devkulkarni has quit IRC | 06:04 | |
| *** Longgeek has quit IRC | 06:04 | |
| *** devkulkarni has joined #openstack-infra | 06:05 | |
| *** Longgeek has joined #openstack-infra | 06:05 | |
| *** Longgeek has quit IRC | 06:06 | |
| *** Longgeek_ has joined #openstack-infra | 06:06 | |
| *** devkulkarni has quit IRC | 06:07 | |
| *** arnaud__ has quit IRC | 06:07 | |
| *** arnaud has quit IRC | 06:08 | |
| *** _nadya_ has joined #openstack-infra | 06:10 | |
| *** CaptTofu_ has joined #openstack-infra | 06:14 | |
| *** cody-somerville has joined #openstack-infra | 06:17 | |
| *** cody-somerville has quit IRC | 06:17 | |
| *** cody-somerville has joined #openstack-infra | 06:17 | |
| *** dims_ has joined #openstack-infra | 06:17 | |
| *** arnaud has joined #openstack-infra | 06:17 | |
| *** andreykurilin_ has joined #openstack-infra | 06:18 | |
| *** arnaud__ has joined #openstack-infra | 06:19 | |
| *** arnaud__ has quit IRC | 06:19 | |
| *** CaptTofu_ has quit IRC | 06:19 | |
| *** mkoderer has joined #openstack-infra | 06:21 | |
| *** dims_ has quit IRC | 06:22 | |
| *** arnaud__ has joined #openstack-infra | 06:22 | |
| *** arnaud has quit IRC | 06:22 | |
| *** arnaud__ has quit IRC | 06:22 | |
| *** arnaud has joined #openstack-infra | 06:23 | |
| *** arnaud has quit IRC | 06:23 | |
| *** e0ne has joined #openstack-infra | 06:27 | |
| *** rlandy_ has quit IRC | 06:30 | |
| rdopieralski | Hi, I need some help with https://review.openstack.org/#/c/95716/ -- specifically, how would I go about creating a new team for it? | 06:32 |
| *** Ryan_Lane has joined #openstack-infra | 06:35 | |
| *** rlandy_ has joined #openstack-infra | 06:37 | |
| *** Ryan_Lane1 has joined #openstack-infra | 06:41 | |
| *** Ryan_Lane has quit IRC | 06:41 | |
| *** wenlock has quit IRC | 06:44 | |
| *** YorikSar has quit IRC | 06:46 | |
| *** lbragstad has quit IRC | 06:49 | |
| *** lbragstad has joined #openstack-infra | 06:50 | |
| *** afazekas is now known as __afazekas | 06:51 | |
| *** rcarrill` has quit IRC | 06:52 | |
| *** e0ne has quit IRC | 06:54 | |
| *** adrian_otto has quit IRC | 06:55 | |
| *** cody-somerville has quit IRC | 06:55 | |
| *** e0ne has joined #openstack-infra | 06:55 | |
| *** rcarrillocruz has joined #openstack-infra | 06:57 | |
| *** e0ne has quit IRC | 06:57 | |
| *** salv-orlando has quit IRC | 06:58 | |
| *** jcoufal has joined #openstack-infra | 06:59 | |
| *** achuprin has quit IRC | 06:59 | |
| *** adrian_otto has joined #openstack-infra | 06:59 | |
| *** nosnos has quit IRC | 06:59 | |
| *** matsuhashi has quit IRC | 07:01 | |
| *** matsuhas_ has joined #openstack-infra | 07:01 | |
| *** basha has quit IRC | 07:02 | |
| *** basha has joined #openstack-infra | 07:03 | |
| *** nosnos has joined #openstack-infra | 07:05 | |
| *** jpich has joined #openstack-infra | 07:06 | |
| *** flaper87|afk is now known as flaper87 | 07:06 | |
| *** pblaho has joined #openstack-infra | 07:06 | |
| *** cody-somerville has joined #openstack-infra | 07:09 | |
| *** achuprin has joined #openstack-infra | 07:11 | |
| mattoliverau | Night all | 07:12 |
| *** basha has quit IRC | 07:16 | |
| *** dims_ has joined #openstack-infra | 07:17 | |
| openstackgerrit | A change was merged to openstack-infra/git-review: Disable proxies for tests that clone over http https://review.openstack.org/96893 | 07:18 |
| SergeyLukjanov | rdopieralski, all groups will be created automatically and then you'll need to request init group members additions | 07:21 |
| *** dims_ has quit IRC | 07:22 | |
| *** rlandy__ has joined #openstack-infra | 07:22 | |
| *** rlandy_ has quit IRC | 07:22 | |
| *** bogdando has quit IRC | 07:23 | |
| openstackgerrit | A change was merged to openstack-infra/git-review: Keep track of gerrit.war and golden_site versions https://review.openstack.org/96818 | 07:23 |
| *** tkelsey has joined #openstack-infra | 07:24 | |
| *** bogdando has joined #openstack-infra | 07:24 | |
| *** Hal_ has joined #openstack-infra | 07:26 | |
| *** bauzas has joined #openstack-infra | 07:26 | |
| rdopieralski | SergeyLukjanov: how would I do that? | 07:27 |
| *** _nadya_ has quit IRC | 07:28 | |
| *** andreaf has quit IRC | 07:31 | |
| *** dizquierdo has joined #openstack-infra | 07:32 | |
| *** mwagner_lap has quit IRC | 07:34 | |
| *** rlandy__ has quit IRC | 07:35 | |
| *** dizquierdo has quit IRC | 07:36 | |
| *** mwagner_lap has joined #openstack-infra | 07:36 | |
| *** ominakov has joined #openstack-infra | 07:40 | |
| *** jlibosva has joined #openstack-infra | 07:42 | |
| SergeyLukjanov | rdopieralski, you can ask here after the repo adding change will be merged | 07:44 |
| SergeyLukjanov | and someone from infra root team will add init team members | 07:44 |
| SergeyLukjanov | need to go, brb in 1h | 07:44 |
| rdopieralski | SergeyLukjanov: thank you | 07:45 |
| *** Ryan_Lane1 has quit IRC | 07:46 | |
| *** thomasbiege has joined #openstack-infra | 07:47 | |
| *** rlandy__ has joined #openstack-infra | 07:48 | |
| *** e0ne has joined #openstack-infra | 07:49 | |
| *** dizquierdo has joined #openstack-infra | 07:49 | |
| *** sweston has quit IRC | 07:54 | |
| *** afazekas has joined #openstack-infra | 07:55 | |
| *** vponomaryov has joined #openstack-infra | 07:55 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 07:56 |
| *** hashar has joined #openstack-infra | 07:56 | |
| *** andreaf has joined #openstack-infra | 07:58 | |
| *** ihrachyshka has joined #openstack-infra | 07:59 | |
| *** jistr has joined #openstack-infra | 08:00 | |
| *** CaptTofu_ has joined #openstack-infra | 08:03 | |
| *** andreykurilin_ has quit IRC | 08:03 | |
| *** e0ne has quit IRC | 08:03 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Set env vars explicitly for manila jobs https://review.openstack.org/97747 | 08:07 |
| *** bauzas has quit IRC | 08:07 | |
| *** CaptTofu_ has quit IRC | 08:08 | |
| *** basha has joined #openstack-infra | 08:10 | |
| *** e0ne_ has joined #openstack-infra | 08:12 | |
| *** IvanBerezovskiy has joined #openstack-infra | 08:12 | |
| openstackgerrit | A change was merged to openstack-infra/git-review: Restrict tests SSH auth to only the provided key https://review.openstack.org/96894 | 08:12 |
| *** srenatus has joined #openstack-infra | 08:12 | |
| srenatus | i can haz https://review.openstack.org/#/c/98084/ without creating another lame cat meme pic? :} | 08:13 |
| *** roeyc has joined #openstack-infra | 08:15 | |
| *** sweston has joined #openstack-infra | 08:15 | |
| *** derekh_ has joined #openstack-infra | 08:17 | |
| *** dims_ has joined #openstack-infra | 08:18 | |
| *** dims_ has quit IRC | 08:22 | |
| *** marcoemorais1 has quit IRC | 08:23 | |
| *** habib__ has joined #openstack-infra | 08:23 | |
| *** _nadya_ has joined #openstack-infra | 08:25 | |
| *** sweston has quit IRC | 08:35 | |
| *** doude has joined #openstack-infra | 08:36 | |
| *** matsuhas_ has quit IRC | 08:38 | |
| *** nosnos has quit IRC | 08:40 | |
| *** pelix has joined #openstack-infra | 08:41 | |
| *** om has quit IRC | 08:41 | |
| *** lbragstad has quit IRC | 08:43 | |
| *** nosnos has joined #openstack-infra | 08:43 | |
| *** lbragstad has joined #openstack-infra | 08:43 | |
| *** nosnos has quit IRC | 08:44 | |
| *** adrian_otto has quit IRC | 08:44 | |
| *** matsuhashi has joined #openstack-infra | 08:44 | |
| *** mmaglana has quit IRC | 08:54 | |
| *** om has joined #openstack-infra | 08:54 | |
| *** jcoufal has quit IRC | 08:55 | |
| *** adrian_otto has joined #openstack-infra | 08:55 | |
| *** jcoufal has joined #openstack-infra | 08:56 | |
| *** alkari has quit IRC | 08:58 | |
| *** thomasbiege1 has joined #openstack-infra | 09:00 | |
| *** habib___ has joined #openstack-infra | 09:01 | |
| *** sweston has joined #openstack-infra | 09:02 | |
| *** adrian_otto has quit IRC | 09:03 | |
| *** nosnos has joined #openstack-infra | 09:03 | |
| *** habib__ has quit IRC | 09:04 | |
| *** thomasbiege has quit IRC | 09:04 | |
| *** jgallard has joined #openstack-infra | 09:05 | |
| *** habib___ has quit IRC | 09:06 | |
| *** habib___ has joined #openstack-infra | 09:07 | |
| *** habib___ has quit IRC | 09:08 | |
| *** habib___ has joined #openstack-infra | 09:10 | |
| *** ihrachyshka has quit IRC | 09:10 | |
| *** isviridov_ is now known as isviridov | 09:12 | |
| *** ociuhandu has quit IRC | 09:15 | |
| *** zhiyan is now known as zhiyan_ | 09:18 | |
| *** dims_ has joined #openstack-infra | 09:19 | |
| *** sweston has quit IRC | 09:24 | |
| *** habib___ has quit IRC | 09:24 | |
| *** dims_ has quit IRC | 09:25 | |
| *** habib___ has joined #openstack-infra | 09:30 | |
| *** baohua has quit IRC | 09:34 | |
| *** mugsie has quit IRC | 09:40 | |
| *** mugsie has joined #openstack-infra | 09:42 | |
| *** weshay has joined #openstack-infra | 09:43 | |
| *** rlandy_ has joined #openstack-infra | 09:43 | |
| srenatus | alright. I see no other options. http://i.imgur.com/aGkA8DJ.jpg >> https://review.openstack.org/#/c/98084/ | 09:45 |
| *** maxbit has quit IRC | 09:46 | |
| *** rlandy__ has quit IRC | 09:46 | |
| *** mugsie has quit IRC | 09:46 | |
| *** CaptTofu_ has joined #openstack-infra | 09:51 | |
| *** ihrachyshka has joined #openstack-infra | 09:53 | |
| *** roeyc has quit IRC | 09:54 | |
| *** CaptTofu_ has quit IRC | 09:56 | |
| *** salv-orlando has joined #openstack-infra | 10:00 | |
| *** Alexei_987 has quit IRC | 10:02 | |
| *** ociuhandu has joined #openstack-infra | 10:06 | |
| *** jgallard has quit IRC | 10:07 | |
| *** deepakcs has joined #openstack-infra | 10:13 | |
| deepakcs | Hi, has anyone here faced issues with devstack hosted on F20 ? | 10:13 |
| deepakcs | I heard that f20 was recently added to openstack-infra, hence the Q | 10:13 |
| openstackgerrit | A change was merged to openstack/requirements: Updated zake now that zake 0.0.20 is released https://review.openstack.org/93156 | 10:14 |
| *** om has quit IRC | 10:14 | |
| deepakcs | On my local F20 VM + devstack, Nova instance always gets stuck in 'spawning' state | 10:14 |
| openstackgerrit | Roger Luethi proposed a change to openstack-infra/git-review: Fix a typo in HACKING.rst https://review.openstack.org/98998 | 10:15 |
| deepakcs | so was wondering if anyone here has seen such issue on F20 | 10:15 |
| *** rlandy__ has joined #openstack-infra | 10:20 | |
| *** dims_ has joined #openstack-infra | 10:21 | |
| *** rlandy_ has quit IRC | 10:22 | |
| sdague | deepakcs: it's still not running regularly | 10:26 |
| *** e0ne_ has quit IRC | 10:26 | |
| *** e0ne has joined #openstack-infra | 10:27 | |
| *** dims_ has quit IRC | 10:27 | |
| deepakcs | sdague, ok, but when it runs, u dont see any issues ? | 10:28 |
| *** habib___ has quit IRC | 10:29 | |
| deepakcs | sdague, I assume u folks run devstack on F20 as part of CI, rite ? | 10:29 |
| sdague | deepakcs: we do not | 10:29 |
| deepakcs | sdague, I am facing issues with F20+devstack, hence :) | 10:29 |
| deepakcs | sdague, sorry, 'we do not' response was for "not seeign any issues" ? | 10:30 |
| sdague | we do not run it in ci | 10:30 |
| sdague | ianw has been working on getting it in, but it's not | 10:30 |
| *** hashar has quit IRC | 10:32 | |
| deepakcs | sdague, ok, does that mean not many in community would be running F20+devstack combo ? | 10:32 |
| deepakcs | sdague, i am curious to know whether its something that is related to my setup or there is a real issue with F20 and devstakc | 10:32 |
| deepakcs | ianw, ^^ | 10:33 |
| *** pblaho has quit IRC | 10:33 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Use except x as y instead of except x, y https://review.openstack.org/96615 | 10:34 |
| openstackgerrit | A change was merged to openstack-infra/zuul: replace dict.iteritems() with six.iteritems(dict) https://review.openstack.org/98440 | 10:38 |
| *** om has joined #openstack-infra | 10:39 | |
| *** roeyc has joined #openstack-infra | 10:45 | |
| roeyc | Hi, Devstack setup is failing as part of our 3rd party CI, the problem is with installation of the pbr library. | 10:47 |
| roeyc | hoped to get some help here.. | 10:47 |
| roeyc | I added more details here: http://paste.openstack.org/show/83508/ | 10:47 |
| *** rcarrillocruz has quit IRC | 10:48 | |
| *** rlandy_ has joined #openstack-infra | 10:48 | |
| roeyc | can anyone assist please? | 10:48 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Use import from six.moves to import the ConfigParser module https://review.openstack.org/98597 | 10:48 |
| *** rlandy__ has quit IRC | 10:49 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Use import from six.moves to import the queue module https://review.openstack.org/98595 | 10:50 |
| *** lcostantino has joined #openstack-infra | 10:50 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Use import from six.moves to import the queue module https://review.openstack.org/98596 | 10:51 |
| *** rcarrillocruz has joined #openstack-infra | 10:52 | |
| *** Ryan_Lane has joined #openstack-infra | 10:54 | |
| *** mmaglana has joined #openstack-infra | 10:55 | |
| *** rlandy_ has quit IRC | 10:57 | |
| *** rlandy has joined #openstack-infra | 10:57 | |
| *** ekarlso has quit IRC | 10:57 | |
| *** ekarlso has joined #openstack-infra | 10:58 | |
| *** mmaglana has quit IRC | 11:00 | |
| *** om has quit IRC | 11:02 | |
| *** Ryan_Lane has quit IRC | 11:03 | |
| *** simonmcc has quit IRC | 11:04 | |
| *** bogdando has quit IRC | 11:04 | |
| *** bogdando has joined #openstack-infra | 11:04 | |
| *** simonmcc has joined #openstack-infra | 11:04 | |
| *** cyeoh has quit IRC | 11:06 | |
| *** talluri has quit IRC | 11:09 | |
| *** talluri has joined #openstack-infra | 11:09 | |
| *** cyeoh has joined #openstack-infra | 11:10 | |
| *** yaguang has quit IRC | 11:10 | |
| *** talluri has quit IRC | 11:11 | |
| *** talluri has joined #openstack-infra | 11:11 | |
| *** talluri has quit IRC | 11:16 | |
| *** om has joined #openstack-infra | 11:20 | |
| *** dims_ has joined #openstack-infra | 11:23 | |
| openstackgerrit | Alexandre Levine proposed a change to openstack-infra/config: Initial commit for the AWS EC2 and VPC API standalone service project https://review.openstack.org/97795 | 11:24 |
| *** _nadya_ has quit IRC | 11:26 | |
| *** mugsie has joined #openstack-infra | 11:27 | |
| *** salv-orlando has quit IRC | 11:27 | |
| *** om has quit IRC | 11:29 | |
| *** alexpilotti has joined #openstack-infra | 11:29 | |
| *** salv-orlando has joined #openstack-infra | 11:32 | |
| *** yamahata has quit IRC | 11:32 | |
| *** trinaths has quit IRC | 11:35 | |
| *** miqui has quit IRC | 11:38 | |
| openstackgerrit | Ph. Marek proposed a change to openstack/requirements: Add dbus library, needed for the DRBD cinder driver integration. https://review.openstack.org/99013 | 11:39 |
| *** CaptTofu_ has joined #openstack-infra | 11:39 | |
| *** yfried has quit IRC | 11:40 | |
| *** CaptTofu_ has quit IRC | 11:44 | |
| *** om has joined #openstack-infra | 11:44 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: don't update apt repos https://review.openstack.org/99016 | 11:45 |
| *** CaptTofu_ has joined #openstack-infra | 11:47 | |
| *** hashar has joined #openstack-infra | 11:49 | |
| *** ArxCruz has joined #openstack-infra | 11:52 | |
| *** rlandy has quit IRC | 11:56 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Add query for 1328500 https://review.openstack.org/99020 | 11:57 |
| *** _nadya_ has joined #openstack-infra | 11:57 | |
| *** adalbas has joined #openstack-infra | 12:04 | |
| openstackgerrit | Anne Gentle proposed a change to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 12:04 |
| *** radez_g0n3 is now known as radez | 12:05 | |
| *** YorikSar has joined #openstack-infra | 12:06 | |
| *** afazekas has quit IRC | 12:09 | |
| *** aysyd has joined #openstack-infra | 12:09 | |
| *** yfried has joined #openstack-infra | 12:10 | |
| *** mbacchi has joined #openstack-infra | 12:16 | |
| *** afazekas has joined #openstack-infra | 12:24 | |
| *** dkranz has joined #openstack-infra | 12:24 | |
| *** lcostantino has quit IRC | 12:25 | |
| *** Longgeek_ has quit IRC | 12:25 | |
| openstackgerrit | A change was merged to openstack/requirements: Update openstack-doc-tools to at least 0.15 https://review.openstack.org/95879 | 12:25 |
| *** Longgeek has joined #openstack-infra | 12:26 | |
| *** mwagner_lap has quit IRC | 12:26 | |
| *** CaptTofu_ has quit IRC | 12:26 | |
| *** CaptTofu_ has joined #openstack-infra | 12:27 | |
| *** dprince has joined #openstack-infra | 12:28 | |
| *** lcostantino has joined #openstack-infra | 12:28 | |
| *** dhellmann has quit IRC | 12:28 | |
| *** yaguang has joined #openstack-infra | 12:29 | |
| *** yaguang has quit IRC | 12:30 | |
| *** yaguang has joined #openstack-infra | 12:30 | |
| *** oomichi_ has joined #openstack-infra | 12:31 | |
| *** CaptTofu_ has quit IRC | 12:31 | |
| anteaya | hi roeyc I found this: https://bugs.launchpad.net/swift/+bug/1217288 | 12:32 |
| uvirtbot | Launchpad bug 1217288 in swift "Swift installation crashes" [High,Triaged] | 12:32 |
| anteaya | roeyc: which it appears suggests a workaround for the moment | 12:32 |
| *** bknudson has quit IRC | 12:34 | |
| anteaya | roeyc: the suggested workaround is `sudo pip install -r requirements.txt` | 12:35 |
| roeyc | anteaya: thanks, but it doesn't seem to work here. | 12:37 |
| anteaya | kk | 12:37 |
| anteaya | mordred is our pbr expert so unless someone else wants to step in, you might be waiting for him to appear | 12:38 |
| *** deepakcs has quit IRC | 12:39 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adjust ironic tempest regex for new tests https://review.openstack.org/97830 | 12:39 |
| roeyc | alright then, thanks again | 12:39 |
| anteaya | np | 12:41 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 12:42 |
| *** heyongli has joined #openstack-infra | 12:42 | |
| *** palar has joined #openstack-infra | 12:43 | |
| *** heyongli has quit IRC | 12:43 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 12:43 |
| *** heyongli has joined #openstack-infra | 12:43 | |
| *** pdmars has joined #openstack-infra | 12:44 | |
| *** heyongli has quit IRC | 12:44 | |
| *** pdmars has quit IRC | 12:44 | |
| *** pdmars has joined #openstack-infra | 12:45 | |
| *** pblaho has joined #openstack-infra | 12:45 | |
| roeyc | anteaya: problem was with the setuptools package, re-installing it and it looks fine now | 12:45 |
| *** lbragstad has quit IRC | 12:45 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 12:46 |
| *** yhe_ has joined #openstack-infra | 12:46 | |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added docs-on-rtfd for MagnetoDB https://review.openstack.org/99039 | 12:47 |
| *** palar has quit IRC | 12:47 | |
| *** palar has joined #openstack-infra | 12:48 | |
| *** dkranz has quit IRC | 12:48 | |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 12:49 |
| anteaya | roeyc: awesome, thanks for letting me know | 12:49 |
| anteaya | sdague: isn't line 502 missing a %s? https://review.openstack.org/#/c/99002/1/nova/network/manager.py | 12:49 |
| anteaya | and line 567 | 12:50 |
| anteaya | maybe this is a part of python strings I don't understand | 12:50 |
| *** yhe_ has quit IRC | 12:50 | |
| sdague | anteaya: no, becuase our logger knows how to log instance_id separately | 12:51 |
| anteaya | ah | 12:51 |
| anteaya | that was the piece I was missing, thansk | 12:51 |
| anteaya | thanks | 12:51 |
| *** bknudson has joined #openstack-infra | 12:52 | |
| *** HenryG has quit IRC | 12:53 | |
| *** alexpilotti_ has joined #openstack-infra | 12:53 | |
| sdague | there are a bunch of special fields past by name that we pull out - https://github.com/openstack/nova/blob/master/nova/openstack/common/log.py if you want all the details | 12:53 |
| *** afazekas has quit IRC | 12:53 | |
| *** alexpilotti has quit IRC | 12:54 | |
| *** alexpilotti_ is now known as alexpilotti | 12:54 | |
| anteaya | thanks | 12:54 |
| * anteaya looks | 12:54 | |
| sdague | the logic is a little squirely, but a combination of process() and format() will give you the feel | 12:54 |
| *** timrc is now known as timrc-afk | 12:55 | |
| openstackgerrit | Anne Gentle proposed a change to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 12:55 |
| *** rlandy has joined #openstack-infra | 12:56 | |
| *** mmaglana has joined #openstack-infra | 12:56 | |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Make 'puppetmaster_host' configurable' https://review.openstack.org/99041 | 12:57 |
| *** changbl has quit IRC | 12:57 | |
| *** dims_ has quit IRC | 12:59 | |
| *** skolekonov has joined #openstack-infra | 13:00 | |
| sdague | mordred: so this is fun - https://review.openstack.org/#/c/99016/ looks like the local apt caches are only good on 50% of our nodes? | 13:00 |
| *** mmaglana has quit IRC | 13:01 | |
| *** miqui has joined #openstack-infra | 13:05 | |
| *** yamahata has joined #openstack-infra | 13:05 | |
| *** amotoki_ has joined #openstack-infra | 13:06 | |
| *** pblaho` has joined #openstack-infra | 13:06 | |
| *** godp1301 has joined #openstack-infra | 13:06 | |
| *** julim has joined #openstack-infra | 13:06 | |
| *** rlandy has quit IRC | 13:07 | |
| *** godp1301 has left #openstack-infra | 13:07 | |
| *** afazekas has joined #openstack-infra | 13:07 | |
| *** basha has quit IRC | 13:08 | |
| *** pblaho has quit IRC | 13:08 | |
| *** mfer has joined #openstack-infra | 13:16 | |
| *** mriedem has joined #openstack-infra | 13:16 | |
| *** lbragstad has joined #openstack-infra | 13:17 | |
| anteaya | I got distracted by the email list | 13:18 |
| anteaya | folks who make tools that abstract away messiness need to identify the criteria for the decisions the abstraction is making | 13:18 |
| anteaya | found process() yeah what it is doing with context and instance make more sense now | 13:20 |
| *** alexpilotti_ has joined #openstack-infra | 13:21 | |
| *** alexpilotti has quit IRC | 13:22 | |
| *** alexpilotti_ is now known as alexpilotti | 13:22 | |
| pelix | in git-review, is there a reason for the tests to care about having the correct ssh hosts key? | 13:22 |
| pelix | as opposed to using StrictHostKeyChecking=no for the tests? | 13:22 |
| pelix | Seems to be causing a lot of failures in testing at the moment | 13:22 |
| *** rlandy has joined #openstack-infra | 13:22 | |
| *** bcrochet has quit IRC | 13:23 | |
| *** bcrochet has joined #openstack-infra | 13:25 | |
| *** lbragstad1 has joined #openstack-infra | 13:26 | |
| *** trinaths has joined #openstack-infra | 13:26 | |
| anteaya | pelix: what failures in testing? | 13:27 |
| *** smarcet has joined #openstack-infra | 13:27 | |
| anteaya | the host key has changed once since I have been with openstack (1.5 years) | 13:27 |
| anteaya | if the host key settings are accurate, there is no reason for tests to keep failing on host key checking | 13:27 |
| *** basha has joined #openstack-infra | 13:27 | |
| *** [1]trinaths has joined #openstack-infra | 13:28 | |
| *** lbragstad has quit IRC | 13:28 | |
| *** lbragstad has joined #openstack-infra | 13:29 | |
| *** lbragstad1 has quit IRC | 13:30 | |
| *** dims_ has joined #openstack-infra | 13:30 | |
| *** gokrokve has joined #openstack-infra | 13:31 | |
| *** trinaths has quit IRC | 13:31 | |
| *** [1]trinaths is now known as trinaths | 13:31 | |
| *** jgrimm has joined #openstack-infra | 13:34 | |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Make 'puppetmaster_host' configurable' https://review.openstack.org/99041 | 13:34 |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Make 'puppetmaster_host' configurable https://review.openstack.org/99041 | 13:35 |
| *** salv-orlando has quit IRC | 13:35 | |
| *** salv-orlando has joined #openstack-infra | 13:36 | |
| pelix | anteaya: take a look at https://review.openstack.org/#/c/96894/ | 13:36 |
| devananda | sdague: g'morning! not sure you saw my ping last night | 13:36 |
| pelix | anteaya: it's not testing against the openstack gerrit instance, I'm talking about the unit tests in git review | 13:36 |
| sdague | devananda: probably not | 13:37 |
| devananda | sdague: i'd like to land several patches for ironic today. will be working with the core team to rebase/reorganize/reprioritize our queue, since it was completely stalled for a week | 13:37 |
| devananda | sdague: and we've got a few crit and many high pri bug fixes in flight now | 13:37 |
| devananda | sdague: i know the gate is still fragile... so i'm checking before we approve much more. is this ok? | 13:38 |
| *** pblaho` has quit IRC | 13:38 | |
| sdague | devananda: sure, it will just be slow | 13:38 |
| devananda | sdague: understood. thanks | 13:39 |
| sdague | also it looks like ironic is racing on it's own tests quite a bit | 13:39 |
| devananda | sdague: we're getting a lot of failures from network issues | 13:39 |
| devananda | especially Bug 1326754 | 13:39 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,In progress] https://launchpad.net/bugs/1326754 | 13:39 |
| *** CaptTofu_ has joined #openstack-infra | 13:39 | |
| *** matsuhashi has quit IRC | 13:39 | |
| *** devkulkarni has joined #openstack-infra | 13:40 | |
| *** hdd_ has joined #openstack-infra | 13:40 | |
| devananda | sdague: are you seeing some other races in ironic? | 13:40 |
| *** devkulkarni has quit IRC | 13:40 | |
| sdague | honestly, I can't remember | 13:40 |
| sdague | sorry, kind of burnt out on bugs | 13:40 |
| *** devkulkarni has joined #openstack-infra | 13:40 | |
| *** eharney has joined #openstack-infra | 13:41 | |
| *** Longgeek has quit IRC | 13:42 | |
| *** Longgeek has joined #openstack-infra | 13:42 | |
| *** CaptTofu_ has quit IRC | 13:44 | |
| *** dkliban_afk is now known as dkliban | 13:45 | |
| *** signed8bit has joined #openstack-infra | 13:45 | |
| *** lbragstad has quit IRC | 13:48 | |
| *** lbragstad has joined #openstack-infra | 13:48 | |
| *** nosnos has quit IRC | 13:50 | |
| *** vtapia has joined #openstack-infra | 13:50 | |
| anteaya | pelix: okay so that is a patch | 13:50 |
| *** _nadya_ has quit IRC | 13:50 | |
| anteaya | pelix: connect me to where you are seeing failures | 13:50 |
| *** vtapia has left #openstack-infra | 13:50 | |
| *** homeless has joined #openstack-infra | 13:50 | |
| pelix | http://logs.openstack.org/94/96894/2/check/gate-git-review-python26/3daa39f/console.html | 13:51 |
| pelix | http://logs.openstack.org/94/96894/2/check/gate-git-review-python33/42a307a/console.html | 13:51 |
| *** basha has quit IRC | 13:51 | |
| pelix | plenty of others occurring, it seems to be a frequent problem that is just being worked around by commenting 'recheck no bug' in the reviews | 13:52 |
| anteaya | pelix: okay now I am with you: key_verify failed for server_host_key | 13:57 |
| anteaya | fungi hopefully will be around soon | 13:57 |
| anteaya | unless he has had to hide in a closet or work in his car today | 13:57 |
| pelix | anteaya: seems like there is some odd interaction between the ssh client and server | 13:57 |
| *** lbragstad1 has joined #openstack-infra | 13:57 | |
| *** lbragstad has quit IRC | 13:58 | |
| anteaya | pelix: yes, now that I see what you are seeing, I am in agreement with you | 13:58 |
| pelix | I'll try a workaround that may work better by setting HostKeyAlgorithms=ssh-rsa on the client side, and give it about 10-11 runs locally to see if I still hit the problem | 13:59 |
| pelix | if that doesn't work I'll look at disabling the stricthostkey checking | 14:01 |
| *** om has quit IRC | 14:03 | |
| *** jcoufal has quit IRC | 14:03 | |
| fungi | anteaya: yeah, i've been around, just catching up on bugs/e-mail/reviews | 14:05 |
| anteaya | fungi: figured as much | 14:05 |
| anteaya | ah the silence while it lasts | 14:05 |
| fungi | pelix: the git-review tests are racy--nobody's had time to fix them | 14:06 |
| fungi | one theory is adding bouncy castle instead of using gerrit's built-in crypto primitives may alleviate it | 14:06 |
| *** rlandy has quit IRC | 14:06 | |
| openstackgerrit | Christian Berendt proposed a change to openstack-infra/elastic-recheck: Prettified all HTML files https://review.openstack.org/98445 | 14:07 |
| fungi | for a while we were rechecking those failures with the number of the bug which requested adding tests, since it's technically not complete if they're not stable | 14:07 |
| fungi | but i've temporarily forgotten which bug number that is | 14:07 |
| *** CaptTofu_ has joined #openstack-infra | 14:08 | |
| fungi | however it's fundamentally a shortcoming of the current integration test framework for git-review, rather than any one particular test being at fault | 14:08 |
| fungi | we've at least added a lot of additional debugging output and logs to it, which has enabled us to rule out a lot of possible causes | 14:09 |
| *** basha has joined #openstack-infra | 14:10 | |
| boris-42 | fungi hi | 14:11 |
| fungi | boris-42: hi, what's up? | 14:11 |
| boris-42 | fungi I have some issues with new project, recently added | 14:11 |
| boris-42 | fungi I am not core in it =) | 14:11 |
| fungi | boris-42: what's the project name? i'll add you | 14:11 |
| boris-42 | fungi https://review.openstack.org/#/c/97778/ | 14:11 |
| fungi | it's a manual process we have to perform after the project is created | 14:12 |
| boris-42 | fungi hm | 14:12 |
| boris-42 | fungi even if group exist? | 14:12 |
| boris-42 | fungi cause I would like rally-core group to be maintainers of project | 14:13 |
| fungi | boris-42: oh, fun... looks like the acl didn't get applied. checking for typos in the review now | 14:13 |
| fungi | i don't see anything obviously mistyped which would have caused the acl not to apply. checking logs from the manage-projects run now | 14:15 |
| *** oomichi_ has quit IRC | 14:16 | |
| *** basha has quit IRC | 14:16 | |
| *** lbragstad1 has quit IRC | 14:16 | |
| *** devkulkarni has quit IRC | 14:16 | |
| pelix | fungi: I believe I have some fixes from them | 14:16 |
| *** zz_gondoi is now known as gondoi | 14:17 | |
| *** _nadya_ has joined #openstack-infra | 14:17 | |
| pelix | fungi: for git-review I suspect that it's a combination of missing method calls to setup the username correctly and Gerrit not handling the full list of keyalgothrims correctly | 14:17 |
| fungi | pelix: yes, i think you do | 14:17 |
| fungi | pelix: sounds plausible | 14:17 |
| boris-42 | fungi thanks | 14:18 |
| pelix | fungi: stressing my system using --concurrency=8 and 20 iterations at the moment to confirm :) | 14:18 |
| anteaya | sdague: I'm considering proposing a talk entitled "how to read the logs" | 14:20 |
| *** dhellmann has joined #openstack-infra | 14:20 | |
| *** pvo has joined #openstack-infra | 14:20 | |
| anteaya | mind you if I did that I would need to learn how to read the grenade logs | 14:20 |
| sdague | anteaya: ++ | 14:21 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Retry command on "hash mismatch" error in tests https://review.openstack.org/98354 | 14:21 |
| anteaya | okay, I will propose | 14:22 |
| anteaya | time to stop assuming people know how to read our logs and teach them | 14:22 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add a tool for determining unaccounted for rechecks https://review.openstack.org/94624 | 14:23 |
| *** miqui has quit IRC | 14:23 | |
| YorikSar | zzelle, fungi: ^ - I've given up waiting for that error to reproduce, so that would be the final patchset | 14:23 |
| *** miqui has joined #openstack-infra | 14:23 | |
| *** miqui has quit IRC | 14:23 | |
| sdague | anteaya: realistically I tried to do a little of that with the apache footer | 14:23 |
| *** miqui has joined #openstack-infra | 14:23 | |
| YorikSar | zzelle, fungi: I mean my CR to git-review https://review.openstack.org/98354 | 14:23 |
| sdague | however, it could be done better | 14:23 |
| *** rdopieralski has quit IRC | 14:24 | |
| *** isviridov is now known as isviridov|away | 14:24 | |
| anteaya | sdague: the documentation you provide is great | 14:24 |
| anteaya | sdague: mostly I'm casting about for a topic to propose so I can play in the water parks in brisbane | 14:24 |
| anteaya | mikal mattoliverau and jhesketh promised to take me to water | 14:24 |
| fungi | boris-42: it looks from the logs like manage-projects is completely broken again. i have to get ready for a meeting (and am unfortunately in meetings most of the day) but hopefully someone will have time to investigate further and fix it | 14:25 |
| *** mwagner_lap has joined #openstack-infra | 14:25 | |
| anteaya | so while I'm there, forcing myself to learn the logs better and teach others is a win | 14:25 |
| boris-42 | fungi ok will wait | 14:25 |
| boris-42 | fungi thanks for help | 14:25 |
| anteaya | fungi: can you link to the logs that shows manage-projects is broken? | 14:25 |
| *** adrian_otto has joined #openstack-infra | 14:26 | |
| fungi | anteaya: i'm not sure it will help. when that change merged puppet claimed that it though every acl file had changed and then got unspecified errors trying to apply them all | 14:26 |
| pabelanger | ohai! | 14:27 |
| fungi | i don't have time to dig deeper into other logs (for example in the gerrit end of the connection) to see what the issue might be | 14:27 |
| * anteaya nods | 14:27 | |
| anteaya | enjoy your meeting | 14:27 |
| fungi | and by every acl file i mean every acl file in our gerrit | 14:27 |
| anteaya | pabelanger: ohai | 14:27 |
| pabelanger | So, finally have some time to dig into https://review.openstack.org/#/c/83980/ again (delete surplus ready nodes in nodepool). Wanted to see how best to handle it now | 14:27 |
| fungi | but we haven't recently made any changes to jeepyb | 14:27 |
| anteaya | every acl file in gerrit? | 14:27 |
| fungi | yeah | 14:28 |
| belliott | can i get a 2nd core review on https://review.openstack.org/#/c/96570/ please? it's just a stackforge repo request | 14:28 |
| *** dkranz has joined #openstack-infra | 14:28 | |
| * anteaya looks at jeebyb logs | 14:28 | |
| anteaya | git log that is | 14:28 |
| anteaya | 9 days ago we merged this: http://git.openstack.org/cgit/openstack-infra/jeepyb/commit/?id=35bdef857a3b4ccdfe48b76b934a5d664d941b4b | 14:29 |
| anteaya | sdague: would that change how manage projects edits acl files? | 14:29 |
| *** jistr has quit IRC | 14:29 | |
| *** adrian_otto has quit IRC | 14:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Add hacking 0.9.0 series https://review.openstack.org/98824 | 14:30 |
| *** hashar has quit IRC | 14:31 | |
| *** rlandy has joined #openstack-infra | 14:31 | |
| *** jistr has joined #openstack-infra | 14:31 | |
| sdague | anteaya: yeh, and there might have been a change by fungi on top of it | 14:31 |
| anteaya | sdague: hmmmm | 14:32 |
| *** thomasbiege1 has quit IRC | 14:34 | |
| *** thomasbiege has joined #openstack-infra | 14:35 | |
| *** jaypipes has joined #openstack-infra | 14:37 | |
| *** pcrews_ has joined #openstack-infra | 14:38 | |
| *** jaypipes has quit IRC | 14:39 | |
| *** jaypipes has joined #openstack-infra | 14:39 | |
| *** jgrimm has quit IRC | 14:40 | |
| *** jaypipes has quit IRC | 14:41 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/reviewstats: Encode to utf-8 before printing https://review.openstack.org/99104 | 14:41 |
| *** jaypipes has joined #openstack-infra | 14:41 | |
| fungi | anteaya: sdague: yeah https://review.openstack.org/94684 but it's not gotten much reviewing yet | 14:42 |
| fungi | so not approved | 14:42 |
| *** jaypipes has quit IRC | 14:44 | |
| *** yaguang has quit IRC | 14:45 | |
| *** jgallard has joined #openstack-infra | 14:45 | |
| *** jaypipes has joined #openstack-infra | 14:45 | |
| *** thedodd has joined #openstack-infra | 14:45 | |
| fungi | gate-tempest-dsvm-virtual-ironic seems to be an unhappy camper in the gate at the moment | 14:46 |
| *** ominakov has quit IRC | 14:47 | |
| mordred_phone | sdague: spectacular | 14:50 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Fix a typo in HACKING.rst https://review.openstack.org/98998 | 14:50 |
| *** ominakov has joined #openstack-infra | 14:54 | |
| *** reed has joined #openstack-infra | 14:54 | |
| *** ominakov has quit IRC | 14:54 | |
| *** wenlock has joined #openstack-infra | 14:54 | |
| openstackgerrit | Dzmitry Horbach proposed a change to openstack-infra/jenkins-job-builder: support Managed Script Plugin https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin https://review.openstack.org/99109 | 14:56 |
| *** afazekas has quit IRC | 14:56 | |
| *** isviridov|away is now known as isviridov | 14:56 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add md5 checksum to groups tarball publisher https://review.openstack.org/97238 | 14:56 |
| *** mmaglana has joined #openstack-infra | 14:57 | |
| *** otherwiseguy has joined #openstack-infra | 14:57 | |
| fungi | mordred_phone: manage-projects needs love if you get near a real computer today... see review.o.o:/var/log/syslog.1 +/12:50:48 | 14:57 |
| *** roeyc has quit IRC | 14:58 | |
| * fungi will try to poke at it too after this first meeting wraps up | 14:58 | |
| *** maxbit has joined #openstack-infra | 14:58 | |
| *** james_li has joined #openstack-infra | 14:58 | |
| *** gokrokve has quit IRC | 14:59 | |
| *** palar_ has joined #openstack-infra | 15:00 | |
| *** gokrokve has joined #openstack-infra | 15:00 | |
| *** jcoufal has joined #openstack-infra | 15:01 | |
| *** mmaglana has quit IRC | 15:01 | |
| *** hashar has joined #openstack-infra | 15:02 | |
| anteaya | fungi: +1'd trusting merging will happen after current brokeness is addressed | 15:02 |
| *** jcoufal has quit IRC | 15:02 | |
| sdague | fungi: it does occur to me that some unit tests on manage-projects would probably help. It's about 20th on my todo list :) | 15:03 |
| *** palar has quit IRC | 15:03 | |
| *** jcoufal has joined #openstack-infra | 15:03 | |
| *** ihrachyshka has quit IRC | 15:04 | |
| *** penguinRaider has quit IRC | 15:04 | |
| *** HenryG has joined #openstack-infra | 15:04 | |
| *** palar_ has quit IRC | 15:05 | |
| *** wenlock has quit IRC | 15:07 | |
| *** ildikov_ has joined #openstack-infra | 15:08 | |
| *** marun has joined #openstack-infra | 15:08 | |
| *** ildikov has quit IRC | 15:09 | |
| *** skolekonov has quit IRC | 15:10 | |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added docs-on-rtfd for MagnetoDB https://review.openstack.org/99039 | 15:11 |
| *** mmaglana has joined #openstack-infra | 15:12 | |
| *** jistr has quit IRC | 15:12 | |
| *** jistr has joined #openstack-infra | 15:12 | |
| *** jpeeler has quit IRC | 15:13 | |
| *** praneshp has joined #openstack-infra | 15:13 | |
| *** ominakov has joined #openstack-infra | 15:14 | |
| Kiall | Is it just me, or is https://review.openstack.org/ failing to load? Stalls at "Loading Gerrit Code Review ..." | 15:15 |
| Kiall | DOOH | 15:15 |
| *** jcoufal has quit IRC | 15:15 | |
| sdague | just you | 15:15 |
| Kiall | I had a useragent switcher turned on to pretend I was using IE7 -_- | 15:15 |
| sdague | heh | 15:15 |
| mordred | fungi: excellent | 15:16 |
| sdague | you should be punished much for that | 15:16 |
| Kiall | sdague: lol - HP should be punished for having things that say "Only Internet Explorer 7 and Internet Explorer 8 are supported" | 15:16 |
| boris-42 | ^ hehe | 15:17 |
| *** dprince has quit IRC | 15:17 | |
| *** sarob has joined #openstack-infra | 15:17 | |
| *** markwash has joined #openstack-infra | 15:18 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added angular-momentjs to dependencies https://review.openstack.org/96713 | 15:18 |
| anteaya | Kiall: oh yes, do share that message | 15:18 |
| *** Ryan_Lane has joined #openstack-infra | 15:19 | |
| *** dangers_away is now known as dangers | 15:19 | |
| phschwartz | Kiall: Working for me. | 15:21 |
| Kiall | phschwartz: apologies, was my own stupidity ;) | 15:21 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added angular-momentjs to dependencies https://review.openstack.org/96713 | 15:21 |
| phschwartz | Kiall: It happens to us all some times. Me more then others especially yesterday | 15:21 |
| Kiall | ;) | 15:22 |
| *** atiwari has joined #openstack-infra | 15:24 | |
| *** jcoufal has joined #openstack-infra | 15:24 | |
| *** annegentle_ has joined #openstack-infra | 15:26 | |
| *** jgrimm has joined #openstack-infra | 15:28 | |
| *** changbl has joined #openstack-infra | 15:29 | |
| *** viktors is now known as viktors|afk | 15:30 | |
| mriedem | hmm, check-swift-dsvm-functional is voting...but doesn't seem to be checked by e-r? | 15:31 |
| *** gyee has joined #openstack-infra | 15:31 | |
| *** thedodd has quit IRC | 15:32 | |
| mriedem | nm | 15:32 |
| *** thedodd has joined #openstack-infra | 15:34 | |
| mriedem | trying to find anything in logstash that hits in screen-key.txt seems to be very sad atm | 15:36 |
| mriedem | not sure why that would be | 15:36 |
| *** trinaths has quit IRC | 15:36 | |
| sdague | mriedem: we're many hours behind in indexing | 15:38 |
| *** Longgeek has quit IRC | 15:39 | |
| mriedem | ok, there is that keystone IANA address bug that i hit, but e-r status page shows we haven't hit that in 2 weeks | 15:39 |
| mriedem | so let's just remember to not delete that one... | 15:39 |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added new python-magnetodbclient project to Stackforge https://review.openstack.org/91050 | 15:39 |
| mriedem | plus, god forbid i have to actually dig into logs and cross check launchpad for bugs, right? | 15:39 |
| *** jpeeler has joined #openstack-infra | 15:40 | |
| sc68cal | Hey, sorry to blast random questions, but for *-specs, the rendered HTML output, are there plans to take those rendered HTML files and place them in a known URL? I was thinking it'd be useful to be able to link to them in developer docs | 15:40 |
| sdague | fungi / mordred / SergeyLukjanov: https://review.openstack.org/#/c/98871/ | 15:41 |
| sc68cal | for example, specs for Neutron, we could link to them in the devref doc for neutron (http://docs.openstack.org/developer/neutron/devref/) | 15:41 |
| mordred | fungi: investigating manage-projets now | 15:41 |
| mordred | sdague: looking | 15:41 |
| bknudson | is there more to be done for https://review.openstack.org/#/c/97893/ ? Does Jenkins need a nudge to get it to pick up the change? | 15:41 |
| sdague | will reduce overall d-g load, and brings us back to defaults | 15:41 |
| *** basha has joined #openstack-infra | 15:42 | |
| mordred | sdague: +2 | 15:42 |
| *** changbl has quit IRC | 15:44 | |
| fungi | sc68cal: there are plans for a specs.openstack.org webpage where they will be organized and published. creating the site itself is easy and we can reuse the existing job templates for publishing them, but we also need someone to write a job to generate/update an overall index for the site | 15:44 |
| *** doug-fish has joined #openstack-infra | 15:45 | |
| mordred | fungi: ok. SO - the problem seems to be ... | 15:45 |
| sc68cal | fungi: awesome | 15:45 |
| mordred | fungi: for some reason | 15:45 |
| *** adrian_otto has joined #openstack-infra | 15:45 | |
| mordred | fungi: that even though the git diff --quiet is returning 0 | 15:45 |
| mordred | fungi: we're still trying to commit | 15:46 |
| mordred | if not copy_acl_config(project, repo_path, acl_config): | 15:46 |
| mordred | # nothing was copied, so we're done | 15:46 |
| mordred | return | 15:46 |
| mordred | therefore | 15:46 |
| mordred | is failing | 15:46 |
| fungi | oh, funzies | 15:53 |
| fungi | and i wonder why puppet claims they've all changed (says it has a different hash for each of them) | 15:53 |
| mordred | fungi: oh, you're going to LOVE this... | 15:53 |
| *** trinaths has joined #openstack-infra | 15:54 | |
| * fungi braces himself | 15:54 | |
| *** hdd_ has quit IRC | 15:55 | |
| *** tkelsey has quit IRC | 15:55 | |
| mordred | fungi: the boolean logic is reversed | 15:55 |
| *** Hal_ has quit IRC | 15:56 | |
| fungi | gah | 15:56 |
| fungi | exit code coerced into a python boolean type again? | 15:56 |
| mordred | kinda | 15:57 |
| *** IvanBerezovskiy has left #openstack-infra | 15:57 | |
| fungi | there really ought to be a standard method to all the subprocess calls for that situation, so people get used to using it instead of parsing != 0 | 15:57 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Swap the boolean logic on copy_acl_config https://review.openstack.org/99136 | 15:58 |
| mordred | fungi: ^^ | 15:58 |
| mordred | fungi: that's tested, btw - my brain still hurts thinking about it - so I just changed it in place to track what was happening | 15:58 |
| *** basha has quit IRC | 15:58 | |
| *** markwash has quit IRC | 15:58 | |
| *** rdopieralski has joined #openstack-infra | 15:59 | |
| fungi | mordred: ohh, that happened in the recent exceptions-for-errors improvement i'm betting, which is why we hadn't spotted it until now | 15:59 |
| *** julim has quit IRC | 15:59 | |
| fungi | so it's been broken less than a week at least | 16:00 |
| pleia2 | morning | 16:00 |
| mordred | fungi: yah | 16:00 |
| mordred | fungi: and it's an honest mistake - I stared at it for ages | 16:01 |
| *** wenlock has joined #openstack-infra | 16:01 | |
| *** terryw has joined #openstack-infra | 16:01 | |
| fungi | i reviewed the change too and it didn't jump out at me. then re-reviewed it a little while ago when i noticed this, since it was the most recent change... still wasn't obvious | 16:01 |
| *** otherwiseguy has quit IRC | 16:01 | |
| phschwartz | I laughed at that one. That is a tough one to catch too. | 16:02 |
| fungi | it is a persistent thorn in the side of python scripts which call out to shells | 16:02 |
| *** julim has joined #openstack-infra | 16:02 | |
| morganfainberg | mriedem, the IANA bug is a long way off from being 100% solved. but we've reduced the impact as much as reasonable + documenting. | 16:02 |
| mordred | fungi: we could always start using the sh module | 16:03 |
| morganfainberg | mriedem, as aFYI of where we sit with it | 16:03 |
| fungi | mordred: i'm always tempeted by that module, and then i seem to fond reasons not to use it but afterward can't recall why | 16:03 |
| fungi | s/fond/find/ | 16:03 |
| phschwartz | fungi: I have always had the same thing crop up. I forced myself to use it and deal with the issues and seem to have liked it when I have used it. | 16:05 |
| *** otherwiseguy has joined #openstack-infra | 16:06 | |
| *** mbacchi has quit IRC | 16:06 | |
| *** jgrimm has quit IRC | 16:07 | |
| *** dizquierdo has quit IRC | 16:07 | |
| mordred | fungi: last time I poked at it there was one bug we needed fixed and I sent in a PR | 16:07 |
| *** markwash has joined #openstack-infra | 16:07 | |
| *** aconrad has joined #openstack-infra | 16:08 | |
| *** marcoemorais has joined #openstack-infra | 16:08 | |
| mordred | let me look and see if it's fixed | 16:10 |
| *** terryw has quit IRC | 16:10 | |
| anteaya | morning pleia2 | 16:10 |
| *** annegentle_ has quit IRC | 16:10 | |
| phschwartz | mordred: I started moving my playbooks to use the nova_compute module instead of the Rax module that way I can upstream my playbook. I have also moved my dns work to a role for dns that uses the rax_dns module. | 16:13 |
| *** annegent_ has joined #openstack-infra | 16:14 | |
| phschwartz | But that will wait. I have been working this morning on lending a hand with reviews. :) | 16:14 |
| mordred | awesome | 16:14 |
| * mordred hands phschwartz a puppy dog | 16:14 | |
| *** mmaglana has quit IRC | 16:15 | |
| phschwartz | mordred: now I have to stop and clean my laptops keyboard and screen as my coffee just decided of its own accord to relocate from my mouth. | 16:15 |
| *** arnaud has joined #openstack-infra | 16:18 | |
| fungi | dstufft: finally got around to opening https://bitbucket.org/pypa/wheel/issue/116/packages-with-only-filesdata_files-get | 16:19 |
| *** jaypipes has quit IRC | 16:20 | |
| *** adrian_otto has quit IRC | 16:20 | |
| fungi | phschwartz: thank you for reviewing! (and thanks to everyone who is reviewing, and shame on me for not reviewing enough) | 16:21 |
| phschwartz | Hey, I promised that I was going to get more involved. | 16:21 |
| *** jgallard has quit IRC | 16:22 | |
| *** om has joined #openstack-infra | 16:22 | |
| *** e0ne has quit IRC | 16:22 | |
| phschwartz | sdague: When you have some time, I am interested in here what your doing with devstack-vagrant :) | 16:23 |
| *** adrian_otto has joined #openstack-infra | 16:23 | |
| *** thomasbiege1 has joined #openstack-infra | 16:25 | |
| sdague | phschwartz: sure, there was a whole thread on the mailing list about the fact that there were a bunch of these out there | 16:25 |
| rdopieralski | a question -- launchpad allegedly has some poll/voting functionality -- is there a way for me to set up a poll for horizon's new name? | 16:25 |
| sdague | so was trying to consolidate down | 16:25 |
| sdague | to a shared stackforge repo | 16:26 |
| sdague | though that's currently blocked on cla question | 16:26 |
| phschwartz | hmm, interesting that I haven't noticed it at all. | 16:26 |
| *** dprince has joined #openstack-infra | 16:26 | |
| sdague | phschwartz: it was a couple months ago on -dev | 16:26 |
| *** [1]trinaths has joined #openstack-infra | 16:26 | |
| dstufft | fungi: awesome | 16:26 |
| *** thomasbiege has quit IRC | 16:26 | |
| phschwartz | But then I now realize for the past week I am not getting mail from the lists. | 16:26 |
| fungi | dstufft: digging in now but i think it's probably a trivial patch to is_pure() | 16:26 |
| *** marcoemorais has quit IRC | 16:27 | |
| phschwartz | fungi: mordred: Any chance one of you can see if the server for the lists is getting bounces from rackspace.com again? | 16:27 |
| *** marcoemorais has joined #openstack-infra | 16:27 | |
| phschwartz | sdague: I will have to look at the digests, ty | 16:27 |
| *** denis_makogon_ has joined #openstack-infra | 16:27 | |
| *** trinaths has quit IRC | 16:27 | |
| *** [1]trinaths is now known as trinaths | 16:27 | |
| fungi | rdopieralski: that or a civs poll. were you wanting to use that to narrow down the top candidates before sending them to the foundation for vetting/research? | 16:27 |
| sdague | phschwartz: http://lists.openstack.org/pipermail/openstack-dev/2014-April/032495.html | 16:27 |
| phschwartz | sdague: ty | 16:28 |
| fungi | phschwartz: checkingh | 16:28 |
| rdopieralski | fungi: yes, pretty much | 16:28 |
| rdopieralski | fungi: do you know how I would set it up? | 16:28 |
| *** marcoemorais has quit IRC | 16:28 | |
| rdopieralski | fungi: preferably, we would like horizon contributors to be able to vote | 16:28 |
| *** marcoemorais has joined #openstack-infra | 16:29 | |
| fungi | rdopieralski: no, though ttx might have suggestions (and opinions on whether lp or civs is the better option there) | 16:29 |
| *** marcoemorais has quit IRC | 16:29 | |
| *** james_li has quit IRC | 16:29 | |
| rdopieralski | ttx: can you help? :) | 16:29 |
| *** ihrachyshka has joined #openstack-infra | 16:29 | |
| *** marcoemorais has joined #openstack-infra | 16:29 | |
| *** jgrimm has joined #openstack-infra | 16:29 | |
| ttx | rdopieralski: actually polls i nLP are set up at the group level | 16:30 |
| *** thomasbiege has joined #openstack-infra | 16:30 | |
| *** thomasbiege1 has quit IRC | 16:30 | |
| ttx | rdopieralski: so depending on who you want to allow to vote, it may not work for you | 16:31 |
| ttx | for public polls, you should use something like surveymonkey | 16:31 |
| mordred | or civs | 16:31 |
| ttx | for clear electorate polls, you should use CIVS | 16:31 |
| *** thomasbiege has quit IRC | 16:31 | |
| mordred | rdopieralski: horizon is getting renamed? | 16:32 |
| ttx | for pools over a Launchpad group (but who uses those those days) you can use LP polls | 16:32 |
| fungi | mordred: half of horizon is getting renamed ;) | 16:32 |
| ttx | polls* | 16:32 |
| rdopieralski | mordred: it's getting split, and one part of it is getting renamed | 16:32 |
| * ttx needs to run | 16:32 | |
| rdopieralski | ttx: any hint on how to find the polls on lanunchpad? | 16:33 |
| *** jaypipes has joined #openstack-infra | 16:33 | |
| rdopieralski | anybody? :) | 16:33 |
| ttx | rdopieralski: pools in Launchpad are done at a group level | 16:33 |
| ttx | polls* | 16:33 |
| rdopieralski | so there is Horizon Developers group that looks right | 16:33 |
| ttx | rdopieralski: there is no public poll in LP | 16:33 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Ignore newline in bp/bug search in commit message https://review.openstack.org/98324 | 16:33 |
| ttx | rdopieralski: I doubt that it's current though | 16:33 |
| fungi | phschwartz: going back to late last week, the only rackspace.com rejects the lists server has logged are these... 2014-06-07 07:25:00 H=localhost (lists.openstack.org) [127.0.0.1] F=<mailman-loop@lists.openstack.org> rejected RCPT <amesserl@rackspace.com>: Sender verify failed | 16:34 |
| ttx | rdopieralski: but let's use that as an example | 16:34 |
| fungi | phschwartz: any idea what the date/time on the last message you received was? | 16:34 |
| *** annegent_ is now known as annegentle_ | 16:34 | |
| ttx | Poll creation there would be at https://launchpad.net/~horizon/+newpoll | 16:34 |
| rdopieralski | ah, not allowed, I see, I will neet to ask our ptl | 16:34 |
| ttx | You probably need to be a team admin to be able to do that | 16:35 |
| phschwartz | fungi: 06-04 5:59pm est | 16:35 |
| *** mriedem has quit IRC | 16:35 | |
| fungi | phschwartz: oh, that long. i'll dig back further then | 16:35 |
| ttx | rdopieralski: and the team admins are way out of date, which gives you a hint as to whether the group is actually up to date | 16:35 |
| phschwartz | fungi: if you don't see anything it might be on this side. | 16:36 |
| ttx | team admin is Devin Carlen | 16:36 |
| rdopieralski | ttx: ok, I will use civs then, thank you | 16:36 |
| *** adrian_otto has quit IRC | 16:36 | |
| *** markmcclain has joined #openstack-infra | 16:36 | |
| ttx | who has not been touching anything openstack since 2012 | 16:36 |
| ttx | :) | 16:36 |
| ttx | rdopieralski: sounds like a better bet, yes. Also let you express preferences | 16:37 |
| *** gondoi is now known as zz_gondoi | 16:37 | |
| ttx | rather than arbitrary pick | 16:37 |
| * ttx really runs now | 16:37 | |
| rdopieralski | thank you! | 16:37 |
| mordred | ttx: should we delete those teams? | 16:39 |
| *** lakshmiS has joined #openstack-infra | 16:40 | |
| *** isviridov is now known as isviridov|away | 16:40 | |
| *** gokrokve_ has joined #openstack-infra | 16:40 | |
| *** aconrad has quit IRC | 16:41 | |
| *** mmaglana has joined #openstack-infra | 16:41 | |
| *** gokrokve has quit IRC | 16:43 | |
| fungi | phschwartz: i show deliveries to you continually after that time, if you need message ids/next hops to track down in your receiving mta | 16:43 |
| phschwartz | fungi: no, I should need it. As long as they are coming it, the trashing of them by the spam filters should be easy to trace. ty | 16:44 |
| fungi | phschwartz: yw | 16:44 |
| *** hcc has joined #openstack-infra | 16:45 | |
| *** hcc has quit IRC | 16:45 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Swap the boolean logic on copy_acl_config https://review.openstack.org/99136 | 16:46 |
| *** ramashri has joined #openstack-infra | 16:46 | |
| phschwartz | fungi: And fixed. The spam filter was getting me. I just received about 40 messages. | 16:48 |
| *** mwagner_lap has quit IRC | 16:48 | |
| *** dkranz has quit IRC | 16:49 | |
| *** mriedem has joined #openstack-infra | 16:49 | |
| *** trinaths has quit IRC | 16:49 | |
| *** basha has joined #openstack-infra | 16:50 | |
| *** changbl has joined #openstack-infra | 16:50 | |
| zaro | morning | 16:50 |
| phschwartz | ok, time to take a break and clear my head. That is 4 hours of reviewing | 16:51 |
| *** wenlock has quit IRC | 16:52 | |
| *** hashar has quit IRC | 16:54 | |
| *** jcoufal has quit IRC | 16:55 | |
| *** melwitt has joined #openstack-infra | 16:56 | |
| *** _nadya__ has joined #openstack-infra | 16:56 | |
| *** james_li has joined #openstack-infra | 16:57 | |
| jogo | https://review.openstack.org/#/c/99078/ | 16:59 |
| jogo | I thought the proposla bot doesn't push out new hacking changes? | 16:59 |
| Alex_Gaynor | jogo: why wouldn't it? | 16:59 |
| jogo | because you can't just bump hacking versions on there own | 17:00 |
| *** denis_makogon_ is now known as denis_makogon | 17:00 | |
| jogo | you have to update the ignore list or fix things etc | 17:00 |
| *** _nadya_ has quit IRC | 17:00 | |
| jogo | hacking is not backwards compatable accross 0.x.0 releases | 17:00 |
| jogo | in the sense that new rules are added | 17:00 |
| Alex_Gaynor | Personally I think it's fine, this is probably the best way to notify people they need to update their code | 17:01 |
| jogo | also having issues with the requirments sync | 17:01 |
| jogo | but the patch that is proposed cannot be landed | 17:01 |
| jogo | also g-r has a differnt version then what projects should have | 17:01 |
| *** julim has quit IRC | 17:03 | |
| *** weshay has quit IRC | 17:04 | |
| *** weshay has joined #openstack-infra | 17:04 | |
| *** jpich has quit IRC | 17:04 | |
| clarkb | o/ | 17:05 |
| jogo | clarkb: o/ | 17:05 |
| *** doug-fish has quit IRC | 17:05 | |
| *** thomasbiege has joined #openstack-infra | 17:05 | |
| *** yjiang51 has joined #openstack-infra | 17:05 | |
| openstackgerrit | Andrew Melton proposed a change to openstack/requirements: Add IDMapShift for Nova https://review.openstack.org/98888 | 17:05 |
| jogo | who has been working on the requirements bot? and checks | 17:06 |
| *** aconrad has joined #openstack-infra | 17:06 | |
| *** harlowja_away is now known as harlowja | 17:08 | |
| *** melwitt has quit IRC | 17:13 | |
| *** derekh_ has quit IRC | 17:15 | |
| *** zzelle_ has joined #openstack-infra | 17:15 | |
| clarkb | jogo: what do you mean by that? | 17:16 |
| fungi | jogo: i think the ideal solution there is that if a project sees proposed new hacking bumps breaking for them, they should first update their tox.ini to exclude the new things they need excluded and once that's merged the reqs proposal will magically pass | 17:16 |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 17:16 |
| *** basha has quit IRC | 17:17 | |
| mordred | jogo: yeah - what fungi said. the system is working as designed | 17:17 |
| mordred | jogo: it's the same with any other requirement | 17:17 |
| jogo | fungi: but the version limits in global requirements aren't what projects should sue | 17:17 |
| jogo | use | 17:17 |
| mordred | sometimes, to be able to land the req change, you have to git review -d it, fix the code, and re-upload | 17:17 |
| mordred | jogo: the version limits in global requirements should be what projects should use | 17:17 |
| fungi | they *could* just amend the proposal, but as we've seen in the past new proposal updates are frequent enough that they may not get through review before it overwrites them aith another update | 17:18 |
| jogo | but that would be a non overalaping change | 17:18 |
| jogo | overlapping | 17:18 |
| jogo | go from hacking 0.8.x to 0.9.x | 17:18 |
| jogo | so global-reqs has support both 0.8 and 0.9 so that 0.8 stays in the mirror etc | 17:19 |
| *** jgrimm has quit IRC | 17:19 | |
| *** HenryG has quit IRC | 17:19 | |
| *** melwitt has joined #openstack-infra | 17:19 | |
| fungi | jogo: i think overlap only matters for the global-reqs list. sometimes projects lag behind global-reqs long enough that non-overlapping version updates in a project's reqs is unavoidable | 17:19 |
| zaro | pelix: did you do anything besides rebase this? https://review.openstack.org/#/c/48783 | 17:20 |
| fungi | the overlap check we wanted to install was to avoid jobs running in the gate demanding newer hacking than we had updated into our mirror yet | 17:20 |
| *** HenryG has joined #openstack-infra | 17:20 | |
| jogo | fungi: what about the mirror? | 17:20 |
| *** otherwiseguy has quit IRC | 17:20 | |
| fungi | or s/hacking/whatever/ | 17:20 |
| fungi | jogo: what about the mirror? | 17:20 |
| jogo | older versions will stay around? | 17:20 |
| fungi | jogo: yes, indefinitely | 17:20 |
| fungi | we don't purge things from our mirror when global-reqs no longer refers to them | 17:21 |
| fungi | they stick around forever | 17:21 |
| pelix | zaro: some rework around the tests/base.py due to another merged change | 17:21 |
| pelix | zaro: but besides that, it's just a rebase so that it'll merge cleanly | 17:22 |
| *** rdopieralski has quit IRC | 17:23 | |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Update hacking version to align with what projects should use https://review.openstack.org/99172 | 17:24 |
| jogo | fungi: in that case ^ should fix things | 17:24 |
| *** doug-fish has joined #openstack-infra | 17:25 | |
| *** wenlock has joined #openstack-infra | 17:25 | |
| *** sabari has quit IRC | 17:25 | |
| *** markmcclain has quit IRC | 17:25 | |
| *** om has quit IRC | 17:26 | |
| *** wenlock_ has joined #openstack-infra | 17:26 | |
| *** dims_ has quit IRC | 17:28 | |
| *** markmcclain has joined #openstack-infra | 17:29 | |
| *** wenlock has quit IRC | 17:29 | |
| fungi | jogo: yep, as long as the earlier change to set it to hacking>=0.8.0,<0.10,!=0.9.0 already merged, that should be safe | 17:29 |
| jogo | at one point the requirments bot didn't push out new hacking versions I thought | 17:29 |
| *** wenlock_ has quit IRC | 17:30 | |
| fungi | which it seems to have | 17:30 |
| *** wenlock has joined #openstack-infra | 17:30 | |
| fungi | jogo: i have no idea what would have prevented it from suggesting a sync with the hacking version in global requirements, other than a bug (the proposal bot was broken for extended periods of time last cycle) | 17:30 |
| *** trinaths has joined #openstack-infra | 17:32 | |
| *** penguinRaider has joined #openstack-infra | 17:33 | |
| mriedem | is there a config script somewhere that tells jenkins which files to archive for builds? | 17:33 |
| *** ociuhandu has quit IRC | 17:33 | |
| mriedem | not seeing grenade status/*.failure logs archived | 17:33 |
| jogo | fungi: hrm anyway looks like this should solve things | 17:33 |
| *** aconrad has quit IRC | 17:34 | |
| *** wenlock has quit IRC | 17:36 | |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Minor updates, new gerrit and zuul screenshots https://review.openstack.org/99173 | 17:37 |
| *** denis_makogon has quit IRC | 17:37 | |
| *** amotoki_ has quit IRC | 17:37 | |
| pleia2 | image diffs ftw! | 17:37 |
| fungi | mriedem: that gets enumerated in openstack-infra/devstack-gate functions.sh | 17:37 |
| zaro | pelix: do you think we need a 'yaml_loader' section or can include_path just get put under job_builder section in jenkins.ini? | 17:38 |
| pelix | previous reviews complained when I had it that way :) | 17:38 |
| pelix | but also it probably is a better separation in that other options could in theory be added to control how the yaml loading is performed which is entirely separate to how the jobs themselves are parsed/generated or uploaded | 17:39 |
| *** trinaths has quit IRC | 17:39 | |
| *** wenlock has joined #openstack-infra | 17:39 | |
| *** duran has joined #openstack-infra | 17:40 | |
| pelix | if we put the include_path under the job_builder section, what's the argument for separating any of the other options? | 17:40 |
| *** jistr has quit IRC | 17:40 | |
| pelix | either way doesn't really bother me, as long as it seems consistent and is acceptable | 17:41 |
| *** markwash_ has joined #openstack-infra | 17:41 | |
| *** dims_ has joined #openstack-infra | 17:42 | |
| *** thedodd has quit IRC | 17:42 | |
| jesusaurus | Can I get some reviews on https://review.openstack.org/#/c/98549/ and https://review.openstack.org/#/c/98561/ ? I would really like to fix the hostname issue on the hpcloud bare-precise images | 17:42 |
| *** markwash has quit IRC | 17:43 | |
| *** markwash_ is now known as markwash | 17:43 | |
| zaro | good point. i'm asking because i just reviewed the recursive option. the proposal for that one is in job_builder. i'm thinking these should be in same section? | 17:43 |
| *** _nadya__ has quit IRC | 17:45 | |
| *** e0ne has joined #openstack-infra | 17:47 | |
| zaro | pelix: hmm, i can't seem to find any comments in the change discussing the jenkins.ini sections. | 17:47 |
| *** wenlock has quit IRC | 17:48 | |
| pelix | hmm, maybe I'm confusing it with another review? | 17:48 |
| pelix | or possibly just asked in here at the time | 17:48 |
| *** sballe_ has quit IRC | 17:50 | |
| *** zns has joined #openstack-infra | 17:51 | |
| *** ildikov_ has quit IRC | 17:53 | |
| *** wenlock has joined #openstack-infra | 17:54 | |
| *** ildikov_ has joined #openstack-infra | 17:54 | |
| *** alkari has joined #openstack-infra | 17:56 | |
| *** ihrachyshka has quit IRC | 17:57 | |
| *** andreykurilin_ has joined #openstack-infra | 17:57 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint fur bug 1268725 https://review.openstack.org/98913 | 17:57 |
| uvirtbot | Launchpad bug 1268725 in openstack-ci "ConnectionError: HTTPConnectionPool(host='pypi.openstack.org', port=80): Max retries exceeded" [Medium,Triaged] https://launchpad.net/bugs/1268725 | 17:57 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 and 1326855 https://review.openstack.org/98919 | 17:57 |
| devananda | hi! any chance to get another infra core to review/approve this -- https://review.openstack.org/#/c/98886/ -- it's causing about half of ironic's tests to fail in teh gate right now | 17:58 |
| devananda | see the E-R query for bug 1326754 | 17:58 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,In progress] https://launchpad.net/bugs/1326754 | 17:58 |
| devananda | that fix looks like it should help a lot, at least afaict | 17:59 |
| *** e0ne has quit IRC | 17:59 | |
| fungi | jd__: the reply on https://github.com/smontanaro/pylockfile/issues/11 makes it sound like we should consider offering to absorb pylockfile (like we've done for sqlalchemy-migrate) or look at replacing use of it with a different module | 17:59 |
| zaro | pelix: discussion about jenkins sections was in this change: https://review.openstack.org/#/c/52080 | 18:00 |
| clarkb | devananda: done | 18:01 |
| *** jlibosva has quit IRC | 18:01 | |
| devananda | clarkb: tyvm! | 18:01 |
| zaro | pelix: PS #12 | 18:01 |
| clarkb | devananda: you should see that in ~25 minutes or so | 18:01 |
| clarkb | devananda: assuming it merges shortly | 18:01 |
| pelix | zaro: looking | 18:01 |
| devananda | clarkb: fantastic. | 18:01 |
| *** rwsu has joined #openstack-infra | 18:02 | |
| pelix | zaro: guess since it's not encapsulated within the localyaml class, and is instead controlled by what is passed to that class, it makes sense to have it under the job_builder section to start with | 18:02 |
| mordred | devananda: everyhting about that patch makes me sad | 18:02 |
| mordred | devananda: why, in the name of GOD does a-a-r try to install ubuntu-cloud-keyring at run time? | 18:03 |
| mordred | shouldn't that just be a depend of a-a-r? | 18:03 |
| zaro | pelix: i think i agree, doesn't seem like there's a need for seperate section in this case. | 18:03 |
| mordred | zul: ^^ I blame you | 18:03 |
| *** jaypipes has quit IRC | 18:03 | |
| devananda | clarkb: fyi, we've already gotten 7 bug fixes landed today, with another ~10 on the way | 18:03 |
| clarkb | woot | 18:03 |
| clarkb | thanks | 18:03 |
| zul | mordred: what did i do or didnt do? | 18:03 |
| pelix | zaro: I was thinking about adding a safe_load option at some stage in case people wanted to explicitly disable within their configuration the possibility of yaml instantiating python objects. Does that follow on well enough? | 18:03 |
| lifeless | clarkb: https://review.openstack.org/#/c/98043/ <- any red flags on that? Its blocking us getting much better logs out of things | 18:04 |
| clarkb | sdague: question about 99016, doesn't devstack minimally do updates now? has that been insufficient? | 18:04 |
| clarkb | lifeless: what does logs/** look like? | 18:05 |
| *** mrodden has joined #openstack-infra | 18:05 | |
| clarkb | lifeless: is it logs/new/nova-api.txt and so on? | 18:05 |
| *** ildikov_ has quit IRC | 18:06 | |
| *** changbl has quit IRC | 18:06 | |
| sdague | clarkb: the reason for trying 99016 is that we've got a pretty big failure rate due to apt mirrors being down | 18:06 |
| clarkb | sdague: even after the devstack change? | 18:06 |
| zaro | pelix: you mean as another option in job_builer? | 18:06 |
| pelix | zaro: yeap, one that makes the yaml loading go through the safe loader instead of the default | 18:07 |
| clarkb | sdague: and why are we reverting the ceilometer poll interval change? | 18:07 |
| sdague | clarkb: the devstack change is inbound, this is sort of orthonginal, but based on what I see in the devstack logs, I doubt the retry is going to help | 18:07 |
| devananda | mordred: where should d-g be pulling that instead? | 18:07 |
| clarkb | sdague: well importantly it only updates once | 18:07 |
| sdague | more interestingly, 50% of our nodepool images don't have an actual apt cache | 18:08 |
| clarkb | sdague: which I think is more important than the retry. Once we have a good update we don't update again | 18:08 |
| sdague | clarkb: if it works | 18:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Update apt cache before installing UCA keyring https://review.openstack.org/98886 | 18:08 |
| sdague | honestly, I really doubt the blip was that quick | 18:08 |
| sdague | it's not like they die on 1 update file | 18:08 |
| clarkb | sdague: yeah that was my concern too | 18:09 |
| lifeless | clarkb: seed_logs/nova-compute.log, seed_logs/nova-conductor.log, seed_logs/rabbitmq-server.log etc https://review.openstack.org/#/c/97947/2/toci_devtest.sh is the change that sets that up | 18:09 |
| sdague | they typically fail all of them | 18:09 |
| clarkb | lifeless: thanks | 18:09 |
| lifeless | clarkb: the ubuntu logs aren't quite as pretty | 18:09 |
| lifeless | clarkb: the problem we're trying to solve is that we need N where N may be large copies of nova-conductor.log | 18:09 |
| fungi | clarkb: sdague: yeah, i pointed that out as well. i think it solves some classes of failure (and does reduce the number of times we connect) but won't solve the main ones we've seen recently i think | 18:09 |
| devananda | huh, i just started getting 500 from github.com | 18:09 |
| zaro | pelix: that makes sense to me. can you tell me what the use case would be for safe load? | 18:09 |
| lifeless | and of mysql-server.log | 18:09 |
| lifeless | etc | 18:09 |
| lifeless | clarkb: so we can I guess prefix-mangle each file, if that will work better | 18:10 |
| clarkb | gotcha | 18:10 |
| lifeless | clarkb: but a directory structure just seems more intuitive | 18:10 |
| anteaya | people being idiots | 18:10 |
| sdague | clarkb: on the ceilometer revert, I'd like to take the polling back to the defaults | 18:10 |
| clarkb | yup works for me. just wanted to make sure we werent doing something like logs/var/log/upstart/foo | 18:10 |
| lifeless | clarkb: so we may do that on Ubuntu | 18:10 |
| clarkb | sdague: oh right I thought we had to set the value but the default is fine | 18:11 |
| clarkb | sdague: just needed to wire that up in my brain. thanks | 18:11 |
| lifeless | clarkb: we have two hierarchies there - the /mnt/state/var/log and the root /var/log one | 18:11 |
| clarkb | lifeless: oh, please don't :P | 18:11 |
| pelix | zaro: protection from code injection security issues | 18:11 |
| lifeless | clarkb: the common path is / | 18:11 |
| johnthetubaguy | hey I had noticed lots of failed builds in Rackspace IAD for the node pool builds, noticed from our side, that should all be sorted now, I hope | 18:11 |
| clarkb | lifeless: or at least make the prefix minimal | 18:11 |
| fungi | johnthetubaguy: thanks! | 18:11 |
| lifeless | clarkb: is that a UI concern, or system resources/performance? | 18:12 |
| *** arnaud has quit IRC | 18:12 | |
| fungi | johnthetubaguy: we've sort of become numb to instance boot errors in providers since they come and go so often any more | 18:12 |
| clarkb | lifeless: a little of both. dir cleanup with find is a bit clunky right now so that deeper the tree the longer it takes to free those inodes | 18:12 |
| fungi | johnthetubaguy: unless they get so severe we're mostly out of resources to run things | 18:12 |
| pelix | zaro: you can use yaml to create python objects, the default yaml.load is only supposed to be used with trusted yaml code, while the yaml.safe_load is what should be used if you are accepting code from other sources | 18:12 |
| clarkb | lifeless: and as a user digging into those deeper trees is a bit annoying | 18:12 |
| clarkb | lifeless: eventually we will be all swifted then it wont matter | 18:13 |
| johnthetubaguy | fungi: yeah, that sucks, I kinda surprised things kept working given how many errors were happening, anyways should be sorted now | 18:13 |
| clarkb | sdague: fungi: was there any reason to keep the ceilometer poll interval? eg tests that needed to run? I think there are some proposed changes up now for that. | 18:13 |
| ArxCruz | Is there a command to upgrade nodepool db? latest upgrade create a new column in database, and nodepool is failing because this wasn't created | 18:13 |
| sdague | went through that with the ceilo team | 18:14 |
| sdague | almost all the proposed changes now don't need it | 18:14 |
| zaro | pelix: ahh, thanks for the security lesson. that would be great! | 18:14 |
| clarkb | ArxCruz: you can either drop the entire DB and restart nodepool to recreate it (you will have to do manual cleanup of reosurces). Or you can add the column to the node table | 18:14 |
| ArxCruz | clarkb: okay, thanks :) | 18:14 |
| sdague | it will also reduce the ceilometer API traffic by a factor of 40 | 18:15 |
| sdague | which I think is probably a better place to hang for now | 18:15 |
| clarkb | sdague: ++ approving | 18:15 |
| *** wenlock has quit IRC | 18:15 | |
| lifeless | clarkb: so, AIUI the choice we have is a) flatten the namespace ourselves, or allow for a hierarchy and iterate the test-script code towards a better shape | 18:15 |
| *** aconrad has joined #openstack-infra | 18:15 | |
| clarkb | lifeless: right. d-g does flattening of the namespace for the devstack logs | 18:16 |
| *** nikhil___ has joined #openstack-infra | 18:16 | |
| lifeless | clarkb: but right now we bundle all the logs into one tar, which doesn't get indexed by logstash AIUI | 18:16 |
| clarkb | correct | 18:16 |
| lifeless | clarkb: and having lots (lots! for scale tests, coming soon to a crazy pants person near you) | 18:16 |
| lifeless | clarkb: of flattened logs terrifies me | 18:16 |
| nikhil___ | hey, I'm looking to do a recheck on a MP and was wondering if the gates were looking okay or do you want us to wait a bit more? | 18:16 |
| clarkb | lifeless: you have to do manual flatting of some sort across different nodes | 18:17 |
| *** Alexei_987 has joined #openstack-infra | 18:17 | |
| phschwartz | who is the best person to reach out to with questions about the xenserver ci gating? | 18:17 |
| lifeless | clarkb: also searching for 'nova-conductor.log with foo in it' in logstash might be tricky if nova-conductor.log is seed_nova-conductor.log, undercloud-$uuid_nova-conductor.log etc | 18:17 |
| clarkb | phschwartz: BobBall is probably the best person | 18:18 |
| clarkb | lifeless: no it should be seed/nova-conductor | 18:18 |
| *** __afazekas is now known as afazekas | 18:18 | |
| *** adrian_otto has joined #openstack-infra | 18:18 | |
| johnthetubaguy | phschwartz: thats me or BobBall | 18:18 |
| anteaya | phschwartz: BobBall | 18:18 |
| lifeless | clarkb: can only do that if we can turn the preservation of hierarchy on | 18:18 |
| clarkb | lifeless: we grok basename correcntly | 18:18 |
| johnthetubaguy | I am trying to help BobBall with that at the moment | 18:18 |
| anteaya | johnthetubaguy: are you xenserver ci too?, awesome | 18:18 |
| lifeless | clarkb: I was investigating the consequences of doing flattening ourselves. | 18:18 |
| clarkb | lifeless: I am saying flatten the hierarchy | 18:19 |
| anteaya | nice to meet you, johnthetubaguy | 18:19 |
| lifeless | clarkb: now I'm confused | 18:19 |
| clarkb | logs/state/seed/nova-foo | 18:19 |
| johnthetubaguy | anteaya: kinda, yeah, trying to get some help for Bob | 18:19 |
| clarkb | logs/var/seed/nova-foo | 18:19 |
| clarkb | or whatever | 18:19 |
| *** _nadya_ has joined #openstack-infra | 18:19 | |
| clarkb | as long as basename is correct things should work | 18:19 |
| anteaya | johnthetubaguy: glad to hear it, Bob is awesome and its great you are helping him | 18:19 |
| lifeless | clarkb: sure, but thats lost when the scp flattens it | 18:19 |
| phschwartz | clarkb: anteaya: johnthetubaguy: ty | 18:19 |
| clarkb | lifeless: right, I am on board with scp keeping the hierarchy | 18:19 |
| lifeless | clarkb: ok | 18:20 |
| *** james_li has quit IRC | 18:20 | |
| lifeless | clarkb: so now you're talking about what we should do to make the impact on storage cleanup and so forth tolerable | 18:20 |
| johnthetubaguy | anteaya: I am over at Rackspace in the cloud servers team, nova blueprint czar and such like, but starting to help Bob out, and trying to find someone with more time than me to help more! | 18:20 |
| clarkb | lifeless: yup | 18:20 |
| lifeless | clarkb: I think we can do seed/mnt/nova-conductorlog and seed/root/nova-conductor.log | 18:20 |
| clarkb | something like that would be perfect thanks | 18:20 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Refine queries for tripleo jobs https://review.openstack.org/98096 | 18:20 |
| lifeless | clarkb: because we may have a name conflict between /var/log and /mnt/state/var/log | 18:21 |
| lifeless | we shouldn't but it can happy | 18:21 |
| lifeless | clarkb: ok, so we'll get right on that; do you want to wait scp wise for that to be in place, or can we land the scp patch to get Fedora jobs sorted ? | 18:21 |
| *** james_li has joined #openstack-infra | 18:22 | |
| clarkb | lifeless: I approved scp patch I think we are ok with having it in while you get sorted | 18:22 |
| lifeless | awesome | 18:23 |
| lifeless | pcrews_: around ? | 18:23 |
| *** rlandy has quit IRC | 18:23 | |
| pcrews_ | lifeless: si! what's up? | 18:23 |
| *** melwitt has quit IRC | 18:24 | |
| *** melwitt has joined #openstack-infra | 18:24 | |
| lifeless | pcrews_: tripleo + CI + thought you might like a pointer at something topical and gentle introduction | 18:24 |
| lifeless | I'm just filing a bug to capture state; gimme a sec | 18:25 |
| *** dangers is now known as dangers_away | 18:25 | |
| clarkb | lifeless: fwiw we had to put the inode heavy doc builds on a different filesystem hence my slight concern here | 18:25 |
| *** harlowja is now known as harlowja_away | 18:25 | |
| pcrews_ | lifeless: sounds great :) | 18:25 |
| *** dims_ has quit IRC | 18:25 | |
| lifeless | clarkb: yup. right now ubuntu is tarred up | 18:25 |
| lifeless | clarkb: so its when we stop tarring that we have to get it right | 18:26 |
| lifeless | jogo: around? stopping tarring *is* needed for logstashing, right ? | 18:26 |
| *** gokrokve_ has quit IRC | 18:26 | |
| clarkb | lifeless: jogo yes | 18:27 |
| lifeless | ok great | 18:27 |
| * lifeless finishes filing ze bug | 18:27 | |
| anteaya | johnthetubaguy: good to know your list, thanks, I hope you find that special someone, good luck in the search :D | 18:27 |
| johnthetubaguy | anteaya: hehe, thanks :) | 18:27 |
| phschwartz | are we currently logging info from instances that end up in ERROR state from node-pool? really best thing would be if we have logs that we can scrape to get uuid's | 18:28 |
| *** melwitt has quit IRC | 18:30 | |
| johnthetubaguy | phschwartz: are you looking at the rax-IAD errors that were happening before? (that were probably hurting the xenserver-ci too) or is this just a general thing? | 18:30 |
| phschwartz | rax-iad and rax-dfw errors | 18:30 |
| lifeless | clarkb: do we need to compress ourselves or will scp do that for us ? | 18:32 |
| *** ildikov has joined #openstack-infra | 18:33 | |
| *** jgrimm has joined #openstack-infra | 18:33 | |
| *** adrian_otto has quit IRC | 18:33 | |
| fungi | lifeless: compressing before copy helps. compression in-transit is a waste since we have a mop-up cron job on the logs.o.o server which tries to find and compress any which weren't compressed prior to collection | 18:34 |
| sarob | im getting fail on check-projects-yaml-alphabetized https://review.openstack.org/#/c/96334/ | 18:34 |
| sarob | any hints on how i can fix to pass? | 18:34 |
| *** nati_ueno has joined #openstack-infra | 18:35 | |
| *** rlandy has joined #openstack-infra | 18:35 | |
| mordred | phschwartz: yes, we are currently logging info | 18:35 |
| fungi | openstack/training-guides needs to be added to review.projects.yaml in alphabetic order with respect to the other entries in that file | 18:36 |
| fungi | sarob: ^ | 18:36 |
| phschwartz | mordred: is there any way I can get access to the logs to scrub uuid's for error state instances for say a 1 week periood? or would there be a way I could be provided them. | 18:36 |
| anteaya | fungi: I have tidied up the agenda for today: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting_.2810_Jun_2014.29 | 18:36 |
| clarkb | lifeless: you need to compress yourselves | 18:37 |
| anteaya | fungi: do you want me to keep last week's agenda or remove it | 18:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Keep the hierarchy of tripleo files copied to log server https://review.openstack.org/98043 | 18:37 |
| anteaya | fungi: we tend to remove them | 18:37 |
| fungi | sarob: between openstack/tempest and openstack/tripleo-heat-templates | 18:37 |
| mordred | phschwartz: sure. I can ship you a week of logs ... one sec | 18:37 |
| anteaya | fungi: I am assuming you are chairing | 18:37 |
| phschwartz | mordred: awesome, ty | 18:37 |
| fungi | anteaya: i usually remove entries i know don't need to be covered and leave topics which i think are being revisited | 18:37 |
| anteaya | fungi: okay I will leave for you to clean up then | 18:38 |
| lifeless | pcrews_: whats your LP uid ? | 18:38 |
| anteaya | I can remove dups though | 18:38 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/reviewstats: Oslo project updates https://review.openstack.org/99184 | 18:38 |
| mordred | phschwartz: don't thank me yet - you haven't seen the logs :) | 18:38 |
| pcrews_ | lifeless: patrick-crews | 18:38 |
| fungi | anteaya: oh, neat, when did we start separating last and next meeting agendas rather than just carrying forward? i clearly haven't been paying close attention | 18:38 |
| *** srenatus has quit IRC | 18:38 | |
| *** srenatus has joined #openstack-infra | 18:39 | |
| anteaya | reed did that | 18:39 |
| lifeless | ah, not a driver, can't assign to you | 18:39 |
| lifeless | pcrews_: grab https://bugs.launchpad.net/openstack-ci/+bug/1328645 :) | 18:39 |
| uvirtbot | Launchpad bug 1328645 in openstack-ci "ubuntu tripleo-ci jobs are not logstash indexable" [Undecided,New] | 18:39 |
| reed | I deny | 18:39 |
| reed | what? | 18:39 |
| anteaya | fungi: jeblair doesn't tend to, so I am cleaning them | 18:39 |
| phschwartz | mordred: I have scrapped web logs from a hosting companies control panels that services 290k customers with almost 195,000,000 requests per day. So don't worry, you won't scare me for that long ;) | 18:39 |
| anteaya | reed: just that every project agenda has a different style | 18:39 |
| lifeless | ok, I need to make C's lunch before the tripleo meeting, bbs | 18:39 |
| * mordred tries harder to scare phschwartz | 18:39 | |
| anteaya | reed: you went with a style qa uses | 18:39 |
| * phschwartz | 18:40 | |
| anteaya | reed: that's all, no problem, just figuring out what fungi wants to chair today's meeting | 18:40 |
| anteaya | fungi: did you want to talk about git-review this week? | 18:40 |
| reed | thanks for the explanation anteaya | 18:40 |
| pcrews_ | lifeless: 'tis grabbed | 18:40 |
| anteaya | ianw: more on fedora/centos7 this week? | 18:40 |
| clarkb | phschwartz: we do have a few billion things indexed in elasticsearch | 18:40 |
| anteaya | reed: np :D | 18:40 |
| clarkb | ~6 or so | 18:40 |
| fungi | anteaya: nah, i think we covered what we needed to on that | 18:41 |
| mordred | phschwartz: http://nodepool.openstack.org/nodepool-log-week.tgz | 18:41 |
| anteaya | fungi: very good | 18:41 |
| mordred | phschwartz: it's actually about 10 days | 18:41 |
| mordred | phschwartz: because the glob was easier | 18:41 |
| phschwartz | mordred: that would work :) | 18:41 |
| clarkb | sdague: on the indexing front adding 4 new hosts seems to have helped | 18:42 |
| anteaya | fungi: I'll stick ianw in there anyway and we can see what he wants to say | 18:42 |
| clarkb | sdague: I think based on the current numbers if we can just keep up we should stay caught up | 18:42 |
| mordred | phschwartz: btw - there is no leading dir on that tarball | 18:42 |
| clarkb | sdague: but I may need to add a few more nodes | 18:42 |
| mordred | phschwartz: so untar it carefully | 18:42 |
| clarkb | mordred: boo! | 18:42 |
| phschwartz | mordred: 310m compressed. not bad. | 18:42 |
| phschwartz | mordred: ty for the warning | 18:42 |
| anteaya | fungi: cleaned up: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 18:43 |
| fungi | anteaya: me too. hopefully we didn't step on one another's edits | 18:43 |
| anteaya | oh okay | 18:43 |
| anteaya | wiki didn't complain for me | 18:43 |
| anteaya | did I ruin yours | 18:44 |
| fungi | nor me. good work ;) | 18:44 |
| anteaya | yay | 18:44 |
| *** ilyashak_ has joined #openstack-infra | 18:44 | |
| mordred | clarkb, fungi: should we change the nodepool apache config so that it just serves all of /var/log/nodepool? I'm pretty sure those logs are safe/clean | 18:45 |
| *** ilyashak_ has quit IRC | 18:45 | |
| *** ilyashak_ has joined #openstack-infra | 18:45 | |
| *** ilyashak_ has quit IRC | 18:46 | |
| *** thedodd has joined #openstack-infra | 18:46 | |
| *** gokrokve has joined #openstack-infra | 18:46 | |
| fungi | mordred: is it possible for tracebacks to leak provider credentials from novaclient in our debug logs? | 18:48 |
| *** annegentle_ has quit IRC | 18:48 | |
| ekarlso | ls | 18:48 |
| fungi | ls: cannot open directory: Permission denied | 18:48 |
| *** pelix has quit IRC | 18:48 | |
| mordred | fungi: I do not believe so - I do not believe we've ever seen that | 18:50 |
| mordred | jogo: ^^ opinions? | 18:50 |
| clarkb | mtreinish: comment on 98162 | 18:50 |
| mtreinish | clarkb: oh I should abandon that we decided to keep the fail logs in the console just put them at the end instead of inline | 18:51 |
| jogo | mordred: tracebacks won't leak credentials since they don't dump variables. but the nova console lgos do | 18:51 |
| clarkb | mtreinish: that works thanks | 18:51 |
| jogo | err novaclient debug logs do | 18:51 |
| clarkb | sdague: in 97965 can we just echo "nprocs: $(nproc)" | 18:52 |
| mordred | jogo: I don't think we do novaclient debug in our debug logs | 18:52 |
| fungi | mordred: i guess the question is whether anyone is likely to accidentally switch novaclient into debug mode in production to track down a bug | 18:52 |
| clarkb | sdague: /proc/cpuinfo is fairly verbose with lots of irrelevant info that the hypervisor lies to us about | 18:52 |
| *** hashar has joined #openstack-infra | 18:52 | |
| *** olaph has joined #openstack-infra | 18:52 | |
| fungi | mordred: though if we don't capture it, then i guess that's safe | 18:52 |
| mordred | fungi: nod | 18:52 |
| jogo | sdague: ^ | 18:53 |
| *** e0ne has joined #openstack-infra | 18:55 | |
| *** pvo has quit IRC | 18:55 | |
| *** rlandy has quit IRC | 18:56 | |
| *** markmcclain has quit IRC | 18:56 | |
| *** rfolco has quit IRC | 18:57 | |
| *** dkliban is now known as dkliban_afk | 18:57 | |
| *** rfolco has joined #openstack-infra | 18:57 | |
| *** talluri has joined #openstack-infra | 18:57 | |
| *** nati_ueno has quit IRC | 18:57 | |
| *** marun is now known as marun_afk | 18:59 | |
| ianw | anteaya: no, not this week | 19:00 |
| fungi | meeting time... -> #openstack-meeting | 19:01 |
| *** spiffxp_ has quit IRC | 19:01 | |
| *** SlickN1k has quit IRC | 19:01 | |
| clarkb | jesusaurus: I reviewed 98561 | 19:02 |
| *** bknudson has quit IRC | 19:02 | |
| *** arnaud has joined #openstack-infra | 19:03 | |
| *** zz_gondoi is now known as gondoi | 19:03 | |
| *** harlowja_away is now known as harlowja | 19:03 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: ceilometer: enable gate-grenade-dsvm-forward https://review.openstack.org/97430 | 19:03 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: ceilometer: enable gate-grenade-dsvm-forward https://review.openstack.org/97430 | 19:04 |
| *** thedodd has quit IRC | 19:04 | |
| *** thedodd has joined #openstack-infra | 19:05 | |
| lifeless | clarkb: mordred says you're hitting a wall with dib | 19:06 |
| lifeless | clarkb: if you want to brainstorm (here, or voice or whatever, I'd be delighted to help) | 19:06 |
| clarkb | lifeless: yes the service things are wonky | 19:06 |
| clarkb | lifeless: can do after meeting and lunch | 19:06 |
| *** praneshp_ has joined #openstack-infra | 19:06 | |
| lifeless | clarkb: reach out to me whenever | 19:07 |
| *** sarob has quit IRC | 19:07 | |
| *** rdopieralski has joined #openstack-infra | 19:07 | |
| mordred | lifeless: tl;dr we need to run database commands at image build time. we're also worried that lennart, being lennart, is about to tie systemd versions to kernel versions which will make building modern fedora on ubuntu lts problematic | 19:07 |
| *** praneshp has quit IRC | 19:08 | |
| *** praneshp_ is now known as praneshp | 19:08 | |
| lifeless | my opinion of lennart is unprintable | 19:08 |
| lifeless | I saw that particular thread | 19:09 |
| openstackgerrit | A change was merged to openstack-infra/config: StoryBoard module bounces apache on update https://review.openstack.org/98512 | 19:09 |
| lifeless | Ubuntu had a similar issue with upstart in chroots at one point | 19:09 |
| lifeless | I can't believe fedora folk won't notice and lose their shit | 19:09 |
| *** nati_ueno has joined #openstack-infra | 19:09 | |
| *** dims_ has joined #openstack-infra | 19:10 | |
| *** marcoemorais has quit IRC | 19:11 | |
| *** marcoemorais has joined #openstack-infra | 19:11 | |
| *** marcoemorais has quit IRC | 19:12 | |
| clarkb | right I think there are some fundamental issues using lightweight separation of resources when doing cross platform image builds | 19:12 |
| clarkb | dib users that have a consistent platform they deploy on will have no issues | 19:12 |
| *** marcoemorais has joined #openstack-infra | 19:12 | |
| clarkb | but we have to support 2 ubuntu LTS versions, 2 centos versions, and at least one fedora version | 19:12 |
| lifeless | so we can add cross-platform 'gate' tests to tripleo-ci | 19:14 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Updated taskflow now that 0.3.x is released https://review.openstack.org/99188 | 19:14 |
| *** dangers_away is now known as dangers | 19:16 | |
| fungi | perhaps pottering wants to deprecate chroots | 19:16 |
| lifeless | pcrews_: ok so I'm back if you want to talk about that bugs details, or you can follow your nose and ask me/derekh/spamaps/etc if you get lost | 19:16 |
| lifeless | fungi: and containers... no | 19:16 |
| fungi | lifeless: heh, right | 19:16 |
| lifeless | I think their answer will be 'build on a latest kernel and its all fine' | 19:17 |
| lifeless | e.g. you can build old stuff on new, but not new stuff on old. | 19:17 |
| harlowja | qq, is it supported to do oslo development releases yet, taskflow just released 0.3.21 (prior 0.3 push failed) and would like to make it so that version numbering starts from 0.4.0.a0 (or similar), is all that is needed is that i tag the current master and the rest will happen automatically (with regards to versioning) | 19:17 |
| lifeless | I'm not endorsing this argument, just observing that it is the obvious solution when one side of an API cares about backwards compat and the other doesn't. | 19:17 |
| * fungi rather likes being able to develop more recent debian in chroots on stable debian, but maybe that's greybeardthink | 19:17 | |
| *** _nadya_ has quit IRC | 19:17 | |
| pcrews_ | lifeless: ack - been reading up on it and making sure I grok it. Have to tend to my 4-legged master quickly, but will likely have some questions when I dive back into things. | 19:18 |
| lifeless | :) | 19:18 |
| *** signed8bit has quit IRC | 19:18 | |
| fungi | harlowja: it should--and prereleases should now generate only wheels, so that older pip won't upgrade to them | 19:18 |
| harlowja | fungi cool | 19:19 |
| harlowja | so i'll try this, we have one patch going through gating, i guess once that merges, i can tag that commit with 0.4.0.a0 and then further patches will be in the 0.4.0a? series | 19:19 |
| lifeless | harlowja: btw see me reply on the pbr semver spec? | 19:20 |
| harlowja | lifeless saw it, ya its all cool, and understandable :) | 19:20 |
| harlowja | good to match with a use-case so u know it works out | 19:20 |
| harlowja | just after that gets goign i'll probably come in with rpm support :-P | 19:20 |
| harlowja | lifeless https://review.openstack.org/#/c/95234/ (for example) | 19:21 |
| harlowja | which is similar to yours ;) | 19:21 |
| harlowja | sound good? :) | 19:21 |
| lifeless | harlowja: I was thinking it would be good to get the rpm use case into the current spec | 19:22 |
| *** ociuhandu has joined #openstack-infra | 19:22 | |
| harlowja | hmmm | 19:22 |
| lifeless | harlowja: if you can give the high level description .... | 19:22 |
| harlowja | lifeless sure | 19:22 |
| *** amotoki has quit IRC | 19:22 | |
| phschwartz | mordred: would you mind if I opened a review against node-pool and changed the format of the logging for LaunchStatusException? Right now it spits out uuid, node id, and status. Would make tracing easier if it also had the provider. | 19:22 |
| *** duran has quit IRC | 19:22 | |
| *** signed8bit has joined #openstack-infra | 19:22 | |
| harlowja | lifeless let me write something up, shall i email it to u? or put a diff somewhere | 19:22 |
| *** gyee has quit IRC | 19:23 | |
| lifeless | put it as a review comment in the spec, where you'd like it added, and I'll incorporate it when I tweak for dhellmann's latest feedback | 19:23 |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Make 'jenkins-job-builder' repo configurable https://review.openstack.org/99190 | 19:23 |
| harlowja | lifeless k | 19:23 |
| lifeless | fungi: you're such a greybeard :P | 19:25 |
| *** afazekas has quit IRC | 19:26 | |
| *** sarob has joined #openstack-infra | 19:27 | |
| *** sweston has joined #openstack-infra | 19:27 | |
| *** sweston has quit IRC | 19:30 | |
| *** Sukhdev has joined #openstack-infra | 19:31 | |
| *** amcrn has joined #openstack-infra | 19:31 | |
| fungi | lifeless: i'm staving off grey hairs, but not so good at growing beards unfortunately | 19:32 |
| *** pdmars_ has joined #openstack-infra | 19:33 | |
| *** ramashri has quit IRC | 19:35 | |
| *** pdmars has quit IRC | 19:35 | |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Make 'jenkins-job-builder' repo configurable https://review.openstack.org/99190 | 19:36 |
| *** arnaud has quit IRC | 19:38 | |
| *** Sukhdev has quit IRC | 19:40 | |
| *** bknudson has joined #openstack-infra | 19:40 | |
| *** sarob has quit IRC | 19:42 | |
| openstackgerrit | A change was merged to openstack-infra/config: Added 'docs' job for manila project https://review.openstack.org/98465 | 19:43 |
| *** talluri has quit IRC | 19:44 | |
| openstackgerrit | A change was merged to openstack-infra/config: except ImportError when importing cPickle on Python3 https://review.openstack.org/98367 | 19:46 |
| *** thedodd has quit IRC | 19:47 | |
| jerryz | lifeless: see my comments on your spec. both nodepool and jenkins are working with jump host now | 19:48 |
| lifeless | dhellmann: avoiding roundtrips - I've replied to your comment in the spec | 19:48 |
| lifeless | jerryz: awesome! | 19:48 |
| *** doude has quit IRC | 19:48 | |
| *** thedodd has joined #openstack-infra | 19:50 | |
| *** gondoi is now known as zz_gondoi | 19:50 | |
| openstackgerrit | Anne Gentle proposed a change to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 19:51 |
| *** zz_gondoi is now known as gondoi | 19:51 | |
| *** e0ne has quit IRC | 19:51 | |
| lifeless | jerryz: I will review later today | 19:52 |
| *** e0ne has joined #openstack-infra | 19:52 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use the correct name for apache https://review.openstack.org/97994 | 19:52 |
| lifeless | cody-somerville: / mordred: you may find the no-public-ip spec stuff from jerryz and me interesting too | 19:52 |
| *** signed8bit has quit IRC | 19:53 | |
| *** signed8bit has joined #openstack-infra | 19:54 | |
| *** marun_afk is now known as marun | 19:55 | |
| jesusaurus | lifeless: jerryz: is there a plan for how to segregate ip space for the jump hosts? | 19:56 |
| *** e0ne has quit IRC | 19:57 | |
| mordred | phschwartz: sounds good to me - clarkb, fungi ^^ see phschwartz above - any thoughts? | 19:57 |
| *** marcoemorais has quit IRC | 19:57 | |
| *** marcoemorais has joined #openstack-infra | 19:58 | |
| lifeless | jesusaurus: what do you mean? | 19:58 |
| dhellmann | lifeless: ack, I'll read up after the TC & release meetings | 19:58 |
| jesusaurus | lifeless: managing a single pool of slaves behind one jump host is easy, but what happens when a second provider is added that also wants to use a jump host? | 19:59 |
| clarkb | mordred: phschwartz shouldn't be a problem | 20:00 |
| clarkb | it isn't hard to grep for that info though you just have to do a "join" | 20:00 |
| jesusaurus | lifeless: the best way i can see to manage that is to have one ProxyCommand under Host 10.0.* and another under 10.1.* | 20:00 |
| *** rdopieralski has quit IRC | 20:00 | |
| jesusaurus | lifeless: which means that we should have some plan for how to allocate 10.0.* and 10.1.* | 20:00 |
| *** jaypipes has joined #openstack-infra | 20:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update launch script and instructions https://review.openstack.org/97841 | 20:01 |
| lifeless | jesusaurus: separate jump host | 20:01 |
| clarkb | anteaya: fungi: ok that isn't clear from the etherpad | 20:01 |
| anteaya | yes, well I was hoping I would have more time | 20:02 |
| lifeless | jesusaurus: I'm not sure why the IP range matters at all | 20:02 |
| anteaya | but reed's topics were important | 20:02 |
| lifeless | jesusaurus: can you explain ? | 20:02 |
| clarkb | anteaya: gotcha | 20:02 |
| clarkb | anteaya: in any case I am happy with the guidelines as long as we don't require existing accounts to change usernames | 20:02 |
| *** signed8b_ has joined #openstack-infra | 20:02 | |
| *** james_li has quit IRC | 20:02 | |
| *** markmcclain has joined #openstack-infra | 20:02 | |
| anteaya | clarkb: so as it stands I am hoping we can change the full name as fungi said and have new usernames and full names match | 20:02 |
| anteaya | clarkb: kk, thanks | 20:03 |
| fungi | phschwartz: sounds good to me. we've added it other places | 20:03 |
| anteaya | I am also hoping to convince fungi to let me set up a list of people willing to voluntarily change their usernames | 20:03 |
| *** signed8bit has quit IRC | 20:03 | |
| anteaya | but that is new info | 20:03 |
| clarkb | anteaya: that would be fine as well | 20:03 |
| anteaya | and I might be crushed by fires | 20:03 |
| clarkb | anteaya: as we should avoid surprises and confusion goign that route | 20:03 |
| anteaya | since that might require a window with no fires | 20:04 |
| clarkb | btu ya you might not get any volunteers | 20:04 |
| fungi | yeah, it's just that their ability to use the account will be broken until they update their systems to match what they're changed to in gerrit | 20:04 |
| clarkb | yup | 20:04 |
| anteaya | right | 20:04 |
| anteaya | so I'm okay to ask for volunteers for that and not force anyone? | 20:04 |
| anteaya | or worry about folks I can't find | 20:04 |
| jesusaurus | lifeless: if you want to use ProxyCommand in .ssh/config to transparently configure the jump host, then the ProxyCommand will either apply to all ssh connections, or will be tied to a Host in .ssh/config and if you tie the jump host to a Host entry, then you either have a unique entry for every target host that goes through a jump host (and nodepool would need a way to dynamically modify that), you | 20:05 |
| jesusaurus | can * against a dns entry, or you can * against an IP range | 20:05 |
| *** changbl has joined #openstack-infra | 20:06 | |
| clarkb | we will probably do something like zuuls ssh command wrapper (I haven't looked at any actual implementations though) | 20:06 |
| jesusaurus | lifeless: or am I completely confused about what you are trying to do with .ssh/config? | 20:06 |
| clarkb | ok I need to grab lunch now | 20:07 |
| jesusaurus | clarkb: how is zuul's ssh wrapper configured? | 20:07 |
| clarkb | back soonish to talk about dib more and things | 20:07 |
| clarkb | jesusaurus: its a bit git specific but it is just a shell script zuul spits out so that git + ssh does the correct thing | 20:07 |
| *** e0ne has joined #openstack-infra | 20:08 | |
| clarkb | but you could probably do something similar with ssh itself | 20:08 |
| jesusaurus | huh, i'll take a look at that soonish | 20:08 |
| clarkb | where ssh-wrapper is actually ssh -o ProxyCommand | 20:08 |
| *** e0ne has quit IRC | 20:10 | |
| sdague | mordred: we definitely do neutronclient | 20:10 |
| *** e0ne has joined #openstack-infra | 20:10 | |
| *** pcrews_ has quit IRC | 20:11 | |
| *** nati_ueno has quit IRC | 20:11 | |
| lifeless | jesusaurus: you can match on the hostname | 20:11 |
| mordred | sdague: ENOCONTEXT | 20:11 |
| *** pcrews has joined #openstack-infra | 20:11 | |
| lifeless | jesusaurus: rather than sshing to the IP | 20:11 |
| lifeless | jesusaurus: we can map the provider | 20:11 |
| *** pcrews_ has joined #openstack-infra | 20:11 | |
| lifeless | *.cloudname.local | 20:12 |
| jesusaurus | oh, right, it doesnt need to be a dns entry | 20:12 |
| jesusaurus | derp | 20:12 |
| *** pcrews has quit IRC | 20:12 | |
| clarkb | mordred: I think logs for nodepool | 20:13 |
| clarkb | ok really need lunch now | 20:13 |
| mordred | "have gotten it to work by parsing nodepool.yaml, reading parameters like proxyhost, proxyuser, proxykey and joining them into a proxycommand used by paramiko.ProxyCommand." | 20:13 |
| mordred | ^^ this == yes | 20:13 |
| *** e0ne has quit IRC | 20:13 | |
| * mordred does not want this directly in a .ssh/config file - but if it's in nodepool.yaml and we can tell paramiko to do the right hting, then AWESOME | 20:13 | |
| mordred | so, basically, jerryz++ | 20:14 |
| *** duran has joined #openstack-infra | 20:15 | |
| *** denis_makogon has joined #openstack-infra | 20:15 | |
| sdague | clarkb: so on the cpu thing, I'd kind of like to understand the differences in runs, even if they are lying to us | 20:16 |
| *** homeless has quit IRC | 20:16 | |
| sdague | because those lies will impact the performance of c libs | 20:16 |
| sdague | and it's information we can't ever get again if we don't dump it with the run | 20:17 |
| sdague | especially when we see one node actually time out | 20:17 |
| *** homeless has joined #openstack-infra | 20:17 | |
| sdague | like take > 2hrs on a tempest run without errors | 20:17 |
| reed | is there an API call to figure out which email addresses have signed the iCLA on gerrit? | 20:17 |
| fungi | reed: yes, i can get it for you after the tc and project meetings, unless someone else wants to dig up the member query api call and the icla group name | 20:18 |
| *** homeless has quit IRC | 20:19 | |
| *** nati_ueno has joined #openstack-infra | 20:19 | |
| harlowja | lifeless pbr-versioning comments added | 20:19 |
| reed | fungi, i found the Group UUID, if that's all is needed then it's easy :) | 20:19 |
| *** miqui has quit IRC | 20:20 | |
| *** homeless has joined #openstack-infra | 20:20 | |
| fungi | reed: you can query by group name too, pretty sure | 20:20 |
| harlowja | dprince yt, u might/probably are interested in https://review.openstack.org/#/c/96608/ (be nice to have a better rpm expert than me there, haha) | 20:20 |
| reed | fungi, indeed, I'm just failing at authenticating via curl right now :) | 20:20 |
| *** e0ne has joined #openstack-infra | 20:21 | |
| fungi | reed: yeah, there is also the ssh api... https://review.openstack.org/Documentation/cmd-ls-members.html#_examples | 20:22 |
| fungi | reed: i haven't played around with the rest api much yet | 20:22 |
| reed | fungi, thanks | 20:22 |
| fungi | looks like the rest api may want you to first query for the group uuid and then use it to query for the member list after all | 20:23 |
| fungi | reed: to authenticate to the rest api, you need to go to your account settings in gerrit and click a button to generate a reusable http password, which you can then pass for authentication in your calls | 20:23 |
| *** afazekas has joined #openstack-infra | 20:24 | |
| *** gondoi is now known as zz_gondoi | 20:25 | |
| *** dmsimard has joined #openstack-infra | 20:26 | |
| *** zz_gondoi is now known as gondoi | 20:27 | |
| *** arnaud has joined #openstack-infra | 20:27 | |
| *** devkulkarni has joined #openstack-infra | 20:31 | |
| *** arunkant has quit IRC | 20:33 | |
| *** kmartin has quit IRC | 20:33 | |
| *** AaronGr has quit IRC | 20:33 | |
| *** nijaba has quit IRC | 20:33 | |
| *** plars-away has quit IRC | 20:33 | |
| *** jeremyb has quit IRC | 20:33 | |
| *** pfalleno1 has quit IRC | 20:33 | |
| *** BadCub has quit IRC | 20:33 | |
| *** enikanorov_ has quit IRC | 20:33 | |
| *** esmute has quit IRC | 20:33 | |
| *** dguerri has quit IRC | 20:33 | |
| *** phschwartz has quit IRC | 20:33 | |
| *** timrc-afk has quit IRC | 20:33 | |
| *** ogelbukh has quit IRC | 20:33 | |
| *** ryanpetrello has quit IRC | 20:33 | |
| *** Sukhdev has joined #openstack-infra | 20:33 | |
| *** esmute has joined #openstack-infra | 20:33 | |
| *** pfallenop has joined #openstack-infra | 20:33 | |
| *** dguerri has joined #openstack-infra | 20:33 | |
| *** BadCub has joined #openstack-infra | 20:33 | |
| *** phschwartz has joined #openstack-infra | 20:33 | |
| *** jeremyb has joined #openstack-infra | 20:33 | |
| *** kmartin has joined #openstack-infra | 20:33 | |
| *** kmartin has quit IRC | 20:33 | |
| *** kmartin has joined #openstack-infra | 20:33 | |
| *** arunkant has joined #openstack-infra | 20:33 | |
| *** timrc-afk has joined #openstack-infra | 20:33 | |
| *** ryanpetrello has joined #openstack-infra | 20:33 | |
| *** AaronGr has joined #openstack-infra | 20:33 | |
| *** timrc-afk is now known as timrc | 20:33 | |
| *** melwitt has joined #openstack-infra | 20:34 | |
| *** nijaba has joined #openstack-infra | 20:34 | |
| *** ogelbukh has joined #openstack-infra | 20:34 | |
| *** devkulkarni has quit IRC | 20:35 | |
| *** devkulkarni has joined #openstack-infra | 20:35 | |
| *** plars has joined #openstack-infra | 20:35 | |
| lifeless | harlowja: so in rpm | 20:37 |
| *** dhellman_ has joined #openstack-infra | 20:37 | |
| lifeless | harlowja: how do you sort something *before* a release ? | 20:37 |
| *** thuc has joined #openstack-infra | 20:37 | |
| *** sarob has joined #openstack-infra | 20:38 | |
| dprince | lifeless: latest release number plus the git commit number in the release | 20:38 |
| lifeless | harlowja: ~ says 'sort before' - | 20:38 |
| lifeless | dprince: that sorts before ? | 20:38 |
| *** dmsimard has left #openstack-infra | 20:38 | |
| lifeless | e.g. let me give a concrete example, we have three versions that should sort A < B < C | 20:39 |
| lifeless | A = 1.2.0 'dev3' | 20:39 |
| lifeless | B = 1.2.0 'dev4' | 20:39 |
| *** gilliard_ has joined #openstack-infra | 20:39 | |
| lifeless | C = 1.2.0 | 20:39 |
| *** plomakin_ has joined #openstack-infra | 20:39 | |
| *** doude has joined #openstack-infra | 20:41 | |
| *** dhellmann_ has joined #openstack-infra | 20:41 | |
| *** zns has quit IRC | 20:42 | |
| *** sarob has quit IRC | 20:43 | |
| fungi | ~ was still a debianism last i recall | 20:43 |
| mestery | Infra folks: I've surveyed the current state of Neutron 3rd party CI systems, and needless to say, it's not very good. | 20:43 |
| phschwartz | The error state builds in graphite contain in the count errors that are not related to the actual instance building, but the bootstrap of it also correct? | 20:43 |
| mestery | anteaya and I are wokring on making this better. | 20:43 |
| dprince | lifeless: Yeah I suppose those would need to be special case. Our pre-release tagging is sort not good in this regard | 20:44 |
| mestery | I have an ehterpad which lists the current state of CI systems for Neutron here: https://etherpad.openstack.org/p/ZLp9Ow3tNq | 20:44 |
| mestery | And here is how these will get better: https://wiki.openstack.org/wiki/NeutronThirdPartyTesting | 20:44 |
| mestery | FYI, as once I send an email with deadlines to fix this, folks may bounce in here for help. | 20:44 |
| fungi | lifeless: what i've seen for rpm is backversioning like 1.1.99really1.2.0dev3 | 20:44 |
| mestery | I'll be around to help as well, but wanted to share this info with eveyrone here. | 20:44 |
| *** HenryG has quit IRC | 20:45 | |
| lifeless | dprince: we're trying to design it in the pbr semver spec - see oslo-specs; I added you as a reviewer last week:) | 20:45 |
| dprince | lifeless: I see that now, will have a look | 20:46 |
| mordred | phschwartz: that's right. "error" for us means anything that did not result in an instance that works | 20:46 |
| mordred | phschwartz: we do most of our bootstrap in image build - so there isn't a ton post-boot | 20:47 |
| fungi | it's useful to distinguish between "nodepool error state" and "nova error state" there | 20:47 |
| mordred | phschwartz: but we do go in and check for things like outbound network connectivity working etc | 20:47 |
| anteaya | mestery: let's ensure there is a link to http://ci.openstack.org/third_party.html in the email that goes out | 20:47 |
| mestery | anteaya: OK, will do, and thanks! | 20:48 |
| anteaya | thanks | 20:48 |
| *** dhellmann_ has quit IRC | 20:48 | |
| fungi | nova error state gets spotted by nodepool right away, while something like not eventually becoming reachable takes a while for nodepool to figure out and give up | 20:48 |
| *** dhellmann_ has joined #openstack-infra | 20:48 | |
| *** sabari has joined #openstack-infra | 20:48 | |
| *** gyee has joined #openstack-infra | 20:49 | |
| phschwartz | mordred: I will have to look more at the logs to see if I can scrap that info out of them as well, but so far I have all the instances that threw the exception of LaunchStatusException with the instance in ERROR state. | 20:49 |
| phschwartz | ~/src/openstack/logs ? cat rax-iad | wc -l | 20:49 |
| phschwartz | 72230 | 20:49 |
| fungi | though we *have* i believe also seen nodes transition from building to error to ready in the past as well | 20:49 |
| phschwartz | ouch is all I can say. | 20:49 |
| mordred | phschwartz: wow | 20:49 |
| phschwartz | That is in the 9 days of logs you gave me | 20:49 |
| mordred | phschwartz: so, one of the things I was noting to pvo earlier | 20:49 |
| *** sballe has joined #openstack-infra | 20:49 | |
| mordred | is that if a cloud is erroring, we actually wind up REALLY hammering it | 20:49 |
| mordred | because we don't get a node, so we ask for a new one quickly | 20:49 |
| mordred | it made me think that we should put in a backoff threshold | 20:50 |
| phschwartz | exactly and since I am working towards using our public cloud for our internal infra setup, it is defn a worry | 20:50 |
| mordred | like, if 50% of the requests in the last X period of time are errors, chill out for a bit with that cloud | 20:50 |
| anteaya | mestery: can we also have a link to the third-party meeting as well? | 20:50 |
| mordred | because the increased requests could actually be making it _harder_ to get a good node | 20:50 |
| mordred | fungi, clarkb ^^ | 20:50 |
| lifeless | dprince: so if using .99 is the right way in RPM world, we can do that | 20:51 |
| fungi | mordred: tcp backoff | 20:51 |
| *** gyee has quit IRC | 20:51 | |
| lifeless | dprince: my goal isn't to be perfect, its to meet the broad requirements at the top in a systematic fashion | 20:51 |
| *** thuc has quit IRC | 20:51 | |
| fungi | mordred: it's a good algorithm to reuse there i think, and we can tweak the scaling on it as we see fit | 20:51 |
| *** thuc has joined #openstack-infra | 20:52 | |
| fungi | something along the lines of truncated binary exponential backoff maybe | 20:53 |
| mordred | fungi: ++ | 20:53 |
| *** gyee has joined #openstack-infra | 20:53 | |
| *** bknudson has quit IRC | 20:54 | |
| *** amcrn has quit IRC | 20:54 | |
| *** olaph has quit IRC | 20:54 | |
| *** mmaglana has quit IRC | 20:54 | |
| *** dhellmann has quit IRC | 20:54 | |
| *** bogdando has quit IRC | 20:54 | |
| *** achuprin has quit IRC | 20:54 | |
| *** michchap has quit IRC | 20:54 | |
| *** masayukig has quit IRC | 20:54 | |
| *** bradm has quit IRC | 20:54 | |
| *** mattymo has quit IRC | 20:54 | |
| *** plomakin has quit IRC | 20:54 | |
| *** openstackgerrit has quit IRC | 20:54 | |
| *** gilliard has quit IRC | 20:54 | |
| *** dhellmann_ is now known as dhellmann | 20:54 | |
| fungi | what i just heard is that phschwartz just volunteered to write the patch because we're hammering his cloud ;) | 20:54 |
| *** e0ne has quit IRC | 20:54 | |
| *** adrian_otto has joined #openstack-infra | 20:55 | |
| mordred | fungi: :) | 20:55 |
| phschwartz | fungi: I can do that. Shouldn't be that hard of an algorithm exponentially increase the wait time to launch more instances till it is actually passing. Key would be the ratio of wait to retry. | 20:55 |
| fungi | yeah, i expect there will be some tuning | 20:55 |
| *** olaph has joined #openstack-infra | 20:56 | |
| fungi | but it will probably also help make nodepool more sane too, because it goes into some real fits when, for example, quotas cause its requests to get rejected | 20:56 |
| *** nati_ueno has quit IRC | 20:56 | |
| *** dhellmann has quit IRC | 20:56 | |
| *** bogdando has joined #openstack-infra | 20:56 | |
| fungi | and it starts rapidly looping them even though there's no hope | 20:56 |
| *** dhellman_ is now known as dhellmann | 20:56 | |
| *** thuc has quit IRC | 20:56 | |
| phschwartz | I have noticed that it falls over bad if it gets pigeon holed into a cell and that cell is over commited. | 20:57 |
| *** mattymo has joined #openstack-infra | 20:58 | |
| mestery | anteaya: I will add that link as well, good idea! | 20:59 |
| anteaya | mestery: thanks | 20:59 |
| *** enikanorov_ has joined #openstack-infra | 20:59 | |
| fungi | phschwartz: yeah, better distribution from the provider's scheduler would be preferable i think | 20:59 |
| *** ArxCruz has quit IRC | 20:59 | |
| *** e0ne has joined #openstack-infra | 21:00 | |
| mordred | fungi, clarkb ^^ SO - it seems rax-iad has neutron now | 21:00 |
| mordred | so some of the thigns we did with hp 1.1 may need doing with rax-iad | 21:01 |
| mordred | do we remember them all? | 21:01 |
| fungi | neat! | 21:01 |
| mordred | one of the issues was hitting fixed ip limits, yeah? | 21:01 |
| fungi | i believe so | 21:02 |
| fungi | basically creating separate networks so that we don't approach 256 nodes in a /24 | 21:02 |
| mordred | phschwartz: you can see here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n347 | 21:02 |
| *** gondoi is now known as zz_gondoi | 21:02 | |
| mordred | phschwartz: that we're tying a config to a particular network on our side | 21:03 |
| *** pdmars_ has quit IRC | 21:03 | |
| fungi | but part of that may have had to do with hpcloud's default choices | 21:03 |
| mordred | and we made five networks within the b region there | 21:03 |
| *** zz_gondoi is now known as gondoi | 21:03 | |
| *** adrian_otto has quit IRC | 21:03 | |
| *** adrian_otto1 has joined #openstack-infra | 21:03 | |
| mordred | fungi: it may do - but if we're seeing similar rampant failures in iad, then applying similar us-side mitigations might be friendly | 21:03 |
| fungi | mordred: sure enough | 21:04 |
| *** gondoi is now known as zz_gondoi | 21:04 | |
| *** nati_ueno has joined #openstack-infra | 21:05 | |
| bodepd | do you guys use heat templates to deploy out infrastrcuture? | 21:05 |
| *** CaptTofu_ has quit IRC | 21:06 | |
| *** mwagner_lap has joined #openstack-infra | 21:06 | |
| *** hashar has quit IRC | 21:06 | |
| fungi | bodepd: i think we want to, but we need standalone heat | 21:07 |
| bodepd | fungi: why? | 21:07 |
| fungi | bodepd: or rather, it's been suggested that we may want to eat that dogfood if we're hungry | 21:07 |
| *** CaptTofu_ has joined #openstack-infra | 21:07 | |
| bodepd | fungi: oh cool, I didn't know what existined | 21:08 |
| * bodepd researches | 21:08 | |
| *** bknudson has joined #openstack-infra | 21:08 | |
| *** amcrn has joined #openstack-infra | 21:08 | |
| *** mmaglana has joined #openstack-infra | 21:08 | |
| *** achuprin has joined #openstack-infra | 21:08 | |
| *** michchap has joined #openstack-infra | 21:08 | |
| *** masayukig has joined #openstack-infra | 21:08 | |
| *** bradm has joined #openstack-infra | 21:08 | |
| *** openstackgerrit has joined #openstack-infra | 21:08 | |
| fungi | bodepd: why would we use heat templates, or why would we need standalone heat? | 21:08 |
| phschwartz | mordred: actually, I am not sure at all that it is neutron causing the issue. | 21:08 |
| phschwartz | Neutron is active in all our regions | 21:08 |
| bodepd | fungi: let me read a bit. I was more of the 'what the heck is it' | 21:09 |
| fungi | bodepd: the standalone functionality need stems from the providers donating resources to us not offering heat services | 21:09 |
| mordred | phschwartz: oh - neat | 21:09 |
| phschwartz | and less then 300 error state in 9 days between ord and dfw | 21:09 |
| *** marcoemorais has quit IRC | 21:09 | |
| mordred | bodepd: we don't right now, even with standalone heat, for a couple of reasons | 21:09 |
| *** marcoemorais has joined #openstack-infra | 21:09 | |
| bodepd | fungi: my heat research is based on the fact that ansible is gpl :) | 21:09 |
| *** adrian_otto has joined #openstack-infra | 21:09 | |
| fungi | heh | 21:09 |
| *** zz_gondoi is now known as gondoi | 21:10 | |
| *** adrian_otto1 has quit IRC | 21:10 | |
| mordred | bodepd: one is multi-cloud -although our main infra control plane is ina single region, so that's not a rEAL problem | 21:10 |
| mordred | but the second is that heat's current response to failure is to just give up | 21:10 |
| bodepd | mordred: gotcha. | 21:10 |
| mordred | there are folks working on that | 21:10 |
| bodepd | ouch | 21:10 |
| mordred | but it's not ready for us yet | 21:10 |
| *** rfolco has quit IRC | 21:10 | |
| mordred | bodepd: is the GPL part giving you problems somewhere? | 21:10 |
| phschwartz | brb, heading to the store | 21:10 |
| bodepd | mordred: it just doesn't align well with contract terms present or future | 21:11 |
| mordred | bodepd: hrm. the fact that ansible is gpl shouldn't really matter if you're not forking it | 21:12 |
| mordred | bodepd: ansible $things won't be gpl themselves | 21:12 |
| mordred | but, you know - I'd LOVE to use heat as the thing that deploys our things | 21:12 |
| *** adrian_otto1 has joined #openstack-infra | 21:12 | |
| *** sweston has joined #openstack-infra | 21:12 | |
| mordred | once it can handle our brand of crazy | 21:13 |
| bodepd | people are just gpl wary for toolchains. It's not all based on logic. | 21:13 |
| mordred | bodepd: they should go look at the license of ls | 21:13 |
| *** doude has quit IRC | 21:13 | |
| mordred | actually, tbh - given our current rate of adding new control plane services, and now taht heat has "adopt-stack" it probably is worthwhile looking at | 21:13 |
| mordred | if we update the stack and it doesn't work, it _does_ just stop, which is what we'd want it to do | 21:14 |
| bodepd | I just need a way to make the base infra self building | 21:14 |
| mordred | and then we can go remedy | 21:14 |
| mordred | bodepd: tell me about it | 21:14 |
| *** adrian_otto has quit IRC | 21:14 | |
| *** doude has joined #openstack-infra | 21:14 | |
| bodepd | I have to choose tools,make them work, etc... :) | 21:14 |
| *** shivharis has joined #openstack-infra | 21:15 | |
| *** nati_ueno has quit IRC | 21:15 | |
| shivharis | fungi: ping | 21:15 |
| fungi | shivharis: what's up? | 21:16 |
| shivharis | fungi: hi how are you? | 21:16 |
| shivharis | fungi:how do i look up an account on review.openstack.org if I know the username | 21:17 |
| shivharis | e.g. brocade_jenkins | 21:17 |
| *** dprince has quit IRC | 21:17 | |
| clarkb | ok back from lunch | 21:17 |
| reed | fungi, I believe the list of members of ICLA group is only visible to admins | 21:17 |
| reed | fungi, I'm talking about gerrit API | 21:17 |
| clarkb | mordred: yes we need to bump a ton of quotas (this is arguably why I find neutron in public cloud by default a bit annoying. It allows me to do amazing things but with a very high overhead when I just want node + public IP) | 21:18 |
| fungi | reed: oh, you may be right. i think i need to add an account to the owners group for it to enable that, but we can add that fairly easily | 21:18 |
| clarkb | mordred: specifically we need to bump network quota, subnet quota, fixed IP quota, floating ip quota, and router quota | 21:18 |
| mordred | phschwartz: ^^ | 21:18 |
| mordred | clarkb: it's possible that the rax folks took care of that already | 21:18 |
| fungi | shivharis: what do you mean by "look up an account"? | 21:19 |
| fungi | shivharis: what do you need to know about the account? | 21:19 |
| reed | fungi, let me know | 21:19 |
| shivharis | i wanted to change the name of the account, not the username | 21:19 |
| clarkb | sdague: ok I can live with that | 21:20 |
| shivharis | fungi: possible? | 21:20 |
| fungi | shivharis: i believe we have to do that manually. if this is in response to anteaya's plan for renaming third-party ci accounts consistently, that's going to be done for you. you won't need to do anything | 21:20 |
| clarkb | mordred: fungi: agreed a tcp like back off would probably work well | 21:20 |
| *** markmc has quit IRC | 21:20 | |
| clarkb | mordred: fungi we can do that simply be tweaking the rate factor in a manager itself | 21:20 |
| shivharis | fungi: yes it is in prep for that. I'll wait as u suggest | 21:21 |
| clarkb | mordred: fungi: phschwartz I think that change should be relatively straightforward since all of that code is failry well self contained | 21:21 |
| clarkb | lifeless: is now still a good time to talk dib? | 21:22 |
| lifeless | sure | 21:22 |
| lifeless | let me wrap up this OSRB call first | 21:22 |
| shivharis | fungi: sorry one more question. How can I trigger a build with updating a patch. I mean not just 'jenkins' but all CI? | 21:22 |
| shivharis | fungi: or a specific CI | 21:23 |
| shivharis | with->without | 21:23 |
| *** alkari has quit IRC | 21:23 | |
| *** gondoi is now known as zz_gondoi | 21:23 | |
| clarkb | lifeless: I think we have a few options available to us. We can use dib, boot resulting image in cloud, perform DB and other state steps, then snapshot that. We can just perform post boot in all VMs we boot. We can dump puppet for this stuff and do it all in native dib. Or we can continue to use nodepool to build images as it does today | 21:24 |
| clarkb | we would like to be able to build images once and use them in all clouds. we want to be able to provide $notinfrapeople a way to build their own images. and I think we want a better story for CI of the images themselves | 21:25 |
| clarkb | I would put those goals as the top 3. mordred did I miss anything? | 21:25 |
| *** adrian_otto1 is now known as adrian_otto | 21:25 | |
| *** andreykurilin_ has quit IRC | 21:25 | |
| *** jaypipes has quit IRC | 21:26 | |
| anteaya | shivharis: you don't have to wait for anything, you don't have to do anything | 21:27 |
| fungi | shivharis: i don't believe we currently have any enforced standard behavior there, so the comments needed to recheck various systems' votes currently varies i think | 21:27 |
| *** zz_gondoi is now known as gondoi | 21:27 | |
| anteaya | shivharis: and by the way you and sweston need to spend some time talking with each other | 21:27 |
| harlowja | lifeless so u can modify the release number to use dotted components, for example | 21:27 |
| *** e0ne has quit IRC | 21:27 | |
| harlowja | nled-2.52-6.fc13.x86_64.rpm is newer than nled-2.52-6.fc12.x86_64.rpm (the release number here is 6.fc12 and 6.fc13 accordingly) | 21:28 |
| lifeless | harlowja: I have a sketch coming in a sec | 21:28 |
| russellb | so, commit message updated on a patch, and it kept my +1 ... expected behavior? | 21:28 |
| lifeless | harlowja: using 9999 | 21:28 |
| *** changbl has quit IRC | 21:28 | |
| sweston | anteaya: shivharis hello | 21:28 |
| harlowja | lifeless k | 21:28 |
| shivharis | sweston: hi | 21:28 |
| anteaya | sweston: so yeah, you and shivharis need to sit down and have a chat | 21:28 |
| sweston | shivharis: We have a meeting to discuss the details tomorrow. I will include you on the meeting invite. | 21:28 |
| shivharis | sweston: thx | 21:29 |
| *** jaypipes has joined #openstack-infra | 21:29 | |
| sweston | shivharis: :-) | 21:29 |
| *** changbl has joined #openstack-infra | 21:29 | |
| clarkb | lifeless: the first and last options fail at the simple for $notinfraperson to make image. the second and third options better fit the goals but with a much higher labor or cloud resource cost | 21:29 |
| clarkb | so looking for alternatives that might better meet goals | 21:29 |
| *** adalbas has quit IRC | 21:29 | |
| *** devkulkarni has quit IRC | 21:31 | |
| *** pdmars has joined #openstack-infra | 21:31 | |
| *** amcrn has quit IRC | 21:31 | |
| *** mfer has quit IRC | 21:32 | |
| *** flaper87 is now known as flaper87|afk | 21:32 | |
| anteaya | so tomorrow I am away most of the day running errands | 21:33 |
| anteaya | and on Thursday I am voting, so gone a chunk of the day | 21:33 |
| anteaya | most likely morning | 21:33 |
| *** gondoi is now known as zz_gondoi | 21:34 | |
| *** nati_ueno has joined #openstack-infra | 21:35 | |
| fungi | oh, i meant to mention it at the end of the meeting today but we were pressed for time as usual... i'll be offline/unreachable for long periods next week while i'm moving, and will probably miss next tuesday's meeting, but jeblair should be back by then | 21:36 |
| *** pdmars has quit IRC | 21:36 | |
| clarkb | ooh moving | 21:37 |
| clarkb | have fun | 21:37 |
| *** jaypipes has quit IRC | 21:37 | |
| lifeless | harlowja: dhellmann: spec updated | 21:37 |
| anteaya | fungi: yes I hope moving goes well | 21:37 |
| lifeless | clarkb: ok I'm with you | 21:37 |
| fungi | clarkb: anteaya: thanks! | 21:38 |
| *** signed8b_ has quit IRC | 21:38 | |
| *** sarob has joined #openstack-infra | 21:38 | |
| lifeless | clarkb: I need an etherpad to track this | 21:38 |
| *** adrian_otto has quit IRC | 21:39 | |
| lifeless | clarkb: is there one? | 21:39 |
| shivharis | fungi, anteaya: thanks | 21:39 |
| clarkb | lifeless: not yet but I can braindump into one really quickly | 21:39 |
| clarkb | jogo: can you see my comment on 98154 and see if that makes sense? | 21:39 |
| *** zz_gondoi is now known as gondoi | 21:39 | |
| lifeless | clarkb: my head just lost 'last option' in the context of a big wall of text :) | 21:40 |
| *** sweston has quit IRC | 21:40 | |
| lifeless | mordred: have you had a chance to poke at 94107 ? | 21:41 |
| *** adrian_otto has joined #openstack-infra | 21:41 | |
| sdague | fungi: so one last thing if I can bug you about. The gerrit option change :) | 21:42 |
| sdague | because I'm daily getting confused people ping me about why I -1ed their patch. | 21:42 |
| *** sarob has quit IRC | 21:43 | |
| fungi | sdague: the reset votes on commit message change setting? | 21:43 |
| sdague | fungi: yes please | 21:43 |
| lifeless | clarkb: highlight me when pad exists :) | 21:43 |
| *** adrian_otto1 has joined #openstack-infra | 21:44 | |
| sdague | russellb: currently expected, but I think wrong behavior, and the subject of me pinging fungi atm | 21:44 |
| fungi | sdague: i'll see if i can slip that into gerrit and not break stuff... though i should also send a change for the gerrit deployment documentation taking that setting out as well | 21:44 |
| sdague | fungi: it would be hugely appreciated | 21:44 |
| *** signed8bit has joined #openstack-infra | 21:44 | |
| russellb | sdague: cool thanks | 21:44 |
| sdague | the old behavior is something we kind of relied on | 21:44 |
| *** CaptTofu_ has quit IRC | 21:45 | |
| *** CaptTofu_ has joined #openstack-infra | 21:46 | |
| *** adrian_otto1 has quit IRC | 21:46 | |
| *** adrian_otto has quit IRC | 21:46 | |
| clarkb | lifeless: mordred https://etherpad.openstack.org/p/nodepool-dib-integration\ | 21:47 |
| clarkb | er s/\\// | 21:47 |
| *** mriedem has quit IRC | 21:47 | |
| clarkb | sdague: huh I didn't realize that was the issue | 21:49 |
| clarkb | gerrit continues to have the info in it... | 21:49 |
| *** ramashri has joined #openstack-infra | 21:50 | |
| *** CaptTofu_ has quit IRC | 21:50 | |
| *** lcostantino has quit IRC | 21:51 | |
| *** dims__ has joined #openstack-infra | 21:52 | |
| *** thuc has joined #openstack-infra | 21:52 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: New StoryBoard Logo https://review.openstack.org/99217 | 21:53 |
| *** UtahDave has joined #openstack-infra | 21:53 | |
| *** dims_ has quit IRC | 21:54 | |
| *** dhellmann has quit IRC | 21:55 | |
| *** ramashri_ has joined #openstack-infra | 21:55 | |
| *** ramashri has quit IRC | 21:55 | |
| *** dhellmann has joined #openstack-infra | 21:55 | |
| sdague | clarkb: yeh, but not reseting your vote by something that was fixable with a commit message change, ends up badly breaking review flow | 21:58 |
| *** signed8bit has quit IRC | 21:59 | |
| *** kimj has joined #openstack-infra | 21:59 | |
| *** kimj has quit IRC | 21:59 | |
| *** kimj has joined #openstack-infra | 21:59 | |
| lifeless | clarkb: so we make one schema for all projects? Or we make N schemas on the node image for the N projects? | 22:00 |
| clarkb | lifeless: we make 2 schemas because ironic was special | 22:00 |
| lifeless | clarkb: I just want to get my head around whats done today - feel free to point me at the config code | 22:00 |
| clarkb | lifeless: sure one sec | 22:00 |
| *** changbl has quit IRC | 22:00 | |
| *** duran has quit IRC | 22:00 | |
| *** thuc has quit IRC | 22:01 | |
| *** kimj has quit IRC | 22:01 | |
| clarkb | lifeless: it starts at https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/thick_slave.pp#n104 | 22:01 |
| *** kimj has joined #openstack-infra | 22:01 | |
| clarkb | lifeless: the install bits are not a problem. but eg https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/thick_slave.pp#n150 is | 22:01 |
| *** shivharis has quit IRC | 22:01 | |
| *** kimj has quit IRC | 22:02 | |
| *** rcarrillocruz has quit IRC | 22:02 | |
| clarkb | lifeless: basically we need a "root" db user without giving tests root on the machine | 22:02 |
| lifeless | clarkb: ok so we don't do the schema | 22:02 |
| clarkb | right we don't fill it in | 22:02 |
| lifeless | we just ensure there's a usable DB service | 22:02 |
| clarkb | yup | 22:02 |
| lifeless | ok | 22:03 |
| *** kimj has joined #openstack-infra | 22:03 | |
| lifeless | clarkb: how long does that take - it looks like a fraction of a second worth of work | 22:03 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Don't copy review scores on commit message updates https://review.openstack.org/99219 | 22:03 |
| fungi | russellb: sdague: ^ already applied in gerrit | 22:03 |
| clarkb | lifeless: well it is puppet... that is the bulk of our puppet runs | 22:04 |
| *** rcarrillocruz has joined #openstack-infra | 22:04 | |
| clarkb | lifeless: easily several seconds | 22:04 |
| sdague | fungi: thanks! | 22:04 |
| fungi | https://review.openstack.org/gitweb?p=All-Projects.git;a=blobdiff;f=project.config;h=34919a7;hp=ca6b31f;hb=a6a36da;hpb=a8c55ed | 22:04 |
| clarkb | lifeless: which may be puppet being slow | 22:04 |
| clarkb | the underlying puppet types do a lot of checks and things to make sure that they do the correct thing | 22:05 |
| lifeless | yep, and they have to because they are meant to run on live in-use systems | 22:05 |
| lifeless | with admins changing stuff around them | 22:05 |
| fungi | clarkb: we should probably update gerrit.rst to reflect the draft disabling change you pushed to the all-projects acl too? | 22:05 |
| clarkb | fungi: ah yup | 22:05 |
| fungi | i can add it | 22:05 |
| clarkb | thanks | 22:05 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Prevent Draft patchsets from being pushed https://review.openstack.org/99220 | 22:07 |
| fungi | clarkb: ^ | 22:07 |
| *** marcoemorais has quit IRC | 22:08 | |
| clarkb | fungi: approved | 22:09 |
| fungi | reed: so it occurs to me that we have some options to weigh on queryability of the icla group members list... we can either make that group visible to everyone, or we can add a separate owner group for it which grants control to specific accounts (but would also grant those specific accounts the ability to add and remove users from it as well) | 22:09 |
| fungi | zaro: ^ am i missing any alternatives you're aware of there? | 22:09 |
| reed | fungi, we need something read only... | 22:10 |
| reed | let's see... is that detail public already? | 22:10 |
| reed | or do we have any concern about privacy for it? | 22:10 |
| reed | I think we need to discuss this publicly on a mlist | 22:11 |
| *** aconrad has quit IRC | 22:11 | |
| fungi | reed: we can potentially add something more convoluted, but if there are no privacy concerns over the list of which gerrit accounts have signed the icla then i'd prefer we just make the group visible to all registered users | 22:11 |
| lifeless | clarkb: back on the etherpad | 22:11 |
| clarkb | lifeless: yup | 22:11 |
| fungi | reed: it's probably a slim superset of the list of e-mail addresses in official openstack git repositories after all, so there's likely little privacy concern in my mind | 22:12 |
| fungi | (i mean, we autogenerate authors files in all official projects which are literally the list of names and e-mail addresses of the people who committed to them, no need to even scrape commit logs for that info) | 22:13 |
| *** smarcet has quit IRC | 22:13 | |
| reed | fungi, makes sense to me... i'd like to have a second opinion | 22:14 |
| zaro | fungi: there's another option, but probably same as latter. i think there's a gerrit plugin that allows giving users direct access to groups. | 22:14 |
| *** nati_ueno has quit IRC | 22:14 | |
| clarkb | fungi: reed I agree that shouldn't be secret info as all contributions are public | 22:14 |
| reed | i know we consider email addresses of committers as public knowledge and that we have been keeping a public list of ICLA signatories until recently, on the wiki | 22:14 |
| zaro | fungi: but easier just to make them visible to all. +1 for that option. | 22:15 |
| reed | i'm convinced | 22:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't copy review scores on commit message updates https://review.openstack.org/99219 | 22:15 |
| reed | clarkb, fungi: you're right, who signed the iCLA is not a secret | 22:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Prevent Draft patchsets from being pushed https://review.openstack.org/99220 | 22:15 |
| *** gondoi is now known as zz_gondoi | 22:16 | |
| reed | same with the Government CLA | 22:16 |
| fungi | reed: i'll fix them right now | 22:17 |
| *** thuc has joined #openstack-infra | 22:17 | |
| *** thuc_ has joined #openstack-infra | 22:19 | |
| *** denis_makogon has quit IRC | 22:19 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Don't deny visibility of ICLA group to its members https://review.openstack.org/99223 | 22:21 |
| fungi | reed: clarkb: zaro: i'm having to do that ^ to accomodate it | 22:21 |
| clarkb | fungi: +2 | 22:21 |
| *** bknudson has quit IRC | 22:22 | |
| *** thuc has quit IRC | 22:22 | |
| reed | ok, let's have that reviewed by Mark Collier | 22:22 |
| reed | CYA :) | 22:23 |
| fungi | reed: will do | 22:23 |
| harlowja | lifeless if u get anytime, your knowledge might be useful on https://review.openstack.org/#/c/98946/ | 22:23 |
| fungi | reed: you want to ping him on that? if he's +1 i'll push the change to the gerrit acl accomodating it | 22:23 |
| lifeless | harlowja: what project is that in ? | 22:24 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add a tool for determining unaccounted for rechecks https://review.openstack.org/94624 | 22:24 |
| fungi | sometimes sparkycollier is in here, but apparently not at the moment | 22:24 |
| * fungi switches to evening tasks now | 22:24 | |
| harlowja | lifeless taskflow :) | 22:24 |
| reed | fungi, pinged | 22:25 |
| reed | fungi, thanks | 22:25 |
| lifeless | harlowja: (using gertty, and haven't gotten around to a 'jump to review' feature) | 22:25 |
| harlowja | lifeless gotcha, np, i gotta work on my gertty skillz | 22:26 |
| lifeless | mmm shiny, ubuntu are going to do an image based server thing next cycle | 22:26 |
| harlowja | lifeless oh, so my misunderstanding, that spec is in oslo-specs (the project its for is taskflow) | 22:27 |
| clarkb | lifeless: ^O | 22:28 |
| clarkb | that should allow you to punch in a change number | 22:29 |
| *** Sukhdev has quit IRC | 22:29 | |
| *** Sukhdev has joined #openstack-infra | 22:29 | |
| lifeless | clarkb: ah! so that wants to be in the top level f1 text | 22:29 |
| clarkb | indeed that was apparently not added | 22:31 |
| *** nati_ueno has joined #openstack-infra | 22:32 | |
| *** zns has joined #openstack-infra | 22:33 | |
| jhesketh | Morning | 22:34 |
| morganfainberg | wow, we're going to hit 100k reviews through gerrit soonish. | 22:36 |
| morganfainberg | well proposed reviews. | 22:36 |
| zaro | fungi, clarkb i think visibility is a gerrit system config setting? | 22:36 |
| lifeless | clarkb: so some more options up about nodepool | 22:37 |
| lifeless | clarkb: I have a call in 23m | 22:37 |
| zaro | clarkb, fungi : https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#accounts | 22:37 |
| lifeless | harlowja: can subgraphs not run today ? | 22:38 |
| *** zns_ has joined #openstack-infra | 22:38 | |
| harlowja | lifeless they do, just not conditionally | 22:38 |
| *** zns has quit IRC | 22:38 | |
| *** penguinRaider has quit IRC | 22:38 | |
| *** sarob has joined #openstack-infra | 22:38 | |
| lifeless | harlowja: if they don't run, is it an error ? | 22:38 |
| clarkb | lifeless: looking. I do have my image running in qemu now though. very slow as epxected | 22:38 |
| clarkb | lifeless: waiting for my ssh connection to give me a prompt | 22:38 |
| *** signed8bit has joined #openstack-infra | 22:38 | |
| lifeless | clarkb: do-be-do-be-do | 22:38 |
| harlowja | lifeless just means its the workflow isn't complete, not neccasarily error | 22:38 |
| harlowja | *just means that the | 22:39 |
| lifeless | ok so specifically, you cannot have a complete workflow unless all nodes of the dag have executed | 22:39 |
| harlowja | lifeless correct | 22:39 |
| clarkb | lifeless: according to the console output we are still booting though... | 22:39 |
| *** doude has quit IRC | 22:39 | |
| harlowja | lifeless i'm working on a update to that to see how the usage would look, we have a way to ignore parts of the workflow and still be complete (so i think we can do that easily) | 22:40 |
| anteaya | morning jhesketh | 22:41 |
| lifeless | harlowja: I'm going to sketch some thoughts, probably using the wrong words | 22:41 |
| lifeless | harlowja: but hopefully useful | 22:41 |
| harlowja | lifeless thats fair :) | 22:41 |
| harlowja | thanks | 22:41 |
| harlowja | your brainpower useful no matter ;) | 22:41 |
| harlowja | *braains | 22:41 |
| harlowja | haha | 22:41 |
| clarkb | fungi: anteaya: should we go ahead and address the infra specs initial commit coments now and push that through? | 22:42 |
| clarkb | I am happy to address them while I wait on things to boot | 22:42 |
| anteaya | clarkb: I'm for that | 22:42 |
| clarkb | lifeless: cloud-init boot finished at Tue, 10 Jun 2014 22:42:13 +0000. Up 673.35 seconds | 22:42 |
| clarkb | lifeless: not really quick :/ | 22:42 |
| anteaya | teh comment you had was /** for createing the docs | 22:42 |
| *** sarob has quit IRC | 22:42 | |
| clarkb | anteaya: correct, though I think I like fungi's idea of explicitly calling out implemented | 22:43 |
| anteaya | clarkb: I don't think adding a second asterisk is that wild of a change | 22:43 |
| *** kimj has quit IRC | 22:43 | |
| *** thedodd has quit IRC | 22:43 | |
| anteaya | clarkb: I'm meh on that, if you like it put it in | 22:45 |
| *** thuc has joined #openstack-infra | 22:45 | |
| anteaya | clarkb: if jim hates it he will take it out | 22:45 |
| *** thuc_ has quit IRC | 22:45 | |
| anteaya | and we will have specs in review at the same time | 22:45 |
| anteaya | though none in implemented/ for a while | 22:45 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep. https://review.openstack.org/96877 | 22:46 |
| *** zns has joined #openstack-infra | 22:46 | |
| *** ArxCruz has joined #openstack-infra | 22:46 | |
| *** penguinRaider has joined #openstack-infra | 22:47 | |
| *** zns_ has quit IRC | 22:49 | |
| clarkb | anteaya: well they fix the same problem | 22:49 |
| *** devkulkarni has joined #openstack-infra | 22:50 | |
| anteaya | do they? | 22:51 |
| anteaya | I guess I don't know enough about what /** does | 22:51 |
| clarkb | anteaya: actually I am leaning back towards my original comment | 22:52 |
| clarkb | well maybe not | 22:52 |
| clarkb | lifeless: Notice: Compiled catalog for precise30gbtest.openstacklocal in environment production in 128.17 seconds | 22:52 |
| lifeless | thats tolerable I guess? | 22:53 |
| lifeless | harlowja: ok, when gertty syncs (it has 750 pending!) you'll get some feedback :) | 22:53 |
| harlowja | harlowja thx much | 22:54 |
| harlowja | *oops, lifeless ^ | 22:54 |
| clarkb | lifeless: well 11 minutes to boot, 2 minutes just to compile catalog then you have to apply it | 22:54 |
| lifeless | clarkb: heh. But its async right, once a day ? | 22:54 |
| clarkb | lifeless: I am running puppet under time so should have a rough guestimate for total time | 22:54 |
| clarkb | lifeless: ya I think once a day is what we would go for | 22:55 |
| *** homeless has quit IRC | 22:55 | |
| mattoliverau | Good Morning | 22:55 |
| clarkb | mordred: fungi re pbr versioning in setup.py I thought we did want it to be versioned | 22:55 |
| clarkb | we just use the big range | 22:56 |
| anteaya | morning mattoliverau | 22:56 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Including UX Feedback on menu and nav. https://review.openstack.org/99209 | 22:56 |
| anteaya | clarkb: yeah I think your comment is a gentle change | 22:56 |
| anteaya | clarkb: I think adding implemented is a design change | 22:57 |
| anteaya | I'm fine with /** and merge | 22:57 |
| anteaya | then we have time to add implemented before anything is actually implemented | 22:57 |
| *** CaptTofu_ has joined #openstack-infra | 23:00 | |
| *** thuc has quit IRC | 23:03 | |
| *** thuc has joined #openstack-infra | 23:03 | |
| *** CaptTofu_ has quit IRC | 23:04 | |
| *** bogdando has quit IRC | 23:04 | |
| *** radez is now known as radez_g0n3 | 23:04 | |
| mordred_phone | clarkb: no. never version pbr in setup.py | 23:05 |
| mordred_phone | never ever ever | 23:05 |
| clarkb | mordred_phone: huh | 23:06 |
| clarkb | mordred_phone: you should probably fix that everywhere then | 23:06 |
| mordred_phone | there is no remediation for setup_requires failure | 23:06 |
| mordred_phone | I did a while back | 23:06 |
| *** dims__ has quit IRC | 23:06 | |
| clarkb | Notice: Finished catalog run in 732.66 seconds | 23:07 |
| clarkb | so we are talking about ~half an hour to do the qemu post dib stuff | 23:07 |
| *** bogdando has joined #openstack-infra | 23:07 | |
| *** zns has quit IRC | 23:08 | |
| *** dims__ has joined #openstack-infra | 23:08 | |
| clarkb | anteaya: ok, so I did it the other way, I will wait and see what fungi says | 23:10 |
| clarkb | anteaya: they really do the same thing | 23:10 |
| clarkb | anteaya: they publish the contents of imeplemented/*. One happens to do it in a nicer way than the other imo | 23:10 |
| *** nati_ueno has quit IRC | 23:10 | |
| anteaya | clarkb: okay | 23:11 |
| anteaya | I trust you | 23:11 |
| *** doug-fish has left #openstack-infra | 23:12 | |
| *** rfolco has joined #openstack-infra | 23:13 | |
| openstackgerrit | Andrew Melton proposed a change to openstack/requirements: Add IDMapShift for Nova https://review.openstack.org/98888 | 23:13 |
| *** zns has joined #openstack-infra | 23:13 | |
| *** UtahDave has quit IRC | 23:14 | |
| SlickNik | Hey guys. | 23:14 |
| SlickNik | So we have a couple of patches to trove which are updates from the global requirements. | 23:14 |
| SlickNik | But they are failing the requirements gate. | 23:15 |
| SlickNik | example: https://review.openstack.org/#/c/99095/ | 23:15 |
| SlickNik | Any idea why this might be happening, or if it is a known issue (before I try and dig deeper)? | 23:16 |
| *** atiwari has quit IRC | 23:16 | |
| clarkb | SlickNik: that may be a new issue with the exclusion | 23:16 |
| clarkb | SlickNik: there is an existing problem where we don't apply that test to the requirements changes themselves | 23:16 |
| harlowja | lifeless good ideas & comments, think we were thinking on the same wave-length (similar changes i was also writing up, haha) | 23:17 |
| harlowja | *scary*, ha | 23:17 |
| *** UtahDave has joined #openstack-infra | 23:17 | |
| *** zns has quit IRC | 23:18 | |
| SlickNik | clarkb: Thanks for the info! | 23:19 |
| *** gokrokve has quit IRC | 23:19 | |
| *** pcrews_ has quit IRC | 23:20 | |
| *** pcrews has joined #openstack-infra | 23:20 | |
| *** reed_ has joined #openstack-infra | 23:21 | |
| *** reed has quit IRC | 23:22 | |
| clarkb | anteaya: fungi I am testing those changes now real quick (want to make sure we can still build sphinx after the pbr and sphinx changes | 23:23 |
| anteaya | kk | 23:23 |
| clarkb | anteaya: and of course it finds a bug with the implemented stuff... | 23:23 |
| clarkb | anteaya: going to switch to /** now | 23:24 |
| anteaya | k | 23:24 |
| clarkb | anteaya: (you need to have a file in there otherwise sphinx goes warning no files and everything fails) | 23:24 |
| anteaya | ah | 23:24 |
| anteaya | helpful sphinx | 23:24 |
| anteaya | now I know | 23:24 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/infra-specs: Initial commit https://review.openstack.org/94440 | 23:25 |
| clarkb | anteaya: mordred_phone jhesketh SergeyLukjanov fungi ^ | 23:25 |
| clarkb | the diff should be relatively small | 23:25 |
| *** reed_ has quit IRC | 23:27 | |
| anteaya | clarkb: and that patch passed? | 23:27 |
| devkulkarni | SlickNick, clarkb: we are seeing similar error in Solum as well https://review.openstack.org/#/c/99103/ .. based on your exchange I am not clear what actions if any we need to take | 23:27 |
| clarkb | anteaya: yes tox -redocs worked with what I pushed | 23:29 |
| anteaya | clarkb: and line 8 was the only change? https://review.openstack.org/#/c/94440/4/doc/source/index.rst | 23:29 |
| clarkb | devkulkarni: SlickNik I believe that specific check was reverted last night | 23:29 |
| anteaya | I wish we had difs between patchsets | 23:29 |
| clarkb | anteaya: no, I fixed pbr and added fungi's security section | 23:30 |
| clarkb | anteaya: you do have diffs between patchsets don't you? | 23:30 |
| *** praneshp has quit IRC | 23:30 | |
| clarkb | anteaya: diff between patchsets works for me | 23:30 |
| anteaya | really? | 23:30 |
| clarkb | yes | 23:30 |
| devkulkarni | clarkb: thanks for the info. so I assume nothing needs to be done from our sides. once the check gets reapplied the review should pass | 23:30 |
| anteaya | how do I get that view? | 23:30 |
| clarkb | anteaya: are you using gerrity or gertty? | 23:30 |
| clarkb | devkulkarni: yes I think that is the case | 23:31 |
| openstackgerrit | Angus Salkeld proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge https://review.openstack.org/98237 | 23:31 |
| clarkb | anteaya: * gerrit or gertty? | 23:31 |
| anteaya | gerrit | 23:31 |
| *** olaph has quit IRC | 23:31 | |
| anteaya | I don't have gertty working reliably yet | 23:31 |
| devkulkarni | clarkb: thanks | 23:32 |
| sdague | clarkb: so we've seemed to level off on logstash catching up | 23:32 |
| clarkb | anteaya: just change the base to 3 on that change | 23:32 |
| sdague | it remains 25k events behind | 23:32 |
| anteaya | k and then? | 23:32 |
| clarkb | sdague: yeah I think we have the properish number of nodes they just aren't fast enough to bring it down while busy | 23:32 |
| clarkb | anteaya: click on diff | 23:32 |
| anteaya | oh you changed 4 files | 23:32 |
| anteaya | cool, thanks | 23:32 |
| clarkb | anteaya: you can also do it in the diff page by picking a patchset on the left hand side | 23:33 |
| *** praneshp has joined #openstack-infra | 23:33 | |
| sdague | clarkb: bummer :( | 23:33 |
| *** devkulkarni has quit IRC | 23:33 | |
| anteaya | clarkb: ah okay | 23:33 |
| anteaya | lightbuld | 23:34 |
| anteaya | lightbulb | 23:34 |
| anteaya | so yeah, +1 to your patch | 23:34 |
| clarkb | anteaya: thanks | 23:34 |
| anteaya | :D | 23:34 |
| anteaya | let's have some specs | 23:34 |
| anteaya | since the world isn't on fire atm | 23:34 |
| anteaya | but the gate got long again | 23:35 |
| sdague | anteaya: it's still on fire, it's just not zomg fire | 23:35 |
| anteaya | 27 patches added in the last two hours | 23:36 |
| anteaya | sdague: did the nova-network logging patch get +A'd yet? | 23:36 |
| anteaya | sdague: that was a priorty for the fire, yeah? | 23:36 |
| sdague | no, it had some issues, I don't think mikal spun a new rev yet | 23:36 |
| anteaya | ah okay | 23:36 |
| anteaya | jhesketh: is mikal around yet? | 23:37 |
| jhesketh | anteaya: I think he's around, but he's very busy | 23:38 |
| anteaya | jhesketh: k thanks | 23:38 |
| *** sarob has joined #openstack-infra | 23:38 | |
| anteaya | sdague: was there anything else on your list of help quench the fire? | 23:38 |
| anteaya | sdague: been sort of pulled off on other things today | 23:38 |
| *** thuc has quit IRC | 23:39 | |
| *** reed_ has joined #openstack-infra | 23:39 | |
| *** thuc has joined #openstack-infra | 23:39 | |
| sdague | honestly, my tank is empty there. Today I mostly was trying to get my 25 outstanding patches to a passing state | 23:39 |
| sdague | and then discovering that 5 - 10% of our logs (by bytes) are keystone auth tokens | 23:40 |
| morganfainberg | sdague, gah, really? sorry :( | 23:41 |
| clarkb | sdague: ya those things are pretty annoying | 23:41 |
| clarkb | sdague: I think they cause the most problems with the crm114 to | 23:41 |
| sdague | 40 MB per run (uncompressed) | 23:41 |
| morganfainberg | ouch | 23:41 |
| sdague | morganfainberg: it's mostly actually the fault that we log them | 23:41 |
| morganfainberg | sdague, right. | 23:41 |
| *** ArxCruz has quit IRC | 23:41 | |
| morganfainberg | sdague, still bleh. | 23:41 |
| sdague | which is actually, arguably, a security issue | 23:41 |
| anteaya | sdague: understood | 23:42 |
| morganfainberg | sdague, yeah, shouldn't log tokens... | 23:42 |
| sdague | I suspect that's going to be the list discussion tomorrow, as there is odd resistance to patches like this - https://review.openstack.org/#/c/98443/ | 23:42 |
| sdague | so we're going to just have to agree as a community that logging auth tokens is terrible, and stop :) | 23:43 |
| *** sarob has quit IRC | 23:43 | |
| morganfainberg | sdague, i'll confirm we're not logging those in the session object | 23:43 |
| morganfainberg | sdague, so perhaps when people move to keystone client session... they'll just "disappear" | 23:43 |
| morganfainberg | >.> | 23:43 |
| morganfainberg | if there really is tha tmuch resistence | 23:43 |
| sdague | well, they log it in requests as well | 23:43 |
| *** thuc has quit IRC | 23:43 | |
| anteaya | clarkb: check my sanity here: https://review.openstack.org/#/c/91050/4 | 23:44 |
| clarkb | sdague: right this is one of the things that came up in the user sessions at the summit | 23:44 |
| anteaya | clarkb: but base to patchset 3 | 23:44 |
| morganfainberg | sdague, the session object abstracts all that out. | 23:44 |
| anteaya | clarkb: what do you see? | 23:44 |
| sdague | clarkb: right, I was in there | 23:44 |
| morganfainberg | sdague, keystoneclient would manage that part not "novaclient" for example. | 23:44 |
| morganfainberg | (needed for auth plugins) | 23:44 |
| clarkb | anteaya: usually that indicates a rebase, so the diff includes things from other changes that merged | 23:44 |
| sdague | morganfainberg: it would manage req to nova api? | 23:44 |
| anteaya | oh okay, thanks | 23:44 |
| clarkb | anteaya: this is one of the issues with gerrit's diffing. You can use git review -m instead | 23:44 |
| sdague | morganfainberg: I'm confused | 23:44 |
| morganfainberg | sdague, keystoneclient has a session object we're trying to convert everyone over to using | 23:45 |
| sdague | morganfainberg: sure | 23:45 |
| anteaya | clarkb: I'll just move ahead without diffs between patchsets | 23:45 |
| morganfainberg | sdague, that handles all forms of auth...in fact, could be passed between client objects | 23:45 |
| morganfainberg | sdague, in _theory_ we could prevent the logging of the token via conversion to the session object | 23:45 |
| sdague | how does that impact the req_logging in novaclient | 23:45 |
| morganfainberg | sdague, if the req doesn't know the token, how can it log it, the session would handle that bit, session holds the token | 23:46 |
| sdague | oh | 23:46 |
| morganfainberg | sdague, i think that is the plan at least. | 23:46 |
| sdague | ok | 23:46 |
| sdague | yeh, what's the eta on that? | 23:46 |
| sdague | that seems like it would take a while | 23:46 |
| morganfainberg | sdague, look at the masssssive number of reviews jamie lennox has open. | 23:46 |
| *** yamahata has quit IRC | 23:46 | |
| morganfainberg | https://review.openstack.org/#/q/owner:%22Jamie+Lennox%22+status:open,n,z | 23:46 |
| sdague | I already have a massive number of reviews to look at :) | 23:46 |
| morganfainberg | a number of them are converting to the session object | 23:47 |
| *** mmaglana has quit IRC | 23:47 | |
| morganfainberg | i'll poke into these and see if we're still logging the token with the session | 23:47 |
| morganfainberg | but i think we can make that part of the conversion (make it some other really out of the way logging if it's realllly desired) instead of part of debug | 23:47 |
| sdague | yeh, that would be good | 23:47 |
| *** thuc has joined #openstack-infra | 23:48 | |
| anteaya | mattoliverau: you might enjoy a peek at the meeting log from last week: http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-06-03-19.02.log.html | 23:48 |
| morganfainberg | as i said in -qa channel, i agree, logging token is only marginally less bad than logging passwords | 23:48 |
| sdague | yeh | 23:48 |
| sdague | morganfainberg: ++ | 23:48 |
| anteaya | mattoliverau: we addressed some consistency issues with the acl files in openstack-infra/config | 23:48 |
| sdague | now you just need to help me convince jogo of that :) | 23:48 |
| *** lcheng has quit IRC | 23:49 | |
| morganfainberg | sdague, i'll do my best! :) | 23:49 |
| morganfainberg | sdague, but i think he wasn't opposed as long as we document it clearly that that change is happening. | 23:49 |
| *** CaptTofu_ has joined #openstack-infra | 23:50 | |
| clarkb | bodepd: let me know what you think of my comments in 96336. | 23:50 |
| sdague | I did that, he still -1ed it | 23:50 |
| sdague | on the updated patch | 23:50 |
| morganfainberg | sdague, bleh | 23:51 |
| sdague | :) | 23:51 |
| clarkb | bodepd: mostly rm -Rvf is scary and that function is known to work so its nice to stick with it | 23:51 |
| sdague | ok, I think it might be drone flying time | 23:51 |
| *** gokrokve has joined #openstack-infra | 23:52 | |
| mattoliverau | anteaya: I'll take a peek, thanks :) | 23:53 |
| *** bogdando has quit IRC | 23:54 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer test_notify_alarm bug 1321826 https://review.openstack.org/98149 | 23:55 |
| uvirtbot | Launchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress] https://launchpad.net/bugs/1321826 | 23:55 |
| morganfainberg | sdague, i commented on that review just now | 23:55 |
| morganfainberg | sdague, i'll get back to you on the state of the logging w/ the session object tomorrow i think. | 23:56 |
| *** zzelle_ has quit IRC | 23:56 | |
| *** bogdando has joined #openstack-infra | 23:57 | |
| *** thuc has quit IRC | 23:57 | |
| *** thuc has joined #openstack-infra | 23:58 | |
| *** dangers is now known as dangers_away | 23:58 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use WatchedFileHandler to avoid copytruncate https://review.openstack.org/95935 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
