| *** bpokorny_ has joined #openstack-infra | 00:00 | |
| anteaya | doug-fish: yes | 00:00 |
|---|---|---|
| doug-fish | fantastic. Thanks! | 00:00 |
| *** flepied has joined #openstack-infra | 00:00 | |
| *** otsuka has joined #openstack-infra | 00:00 | |
| mordred | smarcet: well... I can think of a few ways we might address that, all of them seem like terrible ideas | 00:00 |
| anteaya | doug-fish: hopefully a patch to resolve the issue will merge soon | 00:00 |
| *** thorst has quit IRC | 00:01 | |
| mordred | smarcet: let me go read some of your deploy code real quick and see if I have better non-suck ideas | 00:01 |
| anteaya | doug-fish: and we hope it resolves the issue | 00:01 |
| doug-fish | understood | 00:01 |
| anteaya | doug-fish: thanks | 00:01 |
| jamesmcarthur | ok - i have to take off for a bit. Thanks to anteaya: mordred: and jeblair: for the assist. | 00:02 |
| jamesmcarthur | I’ll check back in a bit after the patch has merged. | 00:02 |
| *** jamesmcarthur has quit IRC | 00:02 | |
| anteaya | jamesmcarthur: thanks for helping us understand the issue | 00:02 |
| *** ZZelle_ has quit IRC | 00:02 | |
| *** rbrndt_ has quit IRC | 00:02 | |
| *** bpokorny has quit IRC | 00:03 | |
| *** zz_dimtruck is now known as dimtruck | 00:08 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Make lbaasv2-minimal job voting in check and gate https://review.openstack.org/286328 | 00:09 |
| *** cloudtrainme has quit IRC | 00:12 | |
| anteaya | pabelanger: how are you deleting spam? | 00:13 |
| anteaya | pabelanger: can I do anything to help? | 00:13 |
| *** tiswanso has joined #openstack-infra | 00:14 | |
| *** sdake has quit IRC | 00:14 | |
| *** rhallisey has joined #openstack-infra | 00:16 | |
| pabelanger | anteaya: currently, I've been manually looking at logs and blocking / deleting by hand | 00:16 |
| ianw | SpamapS: so i guess your point is, if it doesn't have CI, is it really supported? | 00:17 |
| pabelanger | moving forward, we should look at automating the clean up | 00:17 |
| *** Qiming has quit IRC | 00:17 | |
| pabelanger | since there is about 1GB of PDF files | 00:17 |
| jpmaxman | ahh I thought you'd automated | 00:17 |
| pabelanger | not yet | 00:17 |
| pabelanger | will look into that in the morning | 00:17 |
| jpmaxman | so, just to be clear, we still don't know the attack vector but we're effectively tracking spam accounts and blocking them? | 00:17 |
| pabelanger | and use existing tooling from mediawiki to combat it | 00:17 |
| jpmaxman | well the qeustyquest from mediawiki tooling should block it completely | 00:18 |
| jpmaxman | but isn't | 00:18 |
| pabelanger | From what I see, I don't think we have a security issue. Just authenticated accounts spamming our wiki | 00:18 |
| jpmaxman | I'd like to further investigate why that is | 00:18 |
| *** dalgaaf has quit IRC | 00:18 | |
| jpmaxman | but how are they getting past captcha question? | 00:18 |
| *** abitha has quit IRC | 00:18 | |
| jpmaxman | do we need to make it harder? or is it broken? or is there a vulnerability? | 00:18 |
| pabelanger | from what I see and read, they broke our captcha | 00:19 |
| pabelanger | either ORC or a human did it | 00:19 |
| pabelanger | questions aren't that hard | 00:19 |
| jpmaxman | I'd like some verification on that | 00:19 |
| *** abitha has joined #openstack-infra | 00:19 | |
| jpmaxman | like making the questions harder as a start | 00:19 |
| *** ashleighfarnham has joined #openstack-infra | 00:19 | |
| *** ashleighfarnham has quit IRC | 00:19 | |
| jpmaxman | we never saw even a momentary break from the spam | 00:19 |
| jpmaxman | when we enabled the captcha | 00:19 |
| jpmaxman | I'd think it would have taken them at least one minute | 00:19 |
| jpmaxman | to adjust | 00:19 |
| *** ybathia has quit IRC | 00:20 | |
| *** _joes_ has quit IRC | 00:20 | |
| pabelanger | not sure about that, easy to check however, if we did some stats on logs | 00:20 |
| jpmaxman | I was watching it on the recently updated special page | 00:20 |
| pabelanger | So, I'll stop blocking accounts for tonight and wait until the spam starts again | 00:20 |
| jpmaxman | maybe we can start by making the question more difficult? | 00:20 |
| pabelanger | we'll ask an infra-root to block out captcha and see what happens | 00:21 |
| *** arxcruz has quit IRC | 00:21 | |
| *** _joes_ has joined #openstack-infra | 00:21 | |
| jpmaxman | an impossible question would be ultimately telling, but even making it more difficult might provide some indication | 00:21 |
| pleia2 | I can change them back to being unanswerable | 00:21 |
| pleia2 | if we want to test whether they're bypassing entirely | 00:21 |
| pabelanger | pleia2: the issue now is, we need spammers :) | 00:22 |
| pleia2 | did they disappear? :) | 00:22 |
| jpmaxman | pleia2: yes, that was my suggestion earlier. But I agree with pabelanger let's wait until tomorrow | 00:22 |
| *** Sukhdev has quit IRC | 00:22 | |
| pabelanger | https://wiki.openstack.org/wiki/Special:NewPages and https://wiki.openstack.org/wiki/Special:RecentChanges are void of them currently | 00:22 |
| pleia2 | bots don't know about february 29th | 00:22 |
| jpmaxman | pabelanger has been going into hand to hand combat with them | 00:22 |
| jpmaxman | and so far is winning ;) | 00:22 |
| * pleia2 nods | 00:22 | |
| pabelanger | Our current changes have stopped the attack, tomorrow is a new day :) | 00:23 |
| pleia2 | tomorrow is fine, but we do want to do it when dev-time is quiet (since it makes the wiki effectively read only for everyone) | 00:23 |
| jpmaxman | so maybe we can reconvene here at 5:00pm UTC tomorrow | 00:23 |
| pabelanger | pleia2: good idea | 00:23 |
| jpmaxman | sure whenever that is, I can be here, let's just have people online so we can apply the change then undo it quickly | 00:24 |
| *** abitha has quit IRC | 00:24 | |
| pleia2 | 2100 utc or later is probably best | 00:24 |
| fungi | yeah, i would not be at all surprised to see pretty much any solvable captcha bypassed. mechanical turk is a thing | 00:24 |
| * pleia2 nods | 00:25 | |
| fungi | apparently the hip term now is "captcha farmers" | 00:26 |
| jeblair | i generally think of spammers as bot-assisted humans | 00:26 |
| jpmaxman | It's possible but we've blocked a lot of spammers there's so many opportunities out there that generally the "captcha farmers" are used for more valuable targets than updating a wiki page. | 00:28 |
| *** tiswanso has quit IRC | 00:28 | |
| jpmaxman | these seem to be bots that once you figure out what they're doing they just move on to the next undprotected mediawiki | 00:28 |
| fungi | yes, spam evasion is sort of like outrunning a bear. you don't need to be faster than the bear... | 00:28 |
| jpmaxman | exacly fungi | 00:28 |
| *** tiswanso has joined #openstack-infra | 00:28 | |
| fungi | however, being a higher-traffic target, we'll be preferred by ranking-savvy spammers, so we need to be less convenient to spam than other high-traffic wikis out there | 00:30 |
| jpmaxman | yup you probably got some google juice | 00:31 |
| jpmaxman | that they like :) | 00:31 |
| jpmaxman | I could be wrong but my gut says this isn't human assisted - I think it would be easy enough to figure out | 00:32 |
| jpmaxman | what is the fourth word in this sentence is pretty easily scriptable | 00:32 |
| fungi | i'm just glad they've finally lost interest in spamming usenet. in another 10 years e-mail will be mostly spam-free and in 20 years so will wikis ;) | 00:32 |
| jpmaxman | hah | 00:33 |
| SpamapS | ianw: yes that's my point. :) | 00:33 |
| *** boris-42 has quit IRC | 00:34 | |
| anteaya | pabelanger: was having some dinner, seems like the response is see what happens tomorrow, and if we have time try to automate deletion | 00:34 |
| *** Keedya_ has joined #openstack-infra | 00:35 | |
| *** Keedya_ has quit IRC | 00:36 | |
| mtreinish | infra-root: when you get a sec to fix the subunit gearman worker we need: https://review.openstack.org/286304 and then to restart the worker after that's applied | 00:36 |
| mtreinish | it'll also be good to land: https://review.openstack.org/285560 if we need to restart it | 00:36 |
| *** markvoelker has quit IRC | 00:38 | |
| *** Sukhdev has joined #openstack-infra | 00:38 | |
| *** thiagop has quit IRC | 00:38 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run cacti's node generation from cron https://review.openstack.org/284466 | 00:38 |
| *** bpokorny_ has quit IRC | 00:39 | |
| *** bpokorny has joined #openstack-infra | 00:39 | |
| nibalizer | mtreinish: ok i got you | 00:40 |
| mtreinish | nibalizer: cool, thanks | 00:41 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Fix bug on missing subunit2sql data https://review.openstack.org/286304 | 00:46 |
| *** sridhar_ram1 has joined #openstack-infra | 00:47 | |
| nibalizer | infra-root the puppetmaster isn't applying again | 00:51 |
| nibalizer | because the ansible-inventory is failing because the west region isn't there | 00:51 |
| fungi | it won't just skip? | 00:51 |
| *** baoli has joined #openstack-infra | 00:52 | |
| fungi | i have a feeling we need to remove it from nodepool.yaml and then from clouds.yaml to avoid the issue jhesketh ran into | 00:52 |
| nibalizer | the weird thing is I thought jhesketh fixed this last night | 00:52 |
| *** Sukhdev has quit IRC | 00:52 | |
| nibalizer | https://review.openstack.org/#/c/285854/ | 00:53 |
| jhesketh | fungi, nibalizer: yes, I removed it from clouds.yaml... maybe I missed something or it reverted | 00:53 |
| fungi | nodepoold was still trying and failing to reach infra-cloud west earlier today when i was looking in its debug.log | 00:53 |
| nibalizer | fungi: yea when we reference a cloud that doesn't exist it blows up our inventory which aborts | 00:53 |
| jeblair | jhesketh: yeah, you removed it from the wrong clouds.yaml | 00:53 |
| jeblair | jhesketh: you removed it from the one nodepool uses | 00:54 |
| nibalizer | jhesketh: it was working at 2016-03-01 00:17:49,589 | 00:54 |
| jhesketh | ah, and the inventory is cached | 00:54 |
| jeblair | that's why nodepool broke | 00:54 |
| jhesketh | right, I see | 00:54 |
| jhesketh | sorry about that :-( | 00:54 |
| *** smarcet has quit IRC | 00:54 | |
| nibalizer | oh so we need a new patch to /etc/openstack/clouds.yaml on the puppetmaster? | 00:54 |
| jeblair | jhesketh: we have too mayn clouds.yamlses :) | 00:54 |
| fungi | oh, yes, cached inventory means we have up to 24 hours to see it start breaking | 00:54 |
| nibalizer | https://twitter.com/nibalizer/status/702589754223644672 | 00:54 |
| jeblair | nibalizer: i think so, if no one else has done it | 00:54 |
| jhesketh | nibalizer: yes, and we'll need to probably apply it manually to the puppetmaster | 00:55 |
| nibalizer | im on it | 00:55 |
| jhesketh | okay, I'll do a revert for my bad one | 00:55 |
| nibalizer | no i think we need yours too | 00:55 |
| jeblair | yeah, i think nodepool is fine now | 00:55 |
| mordred | jhesketh: oh - also - did you see my response to your patch to ansible? | 00:56 |
| jeblair | jhesketh: nodepool was fixed by a followup change that removed infra-cloud | 00:56 |
| jhesketh | mordred: yes, haven't had time to look at it yet though sorry | 00:56 |
| jhesketh | jeblair: correct, but we should have the credentials there for when infra-west is back | 00:56 |
| mordred | jhesketh: I wholeheartedly agree with you about the patch and think it's great | 00:56 |
| jhesketh | or at least ready for review | 00:56 |
| mordred | jhesketh: I think we need to plumb a config thing through occ- because there isn't really a way to pass parameters to the inventory script in normal operation | 00:57 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Remove infracloudwest from ansible-clouds.yaml https://review.openstack.org/286337 | 00:57 |
| mordred | jhesketh: but we've got an example of one other config setting that the inventory reads already, so cargo-culting should be easy enough | 00:57 |
| fungi | oh, the nodepool debug errors are about failing to image-delete because it can't find infracloud-west defined, so we need to manually clean up the images | 00:57 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Use first test from subunit_stream for run_at value https://review.openstack.org/285560 | 00:57 |
| jhesketh | mordred: yeah I noticed that limitation... but I was thinking about the more general use case of shade/the inventory where you probably want to know when your operation isn't working on a particular cloud | 00:58 |
| nibalizer | infra-root 286337 | 00:58 |
| jhesketh | mordred: so I wanted the flag to be set on list_hosts rather than in the config | 00:58 |
| nibalizer | im going to apply that manually as well since we can't auto un-wedge | 00:58 |
| *** baoli has quit IRC | 00:58 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Revert "Infra-cloud-west is currently offline" https://review.openstack.org/286338 | 00:58 |
| jhesketh | nibalizer: yep, that's what I did yesterday | 00:58 |
| *** esikachev has joined #openstack-infra | 00:59 | |
| *** zhurong has joined #openstack-infra | 00:59 | |
| *** chenli has joined #openstack-infra | 00:59 | |
| *** ajmiller has quit IRC | 00:59 | |
| *** arif-ali has joined #openstack-infra | 01:00 | |
| *** flepied has quit IRC | 01:00 | |
| *** thorst_ has quit IRC | 01:00 | |
| asselin_ | hi, I'm trying to figure out why a job couldn't clone from a zuul merger. I see this 404 in the apache logs: [25/Feb/2016:14:34:10 +0000] "GET /p/xyz/abcdef/info/refs?service=git-upload-pack HTTP/1.1" 404 279 "-" "git/1.9.1" | 01:01 |
| *** ybathia has joined #openstack-infra | 01:01 | |
| *** amotoki has joined #openstack-infra | 01:01 | |
| *** thorst has joined #openstack-infra | 01:01 | |
| *** asselin__ has joined #openstack-infra | 01:01 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Revert "Remove infracloud-west" https://review.openstack.org/286340 | 01:02 |
| anteaya | asselin_: can you offer more of the log, perhaps in a paste? | 01:02 |
| asselin_ | I saw another failed clone with this 500 error in the apache access log: "POST /p/xyz/abcdef/git-upload-pack HTTP/1.1" 500 841 "-" "git/1.9.1" | 01:02 |
| jhesketh | nibalizer: do you have the correct cloud.yaml change up for review? | 01:02 |
| asselin_ | anteaya, it's just that one line | 01:03 |
| nibalizer | ua 286337 | 01:03 |
| anteaya | asselin_: interesting | 01:03 |
| anteaya | asselin_: those errors aren't ringing any bells for me | 01:03 |
| asselin_ | anteaya, well...this is apache log file...so one line per request. the other lines aren't related. they're 200 OK responses. | 01:03 |
| anteaya | asselin_: ah okay | 01:03 |
| *** esikachev has quit IRC | 01:04 | |
| fungi | asselin_: i wouldn't expect a job to clone from a zuul merger, only fetch zuul refs from one | 01:04 |
| asselin_ | fungi, you're right, they are git fetch: git fetch | 01:05 |
| *** asselin__ has quit IRC | 01:06 | |
| *** apoorvad has quit IRC | 01:06 | |
| *** aeng has quit IRC | 01:06 | |
| jeblair | nibalizer: comment on 337 | 01:06 |
| nibalizer | jeblair: ack | 01:07 |
| jeblair | nibalizer: i lean toward leaving the cert in place | 01:07 |
| jeblair | since everything else is in place | 01:07 |
| jeblair | but it's kind of a toss-up | 01:07 |
| *** pvaneck has quit IRC | 01:07 | |
| nibalizer | jeblair: so my thinking is when we get the servers back we will not call them 'west' or 'east' so anything with that name (including dns) should be wiped out | 01:07 |
| fungi | asselin_: so... depending on the job and the repos in use, it's not guaranteed that the zuul merger will have an appropriate ref for any given repo | 01:08 |
| jeblair | nibalizer: yeah, if you want to go the other way (remove from all-clouds) as a first step in that direction, i'll be +2 on that too | 01:08 |
| nibalizer | ya im gonn remove it from all-clouds then | 01:08 |
| jeblair | kk | 01:08 |
| fungi | asselin_: for example, devstack-gate will try to retrieve a zuul ref for basically ever source repo it's integrating, but very few of those are actually expected to have that ref | 01:09 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Remove infracloudwest from ansible-clouds.yaml https://review.openstack.org/286337 | 01:09 |
| *** sputnik13 has quit IRC | 01:09 | |
| fungi | asselin_: and git doesn't have a "check if this ref exists in the remote" feature, other than to try (and potentially fail) to fetch it | 01:09 |
| *** thorst has quit IRC | 01:09 | |
| asselin_ | fungi, true, except that the ref is there and needs to be there. Theses failures are failing the job. | 01:09 |
| *** kzaitsev_mb has quit IRC | 01:09 | |
| asselin_ | (very intermittently) | 01:10 |
| *** jamesmcarthur has joined #openstack-infra | 01:10 | |
| jeblair | jhesketh: i -1d your changes; now that it's out of nodepool, better to just leave it that way | 01:10 |
| jhesketh | jeblair: yep, I think I agree :-) | 01:10 |
| jeblair | jhesketh: also, as nibalizer says above, we probably won't call them west anyway :) | 01:10 |
| fungi | vanilla openstack! | 01:10 |
| *** AndyU has joined #openstack-infra | 01:11 | |
| jeblair | jhesketh: (we will probably have multiple regions in the same data center, and ... yeah, vanilla :) | 01:11 |
| nibalizer | strawberry openstack! | 01:11 |
| asselin_ | fungi, I'm hoping that 500 & 404 errors with thos URLs means something to someone so I can dig deeper..... | 01:11 |
| jeblair | chocolate is my favorite! | 01:11 |
| fungi | infracloud-several-feet-west-of-the-other-racks | 01:11 |
| anteaya | ha ha ha | 01:11 |
| jhesketh | why not use gps coords? | 01:11 |
| jhesketh | that would be safe from physical attacks... | 01:11 |
| anteaya | then someone moves a rack | 01:12 |
| jeblair | fungi: infracloud-pod46row7b | 01:12 |
| anteaya | and we have to redo the config | 01:12 |
| fungi | extreeeeeeemly precise gps coordinates, at floor tile granularity | 01:12 |
| nibalizer | jeblair: jhesketh 286337 reupped | 01:12 |
| jeblair | +2 | 01:12 |
| fungi | asselin_: so, the mergers should be getting gearman work requests to create the merges, and then signal back to the scheduler that they succeeded, before it will issue a work request for the jobs which use that merge | 01:13 |
| jhesketh | nibalizer: +w'd | 01:13 |
| *** Qiming has joined #openstack-infra | 01:13 | |
| fungi | asselin_: are you able to fetch the refs mentioned? do they actually exist but the jobs are not succeeding in getting them? or are they really not there? | 01:14 |
| asselin_ | fungi, the refs exists | 01:14 |
| nibalizer | okay i am reapplying the manual application of 337 because puppet put it back and broke itself again | 01:14 |
| asselin_ | fungi, let me get you a more complete paste of log files with my analysis.... | 01:15 |
| fungi | asselin_: so your jobs are occasionally getting a 404 trying to retrieve a ref which actually exists when you try to fetch it yourself from the same zuul merger? | 01:15 |
| asselin_ | actually I didn't try to fetch it...I just checked it was there in the git filesystem | 01:16 |
| asselin_ | but that's a good idea | 01:16 |
| *** ybathia has quit IRC | 01:16 | |
| *** apoorvad has joined #openstack-infra | 01:17 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:17 | |
| asselin_ | fungi, http://paste.openstack.org/show/488667/ | 01:18 |
| openstackgerrit | Merged openstack-infra/system-config: OpenstackId relase 1.0.12 https://review.openstack.org/286315 | 01:20 |
| *** pfallenop has quit IRC | 01:20 | |
| anteaya | asselin_: that is a great paste | 01:20 |
| *** angdraug has quit IRC | 01:20 | |
| *** yamamoto_ has quit IRC | 01:23 | |
| *** Daisy has joined #openstack-infra | 01:23 | |
| *** aeng has joined #openstack-infra | 01:23 | |
| fungi | asselin_: i agree the debug log concurs with what you found on the filesystem | 01:23 |
| asselin_ | anteaya, fungi full paste of the 500 error issue: http://paste.openstack.org/show/488668/ | 01:26 |
| *** yamamoto has joined #openstack-infra | 01:26 | |
| chenli | hello, anyone has comment on this : http://lists.openstack.org/pipermail/openstack-dev/2016-February/087522.html | 01:27 |
| chenli | Can ovs repository added to \$PROJECTS variable in the job definition ? | 01:27 |
| *** Daisy has quit IRC | 01:27 | |
| *** sripriya has quit IRC | 01:31 | |
| fungi | asselin_: also the zm debug log looks similar to our production mergers, which i have not seen any indication of experiencing the issue you've seen on yours | 01:31 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Remove QA health link from status page https://review.openstack.org/286350 | 01:32 |
| asselin_ | fungi, any insight as to what those urls are doing? | 01:35 |
| *** tphummel has quit IRC | 01:35 | |
| *** [1]Thelo has joined #openstack-infra | 01:37 | |
| asselin_ | info/refs?service=git-upload-pack 404 ? | 01:37 |
| asselin_ | GET ^^ | 01:38 |
| fungi | asselin_: you're using distro packages for your git, not any custom-compiled git? unlikely but checking to be thorough | 01:38 |
| *** markvoelker has joined #openstack-infra | 01:38 | |
| asselin_ | I can't imagine we're using any custom-compiled anything.... | 01:39 |
| *** Thelo has quit IRC | 01:39 | |
| *** [1]Thelo is now known as Thelo | 01:39 | |
| fungi | just digging through various problems with git-upload-pack 404 and 500 responses | 01:40 |
| asselin_ | what exactly? I'm interested to know where to look. just google? | 01:41 |
| *** sam_wan has joined #openstack-infra | 01:41 | |
| fungi | yeah, web searches | 01:43 |
| fungi | you've got apache's mod_cgi enabled? | 01:43 |
| *** bgaifullin has quit IRC | 01:43 | |
| asselin_ | checking | 01:43 |
| fungi | i'm not really finding much anything that could cause this to be intermittent, but i suppose flawed fallback behaviors could provide an explanation | 01:44 |
| *** apoorvad has quit IRC | 01:44 | |
| nibalizer | ansible is chugging along nicely now | 01:45 |
| *** thorst has joined #openstack-infra | 01:45 | |
| *** jamielennox is now known as jamielennox|away | 01:45 | |
| fungi | thanks, nibalizer! | 01:45 |
| *** yamamoto has quit IRC | 01:45 | |
| asselin_ | fungi, http://paste.openstack.org/show/488671/ | 01:45 |
| nibalizer | fungi: can you peek at the emergency file and remove aanything that doesn't need to be there | 01:45 |
| *** Keedya_ has joined #openstack-infra | 01:46 | |
| *** Jeffrey4l has joined #openstack-infra | 01:46 | |
| *** kingia has quit IRC | 01:47 | |
| fungi | asselin_: yeah, looks similar to ours http://paste.openstack.org/show/488673 | 01:48 |
| asselin_ | as expected :) | 01:48 |
| *** watanabe_isao has joined #openstack-infra | 01:50 | |
| craige | o/ | 01:51 |
| fungi | nibalizer: i don't see anything in there i know for sure should be removed, except maybe controller00.hpuseast.ic.openstack.org if we've also removed that region from clouds.yaml | 01:51 |
| nibalizer | fungi: ok | 01:51 |
| fungi | nibalizer: clarkb probably knows why logstash-worker20.openstack.org is in there | 01:51 |
| *** sarob has quit IRC | 01:51 | |
| nibalizer | we can take out cacti right? | 01:51 |
| clarkb | because it is running logstash 2.0 | 01:51 |
| nibalizer | it was just disabled because we wanted to GoFast(TM) | 01:52 |
| clarkb | and I haven't been able to get reviews on the changes to switch everything to 2.0 | 01:52 |
| anteaya | nibalizer: what is the emergency file? | 01:52 |
| fungi | nibalizer: cacti.openstack.org is in there until we stop snmp reindexing from puppet exec | 01:52 |
| fungi | nibalizer: and i think puppet on afstest.openstack.org is probably just plain broken | 01:52 |
| nibalizer | https://review.openstack.org/284466 is a stab at that | 01:52 |
| *** andreykurilin__ has quit IRC | 01:53 | |
| prometheanfire | SpamapS: why didn't you +w this? https://review.openstack.org/281960 | 01:53 |
| fungi | anteaya: the non-puppeted skip file at /etc/ansible/hosts/emergency on puppetmaster.o.o | 01:53 |
| anteaya | fungi: thanks | 01:53 |
| *** thorst has quit IRC | 01:53 | |
| clarkb | nibalizer: https://review.openstack.org/#/c/285473/4 adds osic creds | 01:53 |
| clarkb | nibalizer: if you are in a reviewing mood | 01:54 |
| fungi | anteaya: it's our workaround for timing and/or catch-22 issues with disabling puppet for select hosts through git | 01:54 |
| anteaya | prometheanfire: noone is obliged to +w anything they don't want to | 01:54 |
| nibalizer | clarkb: got the hiera keys set? | 01:54 |
| fungi | which means we should probably move these entries to the proper host list if they're going to stay disabled for a while | 01:54 |
| prometheanfire | anteaya: I know, but it'd be nice to know why, know if I need to do anything | 01:54 |
| anteaya | prometheanfire: just because they can +A after another core +2'd a patch doesn't mean they are obligated to | 01:54 |
| nibalizer | they don't have dns? | 01:54 |
| *** mtanino has quit IRC | 01:55 | |
| anteaya | prometheanfire: well asking for a reason is fine, but it is the reviewer's choice | 01:55 |
| prometheanfire | anteaya: I understand, but can I not ask for clarification? or am I not allowed to talk to them? | 01:55 |
| anteaya | fungi: that makes sense, thank you | 01:55 |
| anteaya | prometheanfire: you certainly may ask questions as may anyone | 01:56 |
| prometheanfire | ok, | 01:56 |
| fungi | asselin_: i'm at a loss. did you try fetching the same refs remotely? i can't recall if you had an example of doing that to see whether a given ref was only temporarily failing or indefinitely failing | 01:56 |
| *** yamamoto has joined #openstack-infra | 01:57 | |
| *** yamamoto has quit IRC | 01:58 | |
| *** sarob has joined #openstack-infra | 01:58 | |
| *** doug-fish has quit IRC | 01:58 | |
| asselin_ | fungi, both work | 01:59 |
| fungi | yeah, i'm out of ideas. looked in syslog/dmesg for filesystem issues? | 02:00 |
| asselin_ | good idea | 02:00 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Add support for layout configuration split https://review.openstack.org/152290 | 02:01 |
| fungi | asselin_: is it a network-attached filesystem maybe? | 02:01 |
| asselin_ | not sure: http://paste.openstack.org/show/488674/ | 02:01 |
| asselin_ | no...openstack base filesystem | 02:02 |
| asselin_ | no cinder volumes or anything like that | 02:02 |
| *** aeng has quit IRC | 02:02 | |
| clarkb | nibalizer: yes keys are all set | 02:02 |
| pleia2 | sprint blogged http://princessleia.com/journal/?p=11335 (it'll go to openstack planet too) | 02:03 |
| jamesmcarthur | openstackID is back up | 02:03 |
| Keedya_ | anteaya: hello | 02:03 |
| *** sarob has quit IRC | 02:04 | |
| *** rhallisey has quit IRC | 02:04 | |
| fungi | asselin_: dmesg -T is probably more helpful, to get timestamps translated from relative to absolute, but none of those entries look related anyway | 02:04 |
| *** dims has quit IRC | 02:04 | |
| * asselin_ learned something new | 02:05 | |
| *** reed_ has quit IRC | 02:05 | |
| Keedya_ | anteaya, clarkb I am hoping to push the new project (shovel plugin) to Openstack | 02:05 |
| asselin_ | feb 16 & 17....so nothing for quite a few days | 02:06 |
| fungi | yeah, don't completely trust the times reported by -T either since it's still based on kernel ticks since boot and can drift pretty significantly from localtime (at least on on-systemd platforms) | 02:06 |
| jamesmcarthur | fungi: mordred: jeblair: jpmaxman: smarcet: I’d love to talk about how to make this easier for all parties moving forward. I realize Infra isn’t available to jump at our every request. At the same time, having to wait 3 hours for a fix is madenning when a website is down. | 02:06 |
| clarkb | fungi: if you ever suspend dmesg -T is very wrong | 02:07 |
| clarkb | fungi: because kernel isn't running while suspended | 02:07 |
| *** rguillebert has quit IRC | 02:07 | |
| fungi | yep, hopefully servers don't get suspended too often | 02:07 |
| fungi | but clouds _do_ actually do that to virtual machines | 02:08 |
| *** markvoelker has quit IRC | 02:08 | |
| fungi | luckily usually not for days at a time | 02:08 |
| *** markvoelker has joined #openstack-infra | 02:08 | |
| *** Sukhdev has joined #openstack-infra | 02:08 | |
| *** dims has joined #openstack-infra | 02:08 | |
| *** baoli has joined #openstack-infra | 02:11 | |
| asselin_ | fungi, ok...well git fetch results in the GET ...git-upload-pack 200 request | 02:11 |
| fungi | jamesmcarthur: i agree, we can encourage more integrated participation from both directions, and find a way to improve the current configuration management for those systems to be more robust and better tested. with automated deployment, we should be able to make deployment validation and functional testing of these changes possible so we avoid as many emergencies | 02:13 |
| anteaya | pleia2: nice post | 02:13 |
| anteaya | pleia2: bunnies! | 02:13 |
| asselin_ | actually, trying the other I got a GET followed by a POST | 02:13 |
| *** dims has quit IRC | 02:13 | |
| pleia2 | anteaya: thanks :) | 02:14 |
| *** Sukhdev has quit IRC | 02:14 | |
| asselin_ | http://paste.openstack.org/show/488676/ | 02:14 |
| fungi | asselin_: are those systems heavily loaded when you hit these errors? maybe the git backend is trying to create packfiles but take too long to complete and end up returning an error at the apache layer? | 02:14 |
| fungi | that would explain why you only issue a get later when you retry (packfile exists at that point and is returned directly) | 02:15 |
| nibalizer | pleia2: awesome! | 02:15 |
| asselin_ | fungi, can you elaborate what you mean by 'systems' and 'git backend'? I'm not familiar with the flow. It is possible the system was loaded at the time...I don't know how to check for that historically though. | 02:16 |
| fungi | asselin_: the server where your zuul-merger's apache is running | 02:17 |
| *** dims has joined #openstack-infra | 02:17 | |
| fungi | asselin_: the git backend cgi called from apache will serve up packfiles if they exist, or create them first if they don't | 02:17 |
| asselin_ | fungi, in theory if they were loaded, what would be the fix? have another zuul merger? | 02:18 |
| *** baoli has quit IRC | 02:18 | |
| *** baoli has joined #openstack-infra | 02:18 | |
| asselin_ | ok and the packfiles are on the local filessystem? | 02:18 |
| fungi | asselin_: well, if you see your zuul mergers under significant load, it's at least designed to be broadly scalable by simply booting more mergers | 02:18 |
| *** aeng has joined #openstack-infra | 02:18 | |
| fungi | yeah, packfiles are in the git repos. they're basically aggregations of refs | 02:19 |
| fungi | they're an abstraction which will be created within a copy of the repo, so in this case by the git-smart-http backend cgi in response to requests for some unpacked refs | 02:19 |
| anteaya | I have to go to bed now in order to get up in the middle of the night | 02:19 |
| anteaya | g'night | 02:20 |
| fungi | g'night, anteaya! | 02:20 |
| asselin_ | anteaya, good night | 02:20 |
| anteaya | night | 02:20 |
| *** zhurong has quit IRC | 02:20 | |
| *** zhurong has joined #openstack-infra | 02:21 | |
| *** baoli has quit IRC | 02:22 | |
| *** baoli has joined #openstack-infra | 02:23 | |
| fungi | asselin_: maybe someone else has alternative ideas, but i'm wiped. need to kick back and get some sleep. good luck tracking it down--this one seems more elusive than usual | 02:24 |
| asselin_ | fungi, thanks for you help | 02:24 |
| asselin_ | good night | 02:24 |
| *** bpokorny_ has joined #openstack-infra | 02:26 | |
| *** fawadkhaliq has quit IRC | 02:27 | |
| *** Keedya_ has quit IRC | 02:28 | |
| *** bpokorny has quit IRC | 02:29 | |
| *** bpokorny_ has quit IRC | 02:31 | |
| *** chenli has quit IRC | 02:32 | |
| *** blogan has quit IRC | 02:32 | |
| *** ptoohill has quit IRC | 02:32 | |
| *** Keedya_ has joined #openstack-infra | 02:33 | |
| *** x00350071 is now known as xiangxinyong | 02:34 | |
| *** flepied has joined #openstack-infra | 02:34 | |
| *** andymaier has quit IRC | 02:34 | |
| *** sridhar_ram1 has quit IRC | 02:35 | |
| *** armax has quit IRC | 02:35 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:35 | |
| *** yamamoto has joined #openstack-infra | 02:42 | |
| *** Keedya_ has quit IRC | 02:45 | |
| *** Keedya_ has joined #openstack-infra | 02:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: fix zuul/layout for puppet-heat https://review.openstack.org/286362 | 02:48 |
| *** Keedya_ has quit IRC | 02:51 | |
| *** thorst has joined #openstack-infra | 02:51 | |
| *** kingia has joined #openstack-infra | 02:54 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove infracloudwest from ansible-clouds.yaml https://review.openstack.org/286337 | 02:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/gerritbot: Add change-created event type https://review.openstack.org/286366 | 02:56 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Fix heat create_stack and delete_stack https://review.openstack.org/276045 | 02:57 |
| *** thorst has quit IRC | 02:58 | |
| *** esikachev has joined #openstack-infra | 02:59 | |
| *** baoli has quit IRC | 03:00 | |
| *** rockyg has joined #openstack-infra | 03:01 | |
| *** sripriya has joined #openstack-infra | 03:01 | |
| *** dimtruck is now known as zz_dimtruck | 03:01 | |
| *** esikachev has quit IRC | 03:04 | |
| *** kzaitsev_mb has quit IRC | 03:05 | |
| *** rockyg has quit IRC | 03:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add OSIC clouds.yaml details https://review.openstack.org/285473 | 03:07 |
| clarkb | now with less merge conflict | 03:07 |
| *** ajmiller has joined #openstack-infra | 03:08 | |
| craige | ping jhesketh or any other infra root folk that may be lurking. | 03:10 |
| jhesketh | craige: pong | 03:10 |
| * craige is in need of another copy storyboard.sql | 03:10 | |
| craige | Pretty please :-D | 03:10 |
| jhesketh | hmm I haven't done that before and not sure what might need scrubbing etc | 03:11 |
| craige | oh, I thought you'd been shown :-/ | 03:11 |
| jhesketh | no, sorry | 03:11 |
| *** zz_dimtruck is now known as dimtruck | 03:12 | |
| craige | IIRC there was a script that dumps it to a location I can snavel it form | 03:12 |
| * craige checks system-config/tools | 03:12 | |
| jhesketh | hmm, might be best to wait for somebody with experience? | 03:12 |
| craige | Perhaps but jeblair was clear anyone can run it. | 03:12 |
| * craige can't seeit though. | 03:13 | |
| craige | so I may have it all wrong :-D | 03:13 |
| *** Keedya_ has joined #openstack-infra | 03:15 | |
| *** gildub has joined #openstack-infra | 03:15 | |
| *** woodster_ has quit IRC | 03:16 | |
| *** Keedya_ has quit IRC | 03:17 | |
| *** dims has quit IRC | 03:19 | |
| *** sripriya has quit IRC | 03:19 | |
| *** yuanying has joined #openstack-infra | 03:21 | |
| *** yuanying_ has quit IRC | 03:23 | |
| *** Keedya_ has joined #openstack-infra | 03:23 | |
| *** baoli has joined #openstack-infra | 03:25 | |
| clarkb | craige: maybe it was in puppet-storyboard/ | 03:26 |
| *** Keedya_ has quit IRC | 03:27 | |
| *** yuanying has quit IRC | 03:27 | |
| *** kencjohnston has joined #openstack-infra | 03:29 | |
| craige | Perhaps clarkb | 03:31 |
| *** Nakato has quit IRC | 03:31 | |
| * craige looks | 03:31 | |
| clarkb | jhesketh: if you get a moment 285473 has been +2'd by mordred and nibalizer at different times but run into rebase fun, is first step in using osic | 03:32 |
| *** rossella_s has quit IRC | 03:32 | |
| *** rossella_s has joined #openstack-infra | 03:33 | |
| clarkb | with that in I can use https://review.openstack.org/#/c/285477/4 to boot a mirror node in osic then we can point nodepool at it | 03:34 |
| *** sarob has joined #openstack-infra | 03:38 | |
| *** sarob has quit IRC | 03:42 | |
| *** Nakato has joined #openstack-infra | 03:45 | |
| *** watanabe_isao has quit IRC | 03:46 | |
| ianw | clarkb: no known issues with uploading images that you know of? trying to figure out a f23 failure that only seems to replicate in upstream and want to make sure i'm not running env too different ... | 03:46 |
| *** watanabe_isao has joined #openstack-infra | 03:46 | |
| clarkb | ianw: not tht I know of but havent really looked in a week or so | 03:46 |
| jhesketh | clarkb: looking now | 03:46 |
| ianw | i know the builds are ok, which is half-way :) | 03:47 |
| ianw | otherwise, i've got some issue on my hand that only replicates in upstream and involves some intersection of polkit, systemd & dbus. any of those are unfun to debug, all three together... | 03:48 |
| *** rlandy has quit IRC | 03:48 | |
| *** baoli has quit IRC | 03:49 | |
| ianw | if any system-config people feel like looking at -> https://review.openstack.org/#/c/285876/ (fix pypi install of requests) that would be great too | 03:49 |
| *** links has joined #openstack-infra | 03:49 | |
| craige | no luck finding to anywhere else, clarkb, jhesketh. | 03:50 |
| craige | I believe jeblair had it on the storyboard host... | 03:50 |
| *** thorst has joined #openstack-infra | 03:56 | |
| clarkb | ianw: this isn't something that can be fixed in fedora I take it? | 03:58 |
| clarkb | (it has been a known issue for a couple years now I think( | 03:58 |
| *** chenli has joined #openstack-infra | 03:59 | |
| ianw | clarkb: i think mostly the problem is that the un-vendored directories are turned into symlinks | 03:59 |
| jhesketh | craige: righto.. unfortuantely I'm not really comfortable with doing that sorry. Might be best to ask fungi or jeblair when they are around | 03:59 |
| craige | Yep, understood jhesketh | 03:59 |
| ianw | clarkb: and rpm barfs at overwriting a symlink with a directory from a package | 03:59 |
| clarkb | ianw: the underlying issue is that fedora/centos/rhel don't pip install to /usr/local | 03:59 |
| clarkb | so yum/dnf and pip fight over the same paths | 04:00 |
| ianw | clarkb: well, yeah, that ... but that's not going to change quickly | 04:00 |
| ianw | even just shipping the files list with packages would allow pip to uninstall them correctly & overwrite | 04:00 |
| *** kzaitsev_mb has joined #openstack-infra | 04:02 | |
| *** chlong_ has quit IRC | 04:03 | |
| *** kencjohnston has quit IRC | 04:03 | |
| *** _amrith_ is now known as amrith | 04:03 | |
| *** thorst has quit IRC | 04:03 | |
| ianw | in *most* cases we have an uneasy truce ... but when directories are replaced with symlinks, things just start getting confused | 04:04 |
| *** kzaitsev_mb has quit IRC | 04:07 | |
| *** bpokorny has joined #openstack-infra | 04:07 | |
| *** chenli has quit IRC | 04:07 | |
| *** yuanying has joined #openstack-infra | 04:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add a dib-builddate file https://review.openstack.org/286374 | 04:12 |
| ianw | clarkb / jhesketh : ^ i think this would allow me to not bother you when i'm checking what image version tests are running on | 04:13 |
| *** armax has joined #openstack-infra | 04:15 | |
| *** yamahata has joined #openstack-infra | 04:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add a dib-builddate file https://review.openstack.org/286374 | 04:16 |
| jhesketh | ianw: cool, lgtm | 04:16 |
| *** fawadkhaliq has joined #openstack-infra | 04:18 | |
| clarkb | ianw: +A | 04:19 |
| *** dimtruck is now known as zz_dimtruck | 04:22 | |
| *** sam_wan has quit IRC | 04:32 | |
| *** chenli has joined #openstack-infra | 04:36 | |
| *** sam_wan has joined #openstack-infra | 04:45 | |
| *** amrith is now known as _amrith_ | 04:46 | |
| clarkb | ianw: image uplaods in rax look staleish | 04:46 |
| clarkb | OpenStackCloudException: Failed at action (create_container) [No tenant specified] | 04:46 |
| clarkb | http://paste.openstack.org/show/488685/ | 04:46 |
| clarkb | looks like we are attempting to reauth now with newer oscc but aren't providing enough information to do so? | 04:47 |
| *** tiswanso has quit IRC | 04:47 | |
| clarkb | previously we would just fail on our auth being invalid so this is progress I suppose | 04:47 |
| *** sridhar_ram1 has joined #openstack-infra | 04:50 | |
| ianw | clarkb: ok, good (?) it would explain things for me ... | 04:58 |
| ianw | if i can help... | 04:58 |
| *** fedexo has joined #openstack-infra | 04:59 | |
| clarkb | ianw: mostly trying to figure out why the tenant/project would be missing and not seeing it | 05:00 |
| clarkb | this is a trickyish thing becuase it involves token expiration | 05:01 |
| clarkb | against a real cloud | 05:01 |
| clarkb | that is the only cloud that does image uploads in this manner | 05:01 |
| *** kingia_ has joined #openstack-infra | 05:01 | |
| *** thorst has joined #openstack-infra | 05:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:04 | |
| *** kingia has quit IRC | 05:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add a dib-builddate file https://review.openstack.org/286374 | 05:07 |
| *** thorst has quit IRC | 05:08 | |
| clarkb | jhesketh: good point on fs detection, I mostly just modified what was already there but ther eprobably is a better way to determine that "this isn't just a raw block device we should leave it be" | 05:09 |
| *** kzaitsev_mb has quit IRC | 05:09 | |
| clarkb | re 285477 | 05:09 |
| jhesketh | righto | 05:14 |
| *** ajmiller has quit IRC | 05:16 | |
| *** blogan_ has joined #openstack-infra | 05:18 | |
| *** salv-orl_ has joined #openstack-infra | 05:19 | |
| *** ptoohill has joined #openstack-infra | 05:20 | |
| *** bpokorny has quit IRC | 05:21 | |
| *** salv-orlando has quit IRC | 05:22 | |
| *** jamesmcarthur has quit IRC | 05:28 | |
| *** abregman has joined #openstack-infra | 05:35 | |
| *** jogo has quit IRC | 05:38 | |
| *** oomichi_ has joined #openstack-infra | 05:40 | |
| *** oomichi_ has quit IRC | 05:40 | |
| *** jogo has joined #openstack-infra | 05:41 | |
| *** fawadkhaliq has quit IRC | 05:42 | |
| *** kushal has joined #openstack-infra | 05:43 | |
| *** chenli has quit IRC | 05:45 | |
| clarkb | ianw: at this point probably the best thing would be to incrase the verbosity of shade, swiftclient, and os-client-config logging when running within the nodepool builder | 05:46 |
| clarkb | ianw: I can work on that patch in the morning (builders have their own config now so yay) | 05:46 |
| *** jaosorior has joined #openstack-infra | 05:49 | |
| *** esker has quit IRC | 05:49 | |
| *** esker has joined #openstack-infra | 05:50 | |
| *** Sukhdev has joined #openstack-infra | 05:55 | |
| *** chenli has joined #openstack-infra | 05:55 | |
| *** fawadkhaliq has joined #openstack-infra | 05:58 | |
| *** esikachev has joined #openstack-infra | 05:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/286393 | 06:01 |
| *** Daisy_ has joined #openstack-infra | 06:02 | |
| *** lucasagomes has quit IRC | 06:02 | |
| *** lucasagomes has joined #openstack-infra | 06:02 | |
| *** Daisy_ has quit IRC | 06:03 | |
| *** esikachev has quit IRC | 06:04 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:05 | |
| *** thorst has joined #openstack-infra | 06:06 | |
| *** Kiall has quit IRC | 06:06 | |
| *** Kiall has joined #openstack-infra | 06:08 | |
| *** rcernin has joined #openstack-infra | 06:09 | |
| *** kzaitsev_mb has quit IRC | 06:10 | |
| *** sdake has joined #openstack-infra | 06:11 | |
| *** sridhar_ram1 has quit IRC | 06:12 | |
| *** thorst has quit IRC | 06:13 | |
| tobiash_ | jhesketh, jeblair: I need to add ProxyCommand support for connecting a zuul instance to our gerrit. What option would be favoured by you? Create a config option for specifying this in zuul. conf or add support for ~/.ssh/config? | 06:17 |
| clarkb | tobiash_: this comes up fairly frequently and my typical suggestion is to not solve this in zuul directly | 06:18 |
| clarkb | you can typically run the proxy setup external to zuul and foraard through it for zuul | 06:19 |
| tobiash_ | clarkb: thx, I'll try this | 06:20 |
| clarkb | unlike say http(s) there aren't cobsistent and reliable methods for proxying ssh so it can get complicated to handle all the cases | 06:20 |
| *** e0ne has joined #openstack-infra | 06:23 | |
| *** lucasagomes has quit IRC | 06:24 | |
| tobiash_ | I use corkscrew for proxying it over an http proxy | 06:25 |
| tobiash_ | works well with normal ssh and proper ssh config | 06:25 |
| jhesketh | +1 to what clarkb said | 06:26 |
| tobiash_ | does zuul manage a single connection to gerrit? | 06:26 |
| *** lucasagomes has joined #openstack-infra | 06:27 | |
| *** sdake has quit IRC | 06:27 | |
| *** e0ne has quit IRC | 06:31 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/285901 | 06:31 |
| ianw | clarkb: ok, ... i do have a rax account so i guess i could setup things to upload and see, but it would take me a fair bit to context switch in | 06:32 |
| *** sdake has joined #openstack-infra | 06:33 | |
| *** e0ne has joined #openstack-infra | 06:33 | |
| *** e0ne has quit IRC | 06:34 | |
| *** dmellado has quit IRC | 06:34 | |
| *** dmellado has joined #openstack-infra | 06:39 | |
| *** roxanaghe has quit IRC | 06:39 | |
| *** Sukhdev has quit IRC | 06:45 | |
| *** harlowja_at_home has quit IRC | 06:49 | |
| *** asalkeld has joined #openstack-infra | 06:52 | |
| jamespage | AJaeger, morning - this may be a ignorant question but what happens next with regards https://review.openstack.org/#/c/232705/ ? | 06:52 |
| *** bgaifullin has joined #openstack-infra | 06:53 | |
| *** maishsk has joined #openstack-infra | 06:53 | |
| pleia2 | jamespage: the commit message leads me to believe the change in question is being moved forward with, but it's been Abandoned, can you adjust the commit message for accuracy? | 06:54 |
| pleia2 | jamespage: I *believe* you're simply moving Juju charms into the namespace, but not yet going for TC approval | 06:55 |
| pleia2 | aside from that, you just need another +2 and approval from one of us infra-core people | 06:55 |
| pleia2 | (or project-config core) | 06:55 |
| jamespage | pleia2, that's correct - TC wanted us to move dev first before assessing as a formal project | 06:56 |
| *** Daisy has joined #openstack-infra | 06:56 | |
| * pleia2 nods | 06:56 | |
| jamespage | pleia2, the project change is currently abandoned/deffered for now | 06:57 |
| *** Daisy has quit IRC | 06:58 | |
| jamespage | pleia2, but i will be re-ssurecting that once we move into the openstack namespace... | 06:58 |
| *** achanda has quit IRC | 06:58 | |
| jamespage | pleia2, do you still need me to amend the commit message? I think the reference is still pertinent | 06:58 |
| nibalizer | jamespage: happy to help you out | 07:00 |
| pleia2 | I suppose it's ok | 07:00 |
| pleia2 | having a look at the rest of the change now | 07:01 |
| nibalizer | jamespage: so are you super super dooper sure those names are right | 07:01 |
| nibalizer | because renaming them is a royal pain | 07:01 |
| jamespage | nibalizer, yup | 07:01 |
| pleia2 | I'm checking all the github links too ;) | 07:01 |
| jamespage | oh one of the checks does that | 07:02 |
| pleia2 | I meant, matching with naming, etc | 07:03 |
| nibalizer | pleia2: this lgtm | 07:03 |
| *** maishsk has quit IRC | 07:03 | |
| nibalizer | ill only +2 if you're going to vote tonight | 07:03 |
| pleia2 | nibalizer: +2ed | 07:03 |
| nibalizer | +a | 07:03 |
| *** korzen has joined #openstack-infra | 07:03 | |
| pleia2 | I should go to BED tonight | 07:03 |
| pleia2 | :D | 07:03 |
| nibalizer | orrrrrr we could stay up all night breaking things! | 07:03 |
| pleia2 | haha | 07:04 |
| jamespage | nibalizer, pleia2: thankyou! | 07:04 |
| nibalizer | jamespage: its been approved so you'll have tasty new repos in a bit | 07:04 |
| jamespage | awesome | 07:04 |
| nibalizer | now the trick is to make sure you don't get pull requests and development on the github | 07:04 |
| *** scheuran has joined #openstack-infra | 07:05 | |
| nibalizer | so maybe make those read only or delete all the code or something | 07:05 |
| nibalizer | (dont do this until our stuff has time to grab though) | 07:05 |
| jamespage | nibalizer, they are transitional only and will be dropped post migration | 07:05 |
| nibalizer | great | 07:05 |
| nibalizer | did you have them in lp before? | 07:05 |
| jamespage | nibalizer, yup | 07:05 |
| jamespage | under bzr | 07:05 |
| *** oanson has quit IRC | 07:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:06 | |
| *** Daisy_ has joined #openstack-infra | 07:06 | |
| *** korzen_ has joined #openstack-infra | 07:07 | |
| *** korzen has quit IRC | 07:08 | |
| *** sdake has quit IRC | 07:10 | |
| jamespage | nibalizer, do I get setup automatically with perms for review in gerrit on those repos? I need to setup the current set of committers as well | 07:10 |
| pleia2 | an initial member of the groups will need to be added by one of us | 07:10 |
| nibalizer | no what we'll do is add you to the group then you can add remove folk | 07:10 |
| *** kzaitsev_mb has quit IRC | 07:11 | |
| *** thorst has joined #openstack-infra | 07:11 | |
| nibalizer | jamespage: do you have a twitter | 07:11 |
| *** Daisy_ has quit IRC | 07:11 | |
| pleia2 | he's teh @javacruft | 07:12 |
| *** sdake has joined #openstack-infra | 07:12 | |
| * nibalizer tweets | 07:12 | |
| jamespage | nibalizer, i do and i am javacruft | 07:12 |
| nibalizer | pleia2: you'll love this | 07:12 |
| nibalizer | i had my ubuntu phone for like 24 hours before I briked it | 07:13 |
| pleia2 | haha | 07:13 |
| pleia2 | my Nexus 7 is running a dev iso of it, but I never bought an actual ubuntu phone | 07:14 |
| openstackgerrit | Yuriy Taraday proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests https://review.openstack.org/285620 | 07:15 |
| jamespage | nibalizer, pleia2: i had a bq for a while but found it to hard to travel with | 07:15 |
| jamespage | only being dual band... | 07:15 |
| * pleia2 nods | 07:15 | |
| *** achanda has joined #openstack-infra | 07:15 | |
| *** mikelk has joined #openstack-infra | 07:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 07:17 |
| pleia2 | :) | 07:17 |
| *** thorst has quit IRC | 07:18 | |
| *** mrmartin has joined #openstack-infra | 07:27 | |
| AJaeger | good morning infra | 07:28 |
| AJaeger | jamespage: congrats to your new repos - and thanks pleia2 and nibalizer for approving. | 07:28 |
| AJaeger | ianw: http://docs.openstack.org/infra/manual/creators.html#decide-status-of-your-project is one place that documents official/unofficial projects | 07:28 |
| AJaeger | lifeless: are you still awake and around? I have a pip question regarding constraints... | 07:29 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/zuul: Keep py3.X compatibility for urllib https://review.openstack.org/261173 | 07:31 |
| *** bnemec has quit IRC | 07:36 | |
| *** bnemec has joined #openstack-infra | 07:39 | |
| *** roxanaghe has joined #openstack-infra | 07:39 | |
| yolanda | good morning | 07:40 |
| yolanda | yay, jamespage, charms! | 07:40 |
| max_lobur | Good Morning All | 07:40 |
| *** vgridnev has joined #openstack-infra | 07:41 | |
| max_lobur | Can someone take a look at the project-config patch please https://review.openstack.org/#/c/281301/ It needs another +2 | 07:41 |
| jamespage | AJaeger, thanks - and thanks for your patient reviews :-) | 07:42 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/286393 | 07:42 |
| openstackgerrit | Merged openstack-infra/project-config: Publish bashate docs https://review.openstack.org/286305 | 07:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Remove check requirements from Kingbird" https://review.openstack.org/285356 | 07:43 |
| lifeless | AJaeger: ish; go on? | 07:44 |
| *** roxanaghe has quit IRC | 07:44 | |
| AJaeger | lifeless: cinder is using constraints now in their tox.ini and they run into problems with older pip | 07:44 |
| AJaeger | lifeless: I added you to an email as well for async communication | 07:45 |
| AJaeger | lifeless: http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2016-02-29.log.html#t2016-02-29T20:58:16 | 07:45 |
| AJaeger | I assume they need pip 7.1, correctly? | 07:45 |
| * AJaeger just wonder why it now pops up... | 07:45 | |
| lifeless | AJaeger: yes; we don't support older pips | 07:46 |
| lifeless | AJaeger: there used to be an infra manual that said to upgrade pip and tox and stuff | 07:46 |
| lifeless | AJaeger: its moved now, and there's a pending patch to overhaul recommendations | 07:46 |
| AJaeger | lifeless: do you have a link handy to share with the cinder folks? | 07:47 |
| *** flepied has quit IRC | 07:47 | |
| lifeless | nope | 07:47 |
| lifeless | python.html from memory | 07:48 |
| *** sdake has quit IRC | 07:48 | |
| * AJaeger can't find it... | 07:50 | |
| *** esker has quit IRC | 07:50 | |
| *** jtomasek has joined #openstack-infra | 07:50 | |
| openstackgerrit | Merged openstack-infra/project-config: fix zuul/layout for puppet-heat https://review.openstack.org/286362 | 07:51 |
| AJaeger | lifeless: http://docs.openstack.org/project-team-guide/project-setup/python.html ;) | 07:52 |
| *** maishsk has joined #openstack-infra | 07:52 | |
| lifeless | AJaeger: that actually has different prose to the older doc; there's a pending patch under discussion to get it updated | 07:52 |
| *** esker has joined #openstack-infra | 07:54 | |
| AJaeger | yolanda: could you review https://review.openstack.org/#/c/284371/ as well, please? | 07:55 |
| AJaeger | lifeless: yes https://review.openstack.org/264398 - send out via email. | 07:55 |
| yolanda | sure | 07:55 |
| AJaeger | lifeless: thanks, I'm sure the discussion will continue, let's see in which form later today (or tomorrow for you). | 07:56 |
| *** sam_wan has quit IRC | 07:57 | |
| *** k4n0 has joined #openstack-infra | 07:57 | |
| AJaeger | lifeless: just to double check: 7.1 has oldest support version but 8.x recommended? | 07:58 |
| *** sam_wan has joined #openstack-infra | 07:58 | |
| *** sam_wan has quit IRC | 07:59 | |
| openstackgerrit | Merged openstack-infra/project-config: Add smaug-dashboard and python-smaugclient to gerritbot https://review.openstack.org/286005 | 07:59 |
| *** sam_wan has joined #openstack-infra | 08:00 | |
| *** achanda_ has joined #openstack-infra | 08:02 | |
| AJaeger | yolanda: https://jenkins04.openstack.org/job/gate-openstack-chef-repo-chef-rake-integration-nv/6/console is running enelessly, can you kill 284730. jklare that's your patch | 08:03 |
| AJaeger | yolanda, jklare that change is running for 18 hours and on top of check queue now http://status.openstack.org/zuul/ | 08:04 |
| jklare | AJaeger just saw this, how can i kill it? | 08:04 |
| openstackgerrit | Merged openstack-infra/project-config: Remove obsolete dvipng install https://review.openstack.org/286241 | 08:04 |
| *** achanda has quit IRC | 08:04 | |
| *** achanda_ has quit IRC | 08:04 | |
| jklare | AJaeger it looks like it ran into some jenkins/java exception, but did not error out | 08:04 |
| anteaya | submitting a new patchset should reset the jobs in the check queue | 08:05 |
| *** k4n0 has quit IRC | 08:05 | |
| jklare | ok | 08:05 |
| AJaeger | jklare: please keep an eye open on it - and feel free to come back to ask for debugging help (I can't do it but an infra-root might be) if this continues | 08:06 |
| AJaeger | morning, anteaya. Thanks! | 08:06 |
| *** slagle has quit IRC | 08:06 | |
| anteaya | AJaeger: morning and welcome | 08:06 |
| jklare | AJaeger i think this is due to the bridging of eth1... we need to change that in the recipe | 08:06 |
| *** kzaitsev_mb has joined #openstack-infra | 08:07 | |
| anteaya | if the problem is with the patch, resetting the check jobs won't help | 08:07 |
| anteaya | as the same issue will keep happening | 08:08 |
| *** achanda has joined #openstack-infra | 08:08 | |
| jklare | anteaya i will try to patch the patch :D | 08:08 |
| jklare | anteaya give me a second | 08:08 |
| *** fawadkhaliq has quit IRC | 08:08 | |
| anteaya | take all the time you need | 08:08 |
| anteaya | and good luck | 08:08 |
| *** armax has quit IRC | 08:08 | |
| jklare | anteaya do you know which interface is used to connect to the jenkins master? eth0 or eth1 ? | 08:08 |
| *** pcaruana has joined #openstack-infra | 08:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Added new projects for the OSA role break out https://review.openstack.org/284512 | 08:09 |
| anteaya | I do not know | 08:09 |
| *** xiangxinyong has quit IRC | 08:10 | |
| *** yamahata has quit IRC | 08:11 | |
| *** kzaitsev_mb has quit IRC | 08:12 | |
| AJaeger | jklare: you might have more success on #openstack-qa - and/or later ;( | 08:12 |
| *** aviau has quit IRC | 08:13 | |
| *** aviau has joined #openstack-infra | 08:15 | |
| *** thorst has joined #openstack-infra | 08:16 | |
| *** [HeOS] has quit IRC | 08:17 | |
| *** amotoki has quit IRC | 08:18 | |
| *** HeOS has joined #openstack-infra | 08:18 | |
| *** vgridnev has quit IRC | 08:19 | |
| *** amotoki has joined #openstack-infra | 08:19 | |
| *** andymaier has joined #openstack-infra | 08:19 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 08:20 |
| *** sshnaidm has quit IRC | 08:22 | |
| *** thorst has quit IRC | 08:23 | |
| *** flepied has joined #openstack-infra | 08:24 | |
| *** hichihara has quit IRC | 08:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Add bareon-ironic project https://review.openstack.org/281301 | 08:24 |
| *** achanda has quit IRC | 08:27 | |
| yolanda | AJaeger, taking a look | 08:28 |
| yolanda | ah it was aborted already | 08:29 |
| *** ifarkas has joined #openstack-infra | 08:30 | |
| *** vincentll has joined #openstack-infra | 08:31 | |
| *** ociuhandu has quit IRC | 08:31 | |
| *** jlanoux has joined #openstack-infra | 08:32 | |
| *** kushal has quit IRC | 08:33 | |
| *** watanabe_isao has quit IRC | 08:34 | |
| AJaeger | yolanda: yeah, jklare "fixed" it - thanks | 08:34 |
| *** dizquierdo has joined #openstack-infra | 08:36 | |
| *** daemontool__ has joined #openstack-infra | 08:38 | |
| *** arxcruz has joined #openstack-infra | 08:39 | |
| *** _nadya_ has joined #openstack-infra | 08:39 | |
| *** yaume has joined #openstack-infra | 08:40 | |
| *** roxanaghe has joined #openstack-infra | 08:41 | |
| *** daemontool_ has quit IRC | 08:41 | |
| *** matrohon has joined #openstack-infra | 08:45 | |
| *** roxanaghe has quit IRC | 08:46 | |
| *** salv-orl_ has quit IRC | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #openstack-infra | 08:48 | |
| trash | mordred, SpamapS: Can you please review https://review.openstack.org/#/c/280178 again? | 08:48 |
| *** salv-orlando has joined #openstack-infra | 08:48 | |
| *** dingyichen has quit IRC | 08:50 | |
| *** zeih has joined #openstack-infra | 08:51 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add pylint, coverage, py34 to sahara-tests https://review.openstack.org/284693 | 08:55 |
| *** fedexo has quit IRC | 08:58 | |
| abregman | hi. where can I find the images the gates use? | 08:59 |
| *** andymaier has quit IRC | 09:01 | |
| *** _degorenko|afk is now known as degorenko | 09:02 | |
| *** arxcruz has quit IRC | 09:04 | |
| *** arxcruz has joined #openstack-infra | 09:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:07 | |
| *** ildikov has quit IRC | 09:08 | |
| rcarrillocruz | abregman: check it out projects.yaml on openstack-infra/project-config | 09:08 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 09:09 |
| *** asettle has joined #openstack-infra | 09:10 | |
| abregman | rcarrillocruz: what exactly am I looking for?: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml | 09:10 |
| *** jcoufal has joined #openstack-infra | 09:10 | |
| rcarrillocruz | abregman: what test are you looking for | 09:11 |
| rcarrillocruz | ? | 09:11 |
| *** salv-orlando has quit IRC | 09:11 | |
| rcarrillocruz | the 'node' tells you the image used | 09:11 |
| rcarrillocruz | bare-trusty => bare ubuntu trusty | 09:11 |
| rcarrillocruz | bare-precise => bare ubuntu precise | 09:11 |
| rcarrillocruz | so on and so forth | 09:11 |
| abregman | rcarrillocruz: I want to download the image this gate job used -> http://logs.openstack.org/74/286074/1/check/gate-neutron-dsvm-api/9c25e1d/console.html | 09:11 |
| rcarrillocruz | dsvm stand for 'devstack vm' | 09:12 |
| abregman | yeah I know | 09:12 |
| rcarrillocruz | in your case | 09:12 |
| rcarrillocruz | 2016-02-29 18:04:55.236 | Building remotely on devstack-trusty-ovh-bhs1-8379741 (devstack-trusty) in workspace /home/jenkins/workspace/gate-neutron-dsvm-api | 09:12 |
| rcarrillocruz | devstack-trusty | 09:12 |
| *** kzaitsev_mb has quit IRC | 09:12 | |
| *** ihrachys has joined #openstack-infra | 09:13 | |
| abregman | rcarrillocruz: where the 'devstack-trusty' image is stored? is it this?: https://cloud-images.ubuntu.com/trusty/ | 09:13 |
| *** hashar has joined #openstack-infra | 09:13 | |
| abregman | rcarrillocruz: where can I download the exact image the job is using? | 09:14 |
| *** amotoki has quit IRC | 09:14 | |
| *** jistr has joined #openstack-infra | 09:14 | |
| *** scheuran has quit IRC | 09:15 | |
| *** andymaier has joined #openstack-infra | 09:18 | |
| rcarrillocruz | https://github.com/openstack-infra/project-config/tree/master/nodepool/elements | 09:18 |
| rcarrillocruz | you can't download, they are built by the infra team | 09:18 |
| rcarrillocruz | check it out ^ docs to build your own | 09:18 |
| *** mattt has joined #openstack-infra | 09:19 | |
| *** amotoki has joined #openstack-infra | 09:19 | |
| *** sshnaidm has joined #openstack-infra | 09:19 | |
| mattt | quite likely i'm being thick ... but is there a clear way to see a dependency tree for dependent reviews in gerrit? | 09:19 |
| mattt | this used to be a lot more obvious in the old interface | 09:19 |
| *** thorst has joined #openstack-infra | 09:22 | |
| markus_z | mtreinish: You created the "cat-pip.txt" file, so this might be interesting for you https://bugs.launchpad.net/devstack/+bug/1542545 | 09:23 |
| openstack | Launchpad bug 1542545 in devstack "devstack is broken" [Undecided,Confirmed] | 09:23 |
| markus_z | mtreinish: I'm wondering why the gate doesn't complain | 09:23 |
| markus_z | dtroyer: ^ It looks like you reviewed that file too (http://git.openstack.org/cgit/openstack-dev/devstack/commit/tools/cap-pip.txt?id=75446deea06107fa63a7f08990f0de26e5761833) | 09:26 |
| openstackgerrit | Fausto Marzi proposed openstack/requirements: Add modules for freezer Mitaka release https://review.openstack.org/271072 | 09:27 |
| *** scheuran has joined #openstack-infra | 09:27 | |
| *** chenli has quit IRC | 09:28 | |
| *** e0ne has joined #openstack-infra | 09:28 | |
| *** ikalnitsky has joined #openstack-infra | 09:28 | |
| *** thorst has quit IRC | 09:28 | |
| *** keedya has quit IRC | 09:29 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list https://review.openstack.org/286062 | 09:29 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 09:29 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of brctl show from hosting node https://review.openstack.org/286279 | 09:29 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 09:29 |
| *** sbelous_ has joined #openstack-infra | 09:32 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** ildikov has joined #openstack-infra | 09:35 | |
| *** ociuhandu has joined #openstack-infra | 09:35 | |
| *** korzen_ is now known as korzen | 09:37 | |
| *** e0ne has quit IRC | 09:38 | |
| *** derekh has joined #openstack-infra | 09:39 | |
| AJaeger | mattt: it's still there in the upper right corner | 09:40 |
| AJaeger | mattt: if you prefer cli, use gertty, it can show dependencies nicely | 09:40 |
| mattt | AJaeger: yeah i see that, but at least for me there's no clear indication what the relationship is | 09:40 |
| AJaeger | mattt: which change? | 09:40 |
| *** roxanaghe has joined #openstack-infra | 09:42 | |
| AJaeger | mattt: if it says "RElated chagnes (2)", then the lower item is bottom of stack and the one above is top of stack | 09:43 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 09:43 |
| AJaeger | https://review.openstack.org/#/c/286242/ - dvipng change is on top of pandoc one | 09:43 |
| mattt | AJaeger: ok, so it's the ordering that is key ... the reviews i was looking at were dependent on one-another i think so it was super confusing what was what | 09:44 |
| mattt | AJaeger: thanks for clearing up! and i really should look at gerrty | 09:45 |
| *** dtardivel has joined #openstack-infra | 09:45 | |
| *** vgridnev has joined #openstack-infra | 09:46 | |
| *** roxanaghe has quit IRC | 09:46 | |
| lucasagomes | hi all, I've this small patch in devstack gate fixing an hardcoded assumption that is preventing the ipmitool jobs in gate to run https://review.openstack.org/#/c/284036/ , if you have some time mind taking a look at it ? thank you | 09:46 |
| *** zhurong has quit IRC | 09:47 | |
| AJaeger | mattt: ;) | 09:47 |
| AJaeger | mattt: yes, ordering is key. They're stacked on top of each other | 09:48 |
| openstackgerrit | Olivier Lemasle proposed openstack-infra/jeepyb: Update projects on GitHub https://review.openstack.org/277175 | 09:48 |
| *** aarefiev has quit IRC | 09:48 | |
| *** aarefiev has joined #openstack-infra | 09:49 | |
| *** vgridnev has quit IRC | 09:54 | |
| *** sorantis has joined #openstack-infra | 09:56 | |
| *** jordanP has joined #openstack-infra | 09:59 | |
| *** ociuhandu has quit IRC | 09:59 | |
| *** vgridnev has joined #openstack-infra | 10:00 | |
| *** gnuoy_ has joined #openstack-infra | 10:03 | |
| *** gnuoy_ has quit IRC | 10:08 | |
| *** rguillebert has joined #openstack-infra | 10:09 | |
| openstackgerrit | Jens proposed openstack-infra/git-review: Make it possible to configure draft as default push mode https://review.openstack.org/220426 | 10:09 |
| *** e0ne has joined #openstack-infra | 10:11 | |
| dtantsur | morning folks! is gerrit terribly slow for everyone or just me? | 10:13 |
| *** jcooley_ has joined #openstack-infra | 10:15 | |
| *** fabio_ has joined #openstack-infra | 10:19 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 10:20 |
| *** andymaier has quit IRC | 10:25 | |
| *** sambetts|afk is now known as sambetts | 10:26 | |
| *** ildikov has quit IRC | 10:26 | |
| *** thorst has joined #openstack-infra | 10:26 | |
| jamespage | morning (again) - please can I be added to the charms-core and charms-release groups created under https://review.openstack.org/#/c/232705/ | 10:30 |
| *** Qiming has quit IRC | 10:30 | |
| yolanda | hi jamespage | 10:30 |
| yolanda | i can do it | 10:30 |
| jamespage | yolanda, thanks muchly | 10:30 |
| *** lucasagomes has quit IRC | 10:30 | |
| openstackgerrit | Francesco Longo proposed openstack-infra/project-config: Added IoTronic project. https://review.openstack.org/286113 | 10:30 |
| yolanda | gerrit is super slow for me today | 10:31 |
| jamespage | yolanda, I'm assuming that will bootstrap me to add other members to both of those groups? | 10:31 |
| yolanda | jamespage yes, i will add you and you can other people | 10:31 |
| jamespage | yolanda, +1 awesome | 10:31 |
| dtantsur | yolanda, gerrit is crazily slow, yeah | 10:31 |
| *** lucasagomes has joined #openstack-infra | 10:31 | |
| dtantsur | and it becomes slower and slower | 10:31 |
| yolanda | dtantsur, going to check, it's not usable for me now | 10:31 |
| dtantsur | ++ | 10:31 |
| *** lucasagomes has quit IRC | 10:32 | |
| *** lucasagomes has joined #openstack-infra | 10:32 | |
| *** thorst has quit IRC | 10:33 | |
| yolanda | #status alert Gerrit is going to be restarted due to poor performance | 10:33 |
| openstackstatus | yolanda: sending alert | 10:33 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to poor performance | 10:36 | |
| *** ChanServ changes topic to "Gerrit is going to be restarted due to poor performance" | 10:36 | |
| *** daemontool has joined #openstack-infra | 10:36 | |
| *** lucasagomes has quit IRC | 10:37 | |
| *** lucasagomes has joined #openstack-infra | 10:37 | |
| yolanda | jamespage, added | 10:38 |
| ikalnitsky | Hey folks! I'm a core in fuel-plugins repo. Recently I landed the patch that should help me to publish releases to PyPI https://review.openstack.org/#/c/283683/ However, yesterday I pushed a new tag but PyPI sdist wasn't uploaded. openstackci user is added on PyPI as owner. Could someone help me to understand what's wrong? | 10:38 |
| yolanda | dtantsur, gerrit was already restarted, but statusbot looks slow on announcing it | 10:38 |
| openstackstatus | yolanda: finished sending alert | 10:39 |
| yolanda | #status ok gerrit finished restartign | 10:40 |
| openstackstatus | yolanda: sending ok | 10:40 |
| *** ildikov has joined #openstack-infra | 10:40 | |
| *** daemontool__ has quit IRC | 10:40 | |
| *** lucasagomes has quit IRC | 10:41 | |
| jamespage | yolanda, thankyou | 10:42 |
| yolanda | anytime | 10:42 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 10:42 | |
| -openstackstatus- NOTICE: gerrit finished restartign | 10:42 | |
| * yolanda is happy to see charms in openstack | 10:42 | |
| *** roxanaghe has joined #openstack-infra | 10:42 | |
| *** lucasagomes has joined #openstack-infra | 10:43 | |
| openstackstatus | yolanda: finished sending ok | 10:45 |
| AJaeger | ikalnitsky: check the log files to see what the error is | 10:46 |
| AJaeger | ikalnitsky: which version did you push? | 10:46 |
| ikalnitsky | AJaeger: 4.0.0 | 10:46 |
| *** kzaitsev_mb has joined #openstack-infra | 10:47 | |
| ikalnitsky | AJaeger: how to check logs? I've sent 4.0.0 tag to gerrit and that's it. | 10:47 |
| *** roxanaghe has quit IRC | 10:47 | |
| *** ihrachys has quit IRC | 10:48 | |
| *** exploreshaifali has joined #openstack-infra | 10:49 | |
| AJaeger | ikalnitsky: git show-ref 4.0.0 | 10:49 |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-cfgdb' https://review.openstack.org/286137 | 10:49 |
| AJaeger | And then look on logs.openstack.org for it | 10:49 |
| AJaeger | It's 6abd3371d870cc5e90ce72aa8cf6103b641f0e42 -> so, look for 6a/6abd3371d870cc5e90ce72aa8cf6103b641f0e42 | 10:50 |
| ikalnitsky | AJaeger: didn't know about that! let me try. thanks! | 10:50 |
| AJaeger | jamespage: seems you have no setup.cfg file. ARe you using pbr? | 10:52 |
| jamespage | AJaeger, not yet | 10:52 |
| jamespage | is that causing issues? | 10:52 |
| AJaeger | jamespage: sorry, I meant ikalnitsky and somehow did not notice that wrong tab completion | 10:53 |
| ikalnitsky | AJaeger: nope, no pbr | 10:53 |
| AJaeger | ikalnitsky: that's your problem, AFAIK we require pbr for it | 10:54 |
| ikalnitsky | AJaeger: yeah, i found that your job relies on setup.cfg | 10:54 |
| ikalnitsky | AJaeger: well, I'll move to pbr then. Thanks :) | 10:54 |
| AJaeger | and then release a 4.0.1 ;) | 10:55 |
| ikalnitsky | AJaeger: however, I'd recommend you guys to use `python setup.py --name` to retrieve project name :) | 10:55 |
| ikalnitsky | AJaeger: besides, we have obsolete branches. any chance to remove them? or they will be there forever? | 10:55 |
| AJaeger | ikalnitsky: didn't I ask you during review whether you want all of them? That's my standard question... | 10:56 |
| AJaeger | ikalnitsky: they can only be removed manually and that's expensive - it should have done before the import ;( | 10:56 |
| *** pfallenop has joined #openstack-infra | 10:56 | |
| AJaeger | you need to bribe ;) one of the admins to do it while there'S no fire ongoing | 10:57 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 10:57 |
| ikalnitsky | AJaeger: heh.. any contacts? or, wait, don't you have such permissions? :) | 10:58 |
| *** bexelbie has quit IRC | 10:58 | |
| *** bexelbie has joined #openstack-infra | 10:58 | |
| *** oanson has joined #openstack-infra | 10:58 | |
| AJaeger | ikalnitsky: I don't have such permissions - come back during US time, please | 10:59 |
| *** rvba` is now known as rvba | 10:59 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config: Skip dsvm jobs on manila docs/reno/unit test changes https://review.openstack.org/286497 | 11:00 |
| ikalnitsky | AJaeger: hehe. ok :) thanks a lot for your help! | 11:00 |
| *** chlong_ has joined #openstack-infra | 11:02 | |
| *** pfallenop has quit IRC | 11:03 | |
| abregman | rcarrillocruz: thanks for the help! | 11:03 |
| *** dims has joined #openstack-infra | 11:03 | |
| rcarrillocruz | np | 11:03 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 11:05 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 11:06 |
| *** amotoki has quit IRC | 11:06 | |
| *** fhubik has joined #openstack-infra | 11:11 | |
| openstackgerrit | Sergey Lukjanov proposed openstack/requirements: Bump to final Mitaka python-saharaclient https://review.openstack.org/286503 | 11:12 |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config: Skip dsvm jobs on manila docs/reno/unit test changes https://review.openstack.org/286497 | 11:13 |
| tbarron | vponomaryov: thanks, that woke me up | 11:14 |
| tbarron | vponomaryov: patch set #2 is up, as you specified | 11:14 |
| *** yamamoto has quit IRC | 11:15 | |
| *** |-paul-| has joined #openstack-infra | 11:15 | |
| *** pfallenop has joined #openstack-infra | 11:16 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Archive all of the delorean logs https://review.openstack.org/271416 | 11:16 |
| *** xiangxinyong has joined #openstack-infra | 11:17 | |
| *** claudiub has joined #openstack-infra | 11:25 | |
| *** jlanoux_ has joined #openstack-infra | 11:30 | |
| abregman | is there a way to connect the gate job machine? | 11:30 |
| openstackgerrit | Fausto Marzi proposed openstack/requirements: Add Freezer modules and projects for Mitaka release https://review.openstack.org/271072 | 11:30 |
| *** jlanoux has quit IRC | 11:30 | |
| *** thorst has joined #openstack-infra | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| *** rossella_s has quit IRC | 11:32 | |
| *** rossella_s has joined #openstack-infra | 11:33 | |
| *** ildikov has quit IRC | 11:36 | |
| *** Qiming has joined #openstack-infra | 11:36 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 11:36 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list https://review.openstack.org/286062 | 11:36 |
| *** gildub has quit IRC | 11:37 | |
| *** [1]Thelo has joined #openstack-infra | 11:37 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 11:37 |
| *** _amrith_ is now known as amrith | 11:38 | |
| *** thorst has quit IRC | 11:38 | |
| *** aysyd has joined #openstack-infra | 11:39 | |
| *** ihrachys has joined #openstack-infra | 11:39 | |
| *** Thelo has quit IRC | 11:40 | |
| *** [1]Thelo is now known as Thelo | 11:40 | |
| *** esker has quit IRC | 11:40 | |
| *** ociuhandu has joined #openstack-infra | 11:40 | |
| *** esker has joined #openstack-infra | 11:40 | |
| *** salv-orlando has joined #openstack-infra | 11:41 | |
| *** tpsilva has joined #openstack-infra | 11:44 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:44 | |
| *** aysyd has quit IRC | 11:46 | |
| *** aysyd has joined #openstack-infra | 11:46 | |
| *** sam_wan has quit IRC | 11:47 | |
| *** jlanoux_ has quit IRC | 11:50 | |
| *** Jeffrey4l has quit IRC | 11:51 | |
| *** sdague has joined #openstack-infra | 11:51 | |
| *** jlanoux has joined #openstack-infra | 11:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Skip dsvm jobs on manila docs/reno/unit test changes https://review.openstack.org/286497 | 11:54 |
| *** sarob has joined #openstack-infra | 11:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Sort projects in big skip list https://review.openstack.org/286527 | 11:57 |
| *** salv-orlando has quit IRC | 11:58 | |
| *** amotoki has joined #openstack-infra | 11:58 | |
| *** sorantis has quit IRC | 11:59 | |
| *** sarob has quit IRC | 12:00 | |
| *** yamamoto has joined #openstack-infra | 12:02 | |
| AJaeger | sdague: could you review https://review.openstack.org/285949 https://review.openstack.org/#/c/285487/ and https://review.openstack.org/#/c/285148/ , please? Those are project-config changes that remove jobs | 12:04 |
| *** fhubik is now known as fhubik_brb | 12:06 | |
| *** daemontool_ has joined #openstack-infra | 12:07 | |
| *** rfolco_ has joined #openstack-infra | 12:07 | |
| *** amotoki has quit IRC | 12:07 | |
| *** Jeffrey4l has joined #openstack-infra | 12:09 | |
| *** jpr has joined #openstack-infra | 12:10 | |
| *** daemontool has quit IRC | 12:10 | |
| *** dizquierdo_afk is now known as dizquierdo | 12:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup specs repos setup https://review.openstack.org/282889 | 12:13 |
| *** fhubik_brb is now known as fhubik | 12:13 | |
| *** dtardivel has quit IRC | 12:28 | |
| *** exploreshaifali has quit IRC | 12:29 | |
| *** |-paul-| has quit IRC | 12:29 | |
| *** andymaier has joined #openstack-infra | 12:30 | |
| *** amrith is now known as _amrith_ | 12:34 | |
| *** kushal has joined #openstack-infra | 12:34 | |
| *** daemontool__ has joined #openstack-infra | 12:34 | |
| *** jaosorior has quit IRC | 12:36 | |
| *** jaosorior has joined #openstack-infra | 12:37 | |
| *** daemontool_ has quit IRC | 12:37 | |
| *** ildikov has joined #openstack-infra | 12:38 | |
| *** gordc has joined #openstack-infra | 12:40 | |
| *** rhallisey has joined #openstack-infra | 12:43 | |
| *** roxanaghe has joined #openstack-infra | 12:44 | |
| *** thorst has joined #openstack-infra | 12:46 | |
| *** shardy has quit IRC | 12:47 | |
| *** thorst_ has joined #openstack-infra | 12:47 | |
| *** roxanaghe has quit IRC | 12:48 | |
| *** fhubik is now known as fhubik_brb | 12:48 | |
| *** jaypipes has joined #openstack-infra | 12:48 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard https://review.openstack.org/286194 | 12:49 |
| *** fhubik_brb is now known as fhubik | 12:49 | |
| *** thorst has quit IRC | 12:51 | |
| *** amotoki has joined #openstack-infra | 12:52 | |
| *** baoli has joined #openstack-infra | 12:54 | |
| *** salv-orlando has joined #openstack-infra | 12:55 | |
| *** andymaier has quit IRC | 12:57 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Add a standalone map page https://review.openstack.org/285943 | 12:59 |
| *** dims has quit IRC | 13:02 | |
| *** fabio_ has quit IRC | 13:06 | |
| *** fabio_ has joined #openstack-infra | 13:06 | |
| *** abregman is now known as abregman|brb | 13:07 | |
| *** links has quit IRC | 13:07 | |
| *** dims has joined #openstack-infra | 13:08 | |
| *** rlandy has joined #openstack-infra | 13:10 | |
| *** julim has joined #openstack-infra | 13:10 | |
| *** yamamoto has quit IRC | 13:10 | |
| *** yamamoto has joined #openstack-infra | 13:11 | |
| *** yamamoto has quit IRC | 13:11 | |
| *** yamamoto has joined #openstack-infra | 13:12 | |
| *** cloudtrainme has joined #openstack-infra | 13:12 | |
| *** exploreshaifali has joined #openstack-infra | 13:12 | |
| odyssey4me | hi everyone - it seems we have a mismatch between git.o.o and github - http://git.openstack.org/cgit/openstack/openstack-ansible-os_horizon is there, but not in github - how do we resolev this? | 13:14 |
| *** pradk has joined #openstack-infra | 13:14 | |
| *** Aegil has quit IRC | 13:15 | |
| *** lucasagomes is now known as lucas-hungry | 13:15 | |
| *** sdake has joined #openstack-infra | 13:16 | |
| AJaeger | odyssey4me: keep in mind that github is only a mirror, git.openstack.org is the master. | 13:16 |
| AJaeger | odyssey4me: you should be able to develop with this setup | 13:16 |
| odyssey4me | AJaeger sure, I'm just trying to understand how the mirror process missed a whole repo | 13:16 |
| AJaeger | odyssey4me: still, it's not what we intend currently, so needs manual fixing | 13:16 |
| AJaeger | odyssey4me: let'S ask yolanda whether she can manually fix this | 13:17 |
| *** zeih has quit IRC | 13:17 | |
| AJaeger | odyssey4me: github rate limits calls and that might screw up setting up a repo... | 13:17 |
| *** zeih has joined #openstack-infra | 13:18 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add a third scenario for packstack integration testing https://review.openstack.org/286579 | 13:19 |
| AJaeger | odyssey4me: An admin needs to fix this | 13:19 |
| *** Daisy has joined #openstack-infra | 13:19 | |
| *** andymaier has joined #openstack-infra | 13:20 | |
| *** sdake has quit IRC | 13:21 | |
| *** sdake has joined #openstack-infra | 13:21 | |
| *** grue_pm has quit IRC | 13:22 | |
| *** kgiusti has joined #openstack-infra | 13:22 | |
| yolanda | hi, back from launch | 13:25 |
| *** dprince has joined #openstack-infra | 13:25 | |
| yolanda | lunch | 13:25 |
| yolanda | so github not replicating again? | 13:25 |
| AJaeger | yolanda: permission problem again? | 13:25 |
| yolanda | i guess, going to fix | 13:25 |
| AJaeger | https://github.com/openstack/openstack-ansible-os_horizon does not exist at all ;( | 13:26 |
| *** esikachev has joined #openstack-infra | 13:26 | |
| yolanda | that's even worse | 13:26 |
| AJaeger | odyssey4me: are all other repos there? | 13:27 |
| openstackgerrit | Merged openstack-infra/release-tools: better handling of stable flag https://review.openstack.org/285520 | 13:27 |
| odyssey4me | AJaeger we're working through the various repositories to check which came through and which didn't and will report back. | 13:27 |
| yolanda | going to check manage-projects output | 13:28 |
| AJaeger | thanks, yolanda and odyssey4me | 13:28 |
| openstackgerrit | Akihiro Motoki proposed openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0 https://review.openstack.org/286587 | 13:29 |
| yolanda | manage_projects - ERROR - Problems creating openstack/openstack-ansible-os_horizon, moving on. | 13:29 |
| yolanda | Traceback (most recent call last): | 13:29 |
| yolanda | going to try that manually | 13:29 |
| odyssey4me | thanks AJaeger & yolanda | 13:30 |
| *** zeih has quit IRC | 13:30 | |
| *** zeih has joined #openstack-infra | 13:31 | |
| *** yamamoto has quit IRC | 13:32 | |
| *** slagle has joined #openstack-infra | 13:33 | |
| *** cloudtrainme has quit IRC | 13:34 | |
| yolanda | odyssey4me, AJaeger, fixed | 13:37 |
| yolanda | let me know if there are more repos to be fixed | 13:37 |
| odyssey4me | thanks yolanda ! | 13:37 |
| daemontool__ | AJaeger, at your convenience: https://review.openstack.org/#/c/271072/ | 13:37 |
| AJaeger | daemontool__: please explain what that is... | 13:38 |
| *** yamamoto has joined #openstack-infra | 13:38 | |
| daemontool__ | AJaeger, is about adding modules to openstack/requirements | 13:38 |
| daemontool__ | and adding the freezer* to projects.txt on that same repo | 13:38 |
| *** yamamoto has quit IRC | 13:39 | |
| AJaeger | daemontool__: I'm not a core for requirements, can't help with that. | 13:39 |
| *** baoli has quit IRC | 13:40 | |
| daemontool__ | AJaeger, ok ty | 13:40 |
| *** baoli has joined #openstack-infra | 13:40 | |
| *** fhubik is now known as fhubik_brb | 13:42 | |
| *** fhubik_brb is now known as fhubik | 13:42 | |
| *** fhubik is now known as fhubik_brb | 13:42 | |
| *** baoli has quit IRC | 13:43 | |
| *** jtomasek_ has joined #openstack-infra | 13:43 | |
| *** baoli has joined #openstack-infra | 13:43 | |
| AJaeger | fungi, infra-root: The openSSL DROWN has been published, see https://drownattack.com/ | 13:44 |
| AJaeger | Codename "DROWN" for "Decrypting RSA using Obsolete and Weakened eNcryption" | 13:44 |
| AJaeger | " server is vulnerable to DROWN if: It allows SSLv2 connections OR Its private key is used on any other server that allows SSLv2 connections" | 13:45 |
| AJaeger | Do we use SSLv2 anywhere in the OpenStack infrastructure? | 13:46 |
| mordred | AJaeger: I believe we shut it off a while ago | 13:46 |
| AJaeger | mordred: I grepped around and hope so as well ;) | 13:46 |
| *** dizquierdo has quit IRC | 13:47 | |
| AJaeger | But I didn't check each and every place | 13:47 |
| mordred | AJaeger: yes | 13:47 |
| mordred | AJaeger: http://codesearch.openstack.org/?q=SSLProtocol&i=nope&files=&repos= | 13:48 |
| *** claudiub|2 has joined #openstack-infra | 13:48 | |
| mordred | AJaeger: not only is infra good, most of openstack deployment is good | 13:48 |
| mordred | with one exceptoin - but that's because of something that's parameterized | 13:48 |
| mordred | and doesn't set it automatically | 13:49 |
| *** dkranz has joined #openstack-infra | 13:49 | |
| *** shardy has joined #openstack-infra | 13:50 | |
| *** sdake_ has joined #openstack-infra | 13:50 | |
| AJaeger | mordred: check http://codesearch.openstack.org/?q=SSLEngine&i=nope&files=&repos= - seems that the newly imported charm is not secure | 13:50 |
| AJaeger | jamespage: http://git.openstack.org/cgit/openstack/charm-heat/tree/hooks/charmhelpers/contrib/openstack/templates/openstack_https_frontend | 13:50 |
| *** claudiub has quit IRC | 13:51 | |
| mordred | AJaeger: wow. am I about to write a patch to a juju charm? | 13:51 |
| jamespage | oh nice | 13:52 |
| mordred | jamespage: ooh, maybe you want to :) | 13:52 |
| jamespage | mordred, let me deal with that - we still have a sync-y thing to deal with | 13:52 |
| mordred | btw - the openstack security team has a great doc: | 13:52 |
| mordred | http://git.openstack.org/cgit/openstack/security-doc/tree/security-guide/source/secure-communication/tls-proxies-and-http-services.rst#n254 | 13:52 |
| AJaeger | and compass as well: http://git.openstack.org/cgit/openstack/compass-web/tree/v2/dboards/sample/apache_ldap.conf | 13:52 |
| mordred | with some specific config lines | 13:52 |
| *** yamamoto has joined #openstack-infra | 13:53 | |
| *** maishsk_ has joined #openstack-infra | 13:53 | |
| *** maishsk has quit IRC | 13:53 | |
| *** maishsk_ is now known as maishsk | 13:53 | |
| *** jtomasek_ has quit IRC | 13:54 | |
| *** sdake has quit IRC | 13:54 | |
| AJaeger | mordred: is this one fine? http://git.openstack.org/cgit/openstack-infra/puppet-httpd/tree/templates/vhost-proxy.conf.erb | 13:54 |
| *** otsuka has quit IRC | 13:54 | |
| AJaeger | It does not set ciphers... | 13:55 |
| openstackgerrit | Anton Arefiev proposed openstack-infra/project-config: Add auto-discovery test job to ironic-inspector https://review.openstack.org/277843 | 13:56 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Create new partitioning element. https://review.openstack.org/259881 | 14:00 |
| *** bgaifullin has quit IRC | 14:01 | |
| mordred | AJaeger: well, I don't think we use that template anywhere | 14:01 |
| *** hichihara has joined #openstack-infra | 14:02 | |
| *** annegentle has joined #openstack-infra | 14:02 | |
| AJaeger | And this one: http://git.openstack.org/cgit/openstack-infra/puppet-storyboard/tree/templates/storyboard_https.vhost.erb ? | 14:02 |
| *** max_lobur has left #openstack-infra | 14:02 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list https://review.openstack.org/286062 | 14:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-httpd: Add SSL Procotol and Cipher config to default vhost https://review.openstack.org/286610 | 14:04 |
| *** derekh is now known as ndipanov_ | 14:04 | |
| mordred | AJaeger: yah - let's update storyboard | 14:04 |
| *** ndipanov_ is now known as derekh | 14:04 | |
| AJaeger | thanks, mordred | 14:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-storyboard: Update SSLProtocol and SSLCipherSuite https://review.openstack.org/286612 | 14:06 |
| yolanda | mordred, cool . Going to review | 14:06 |
| *** amitgandhinz has joined #openstack-infra | 14:07 | |
| yolanda | mordred, i was expecting to see SSLProtocol ALL -SSLv2 -SSLv3 | 14:08 |
| yolanda | is the same as your change or do we miss some protocol? | 14:08 |
| *** baoli has quit IRC | 14:08 | |
| mordred | yolanda: yah - I used the line from http://git.openstack.org/cgit/openstack/security-doc/tree/security-guide/source/secure-communication/tls-proxies-and-http-services.rst#n254 | 14:08 |
| *** abregman|brb is now known as abregman | 14:09 | |
| yolanda | don't we need SSLHonorCipherOrder On ? | 14:09 |
| *** lucas-hungry is now known as lucasagomes | 14:09 | |
| mordred | yolanda: maybe? do we have that set other places? | 14:10 |
| yolanda | i was reading about recommendations | 14:10 |
| yolanda | but haven't checked other places | 14:10 |
| AJaeger | It's not set in security-doc either | 14:11 |
| *** berendt has joined #openstack-infra | 14:11 | |
| AJaeger | https://review.openstack.org/286616 fixes salt-formula-horizon | 14:11 |
| mordred | if it is needed, we should send a patch into the security guide too | 14:12 |
| *** baoli has joined #openstack-infra | 14:12 | |
| yolanda | it may be a nice to have, but not really needed. Is to ensure that the first match of the cipher list is always used | 14:12 |
| mordred | of course, perhaps what we should really do is remove TLS 1.0 from the list | 14:12 |
| mordred | SSLHonorCipherOrder is only needed with SSLv3 and TLS 1.0 | 14:13 |
| berendt | hi. can a core reviewer of system-config please have a look at https://review.openstack.org/#/c/284297/ and give a +2A. it is a request for a openstack-de mailinglist. I have a user group meetup tomorrow and want to be able to announce the availability of the list. | 14:13 |
| yolanda | mordred and yes, tls v1.0 is not recommended | 14:14 |
| *** thiagop has joined #openstack-infra | 14:14 | |
| mordred | disabling tls v1.0 apparently means we'd be dropping support for IE7-10 | 14:14 |
| *** vgridnev has quit IRC | 14:14 | |
| *** ociuhandu has quit IRC | 14:15 | |
| *** baoli has quit IRC | 14:15 | |
| *** jpr has quit IRC | 14:15 | |
| yolanda | we should? :) | 14:15 |
| mordred | well, let's wait for fungi on that one | 14:15 |
| *** claudiub has joined #openstack-infra | 14:15 | |
| *** ociuhandu has joined #openstack-infra | 14:16 | |
| yolanda | mordred, can you take a look at https://review.openstack.org/285433 ? the glean hostname fix, it was not passing tests | 14:17 |
| *** Daisy has quit IRC | 14:17 | |
| *** claudiub|2 has quit IRC | 14:18 | |
| *** berendt has left #openstack-infra | 14:18 | |
| *** sdake_ is now known as sdake | 14:19 | |
| mordred | yolanda: so - jeblair gave me a good argument as to why we should perhaps use 127.0.1.1 instead of the actual ip of the server | 14:19 |
| yolanda | mordred, what should be the reason? | 14:19 |
| mordred | yolanda: which is that if the ip changes, the etc hosts entry will not work anymore | 14:19 |
| mordred | yolanda: but with 127.0.1.1 it will | 14:20 |
| *** vgridnev has joined #openstack-infra | 14:20 | |
| yolanda | mordred, also in the other hand, setting to 127.0.1.1 can give another problems, such as the one we had in rabbitmq and binding address | 14:20 |
| mordred | so if we have glean on a vm, and it runs once at vm boot, and sets up the network interfaces to use dhcp | 14:20 |
| mordred | and then the cloud changes the ip of the server | 14:21 |
| mordred | we could have an issue | 14:21 |
| mordred | the rabbit thing is a good point though | 14:21 |
| yolanda | we had controller pointing to 127.0.0.1 and rabbitmq with a binding address to controller00.xx. As a consecuence, any of the computes could not reach rabbit because it was limited to 127.0.0.1 and we had to open for all | 14:21 |
| yolanda | also looking at documentation, if the server has a fixed ip, is the recommended way to go | 14:21 |
| *** ociuhandu has quit IRC | 14:21 | |
| mordred | ah - SO | 14:22 |
| mordred | maybe we do this | 14:22 |
| yolanda | if the ip changes, we can run glean again | 14:22 |
| mordred | because we have two sets of glean users | 14:22 |
| *** baoli has joined #openstack-infra | 14:22 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Convert the container job to a noop https://review.openstack.org/285325 | 14:22 |
| *** jsavak has joined #openstack-infra | 14:22 | |
| mordred | if we're setting the server up for static IPs (like with bifrost) - that means glean will know the IP and will be writing it to the network config | 14:22 |
| mordred | and thus we shoudl write an IP to the hosts file | 14:23 |
| *** baoli has quit IRC | 14:23 | |
| *** tiswanso has joined #openstack-infra | 14:23 | |
| mordred | but, if glean does not write static network config and instead is doing dhcp - then I think we should do 127.0.1.1 | 14:23 |
| yolanda | well we rely on info present in metadata | 14:23 |
| yolanda | so if nothing is on network, we default to 127.0.1.1 | 14:23 |
| mordred | ah | 14:23 |
| mordred | good | 14:23 |
| mordred | so that logic I just said is already there :) | 14:23 |
| mordred | cool | 14:24 |
| * mordred re-reads the patch | 14:24 | |
| yolanda | ah yes, i implemented that logic | 14:24 |
| mordred | duh. yes. I see that now | 14:24 |
| mordred | thanks yolanda | 14:24 |
| *** woodster_ has joined #openstack-infra | 14:24 | |
| sdague | this seems odd - http://logs.openstack.org/43/281143/16/check/gate-nova-tox-functional/70d60c9/console.html#_2016-02-29_17_35_21_263 | 14:24 |
| yolanda | the thing that i don't trust to much, is picking the first interface found. for my tests it has been fine but i don't know all use cases | 14:24 |
| *** vgridnev has quit IRC | 14:25 | |
| mordred | yolanda: looks great | 14:25 |
| mordred | yolanda: I think first interface found for now is probably fine | 14:25 |
| yolanda | and there was that nice difference betwen python 2.7 and python 3 versions. I had to do a sorted, because python2.7 was not returning the same item than python 3 | 14:25 |
| *** Daisy has joined #openstack-infra | 14:25 | |
| yolanda | so the tests were crazy | 14:25 |
| *** Daisy has quit IRC | 14:25 | |
| *** mrmartin has quit IRC | 14:25 | |
| mordred | sdague: I agree | 14:25 |
| *** baoli has joined #openstack-infra | 14:25 | |
| lucasagomes | clarkb, sdague hi, if you guys have some time today, mind taking a look at https://review.openstack.org/#/c/284036/ ? This will allow we test ipmitool drivers in the ironic gate (which is our reference driver, but it wasn't tested before) | 14:26 |
| mordred | yolanda: :) | 14:26 |
| *** Daisy has joined #openstack-infra | 14:26 | |
| lucasagomes | so we would like to get it tested soon and make the jobs voting as soon as they establish | 14:26 |
| lucasagomes | thank you | 14:26 |
| *** vgridnev has joined #openstack-infra | 14:26 | |
| pabelanger | morning | 14:26 |
| *** aysyd has quit IRC | 14:27 | |
| *** aysyd has joined #openstack-infra | 14:28 | |
| *** _amrith_ is now known as amrith | 14:29 | |
| *** kzaitsev_mb has quit IRC | 14:29 | |
| *** jroll has quit IRC | 14:29 | |
| *** jroll has joined #openstack-infra | 14:30 | |
| *** jroll has quit IRC | 14:30 | |
| *** jroll has joined #openstack-infra | 14:30 | |
| pabelanger | so, we did get a spammer overnight | 14:30 |
| *** kzaitsev_mb has joined #openstack-infra | 14:30 | |
| *** sfinucan has joined #openstack-infra | 14:30 | |
| pabelanger | maybe 4 of them | 14:30 |
| mordred | pabelanger: that's much better! | 14:30 |
| pabelanger | they switched to uploading png files :) | 14:31 |
| pabelanger | mordred: indeed | 14:31 |
| mordred | heh | 14:31 |
| *** andymaier has quit IRC | 14:31 | |
| *** edmondsw has joined #openstack-infra | 14:31 | |
| pabelanger | Should spend some time looking at how https://www.mediawiki.org/wiki/Extension:TitleBlacklist works | 14:31 |
| pabelanger | so we can blacklist the phonenumbers they are using | 14:32 |
| *** shardy has quit IRC | 14:33 | |
| *** eharney has joined #openstack-infra | 14:33 | |
| *** ociuhandu has joined #openstack-infra | 14:34 | |
| *** shardy has joined #openstack-infra | 14:35 | |
| *** sshnaidm has quit IRC | 14:35 | |
| *** weshay has joined #openstack-infra | 14:36 | |
| *** rbrndt has joined #openstack-infra | 14:36 | |
| *** C_W has joined #openstack-infra | 14:37 | |
| *** baoli has quit IRC | 14:39 | |
| *** ociuhandu has quit IRC | 14:39 | |
| *** d0ugal has quit IRC | 14:39 | |
| *** d0ugal has joined #openstack-infra | 14:40 | |
| *** d0ugal has quit IRC | 14:40 | |
| *** _ody has quit IRC | 14:40 | |
| yolanda | mordred, thx for review, let's see if we can pick another +2 today | 14:40 |
| yolanda | that could be causing problems to glean users | 14:40 |
| *** d0ugal has joined #openstack-infra | 14:41 | |
| *** baoli has joined #openstack-infra | 14:41 | |
| *** Daisy has quit IRC | 14:44 | |
| fungi | mordred: yeah, i'm reviewing the advisory now | 14:45 |
| mordred | fungi: cool. when you get done, there are two outstanding questions | 14:46 |
| mordred | fungi: a) should we go ahead and disable tls v1.0 (if not, should we set SSLHonorCipherOrder) | 14:46 |
| fungi | AJaeger: thanks for the heads up | 14:46 |
| mordred | fungi: b) should we update our SSLProtocol and SSLCipherSuite settings in our files to match the recommendations made in the security guide | 14:47 |
| *** ociuhandu has joined #openstack-infra | 14:47 | |
| mtreinish | mordred:, fungi: if you get a sec can you restart the subunit gearman worker the necessary patches landed | 14:47 |
| mtreinish | hopefully it'll stay up for more than 1 test result now | 14:47 |
| *** xyang1 has joined #openstack-infra | 14:48 | |
| *** sdake has quit IRC | 14:49 | |
| fungi | mordred: where did the suggestion to disable tls 1.0 come from? | 14:50 |
| fungi | and yeah, i vaguely recall standardizing all our configs in infra to drop pre-tls protocol versions already but the double-check was great | 14:50 |
| fungi | having a look at the security guide recommendations now | 14:51 |
| mordred | fungi: well - tls 1.0 came from looking at the SSLHonorCipherOrder setting - which is only needed for tls 1.0 and ssl v3 | 14:52 |
| *** alivigni has joined #openstack-infra | 14:52 | |
| mordred | fungi: but then that led me to check in to tls 1.0 - and it's being dropped from PCI compliance as of June of this year - so it's on its way out of general use | 14:52 |
| mordred | fungi: I don't think that one is urgent, but thought I'd mention it | 14:52 |
| mordred | (not that we need to be PCI compliant, mind you, but if _they | 14:53 |
| mordred | _they're_ dropping it ...) | 14:53 |
| fungi | yeah, i'm not opposed, though it would be interesting to find out what platforms/browsers we're dropping compatibility with when doing that | 14:53 |
| *** daemontool_ has joined #openstack-infra | 14:53 | |
| bkero | I'm sure there are some selenium services that can tell you | 14:54 |
| fungi | yeah, there's plenty of documentation out there too | 14:55 |
| *** doug-fish has joined #openstack-infra | 14:55 | |
| fungi | we can just do it and then point ssllabs.com at the site we're testing, for example | 14:55 |
| *** zeih has quit IRC | 14:55 | |
| *** _ody has joined #openstack-infra | 14:56 | |
| *** sdake has joined #openstack-infra | 14:56 | |
| *** daemontool__ has quit IRC | 14:56 | |
| *** e0ne has quit IRC | 14:56 | |
| *** edmondsw has quit IRC | 14:57 | |
| fungi | mordred: so 286610 was the only necessary change you spotted? | 14:57 |
| mordred | fungi: yah | 14:58 |
| mordred | fungi: we'd be dropping support for IE 7-10 | 14:58 |
| *** sorantis has joined #openstack-infra | 14:59 | |
| *** C_W has quit IRC | 14:59 | |
| *** shardy has quit IRC | 15:00 | |
| bkero | There are these handy guidelines too: https://wiki.mozilla.org/Security/Guidelines/Web_Security | 15:01 |
| fungi | i scanned security.o.o for example and this report lists the browsers which are relying on tls 1.0 to use it https://www.ssllabs.com/ssltest/analyze.html?d=security.openstack.org&s=2001%3a4800%3a7813%3a516%3a3bc3%3ad7f6%3aff05%3a4882 | 15:01 |
| bkero | (also with config generator)++ | 15:01 |
| *** claudiub|2 has joined #openstack-infra | 15:01 | |
| fungi | safari on osx 10.8 for example | 15:02 |
| *** fawadkhaliq has joined #openstack-infra | 15:02 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard https://review.openstack.org/286194 | 15:02 |
| *** claudiub has quit IRC | 15:04 | |
| AJaeger | sdague: that looks indeed strange ;( | 15:04 |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quota support https://review.openstack.org/285110 | 15:04 |
| *** Daisy has joined #openstack-infra | 15:04 | |
| fungi | looks like SSLHonorCipherOrder is an i-know-better-than-you option, where you can make the server force its preferences on the client rather than letting the client choose its preferred order | 15:05 |
| *** dizquierdo has joined #openstack-infra | 15:06 | |
| fungi | i'm not especially opposed, but it's trading default secure configuration assumptions for one party over the other | 15:06 |
| *** exploreshaifali has quit IRC | 15:07 | |
| *** bgaifullin has joined #openstack-infra | 15:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Make ceph jobs non-voting until bug 1551305 is fixed https://review.openstack.org/286642 | 15:07 |
| openstack | bug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed] https://launchpad.net/bugs/1551305 | 15:07 |
| *** yamahata has joined #openstack-infra | 15:07 | |
| mriedem | sdague: dansmith: jamespage: ^ | 15:07 |
| fungi | i'm more in favor of just dropping tls v1 as long as we don't expect to cater to the older android, safari, internet explorer, et cetera versions which lack tls v1.1 support | 15:08 |
| *** cloudtrainme has joined #openstack-infra | 15:08 | |
| sdague | mriedem: +2 | 15:08 |
| sdague | fungi / AJaeger / mordred can we get another review on - https://review.openstack.org/#/c/286642 | 15:08 |
| openstackgerrit | Dimitri Mazmanov proposed openstack-infra/project-config: Add check-requirements job to Kingbird https://review.openstack.org/286646 | 15:08 |
| bkero | fungi: would a potential openstack client on older android versions use TLS 1.0? | 15:09 |
| *** Daisy has quit IRC | 15:09 | |
| *** jpr has joined #openstack-infra | 15:09 | |
| fungi | bkero: if it uses openssl 0.9.8y then perhaps | 15:09 |
| bkero | oh jeez | 15:10 |
| bkero | They're going to run out of letters soon | 15:10 |
| fungi | heh | 15:10 |
| fungi | yes, after one more letter | 15:10 |
| bkero | Not saying it isn't worth deprecating. And even at that point if you make a webapp with cordova it'll bundle a new browser which likely does its own SSL/TLS. | 15:11 |
| *** jordanP has quit IRC | 15:11 | |
| *** korzen has quit IRC | 15:12 | |
| *** zz_dimtruck is now known as dimtruck | 15:12 | |
| fungi | we basically already engaged in this exercise once in the not too distant past, when we dropped ssl v3 support | 15:12 |
| AJaeger | sdague: let me check... | 15:12 |
| *** vgridnev has quit IRC | 15:13 | |
| AJaeger | fungi, you're too fast for me ;) | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Display a modal when a card is clicked on https://review.openstack.org/284280 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal to confirm archiving cards https://review.openstack.org/285417 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Allow a custom format string to be passed to time-moment https://review.openstack.org/284275 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a calendar directive https://review.openstack.org/278508 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a $resource wrapper for Due Dates https://review.openstack.org/284278 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add Due Dates to boards https://review.openstack.org/284279 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add onBlur and onFocus callbacks to user-typeahead https://review.openstack.org/284276 | 15:14 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Improve the board CSS a little https://review.openstack.org/284277 | 15:14 |
| clarkb | catching up ssl thing dosnt affect us because we are tls only? | 15:14 |
| AJaeger | project-config cores, could you review https://review.openstack.org/285949 https://review.openstack.org/#/c/285487/ and https://review.openstack.org/#/c/285148/ , please? Those are project-config changes that remove jobs... | 15:14 |
| *** vgridnev has joined #openstack-infra | 15:15 | |
| *** gnuoy_ has joined #openstack-infra | 15:16 | |
| *** vgridnev has quit IRC | 15:16 | |
| *** gnuoy_ has quit IRC | 15:16 | |
| fungi | clarkb: yep | 15:16 |
| Shrews | GheRivero: you're going to want to add a reno release note for the new quota apis | 15:17 |
| *** ajmiller_ has joined #openstack-infra | 15:17 | |
| *** ajmiller has joined #openstack-infra | 15:17 | |
| fungi | clarkb: we've pretty much arrived now at there's one change mordred wrote to improve an apache vhost template we're not actually using but someone else might, and discussing additional tls hardening opportunities unrelated to today's threat advisory | 15:18 |
| *** bgaifullin has quit IRC | 15:18 | |
| *** ajmiller has quit IRC | 15:19 | |
| *** sdake_ has joined #openstack-infra | 15:20 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:20 | |
| *** vgridnev has joined #openstack-infra | 15:21 | |
| *** pradk has quit IRC | 15:22 | |
| *** pradk has joined #openstack-infra | 15:22 | |
| rcarrillocruz | Shrews: shouldn't the get_quota thing something like get_<resource>_quota | 15:22 |
| rcarrillocruz | i.e. | 15:22 |
| rcarrillocruz | get_neutron_port_quota | 15:22 |
| rcarrillocruz | get_nova_ram_quota | 15:23 |
| rcarrillocruz | etc | 15:23 |
| *** sdake has quit IRC | 15:23 | |
| *** daemontool__ has joined #openstack-infra | 15:24 | |
| Shrews | rcarrillocruz: maybe? i haven't reviewed the code thoroughly yet | 15:25 |
| Shrews | rcarrillocruz: best to ask GheRivero | 15:25 |
| GheRivero | I don't know, get_nova_ram_quota looks too much/specific, but get_nova_quota could be an option. depending on the server version, you can have different set of quotas | 15:27 |
| *** mtanino has joined #openstack-infra | 15:27 | |
| *** daemontool_ has quit IRC | 15:28 | |
| GheRivero | and it can be wrapped in a more generic get_quotas, so you can get all the quotas without specifying which service to get quotas from | 15:29 |
| *** rossella_s has quit IRC | 15:32 | |
| *** rossella_s has joined #openstack-infra | 15:32 | |
| *** sshnaidm has joined #openstack-infra | 15:34 | |
| anteaya | AJaeger: is disabling translations for server projects part of feature freeze? https://review.openstack.org/#/c/285949/ | 15:35 |
| odyssey4me | FYI it would seem that the dsvm trusty image for OVH has an out of date/corrupt apt cache - we don't have to update the cache on any other providers as far as we've seen, but on OVH it appears to be mandatory | 15:36 |
| *** asettle has quit IRC | 15:36 | |
| pabelanger | odyssey4me: I've found cache on ubuntu in the gate to be flaky at best. For me, I've simply added the step in to ansible to ensure it has been updated before gating tests. | 15:37 |
| *** pblaho has quit IRC | 15:37 | |
| anteaya | the jenkinsii appear to be jenkinsing | 15:38 |
| odyssey4me | pabelanger yep, we're doing the same - I figured that it should be noted though in case it's not widely known | 15:38 |
| *** sbelous_ has quit IRC | 15:38 | |
| anteaya | ovh has an increased error node launch attempts: http://grafana.openstack.org/dashboard/db/nodepool-ovh | 15:39 |
| AJaeger | anteaya: it's part of translation process - they start now with mitaka | 15:39 |
| AJaeger | and thus liberty gets retired | 15:39 |
| anteaya | does ovh have floating ips, or perhaps we have maxed out on ovh quota? | 15:39 |
| anteaya | AJaeger: ah thank you | 15:39 |
| *** andymaier has joined #openstack-infra | 15:40 | |
| *** annegentle has quit IRC | 15:40 | |
| *** keedya has joined #openstack-infra | 15:40 | |
| clarkb | pabelanger: odyssey4me you should always update apt cache before doing anything else | 15:41 |
| AJaeger | anteaya: I discussed on i18n mailing list | 15:41 |
| AJaeger | anteaya: thanks for reviewing! | 15:41 |
| fungi | anteaya: yeah, now that the rax-iad quota issue has been identified, ovh is next in line for figuring out. i'll take a look shortly | 15:41 |
| openstackgerrit | Diana Whitten proposed openstack/requirements: Bump django-compressor to 2.0 https://review.openstack.org/286663 | 15:41 |
| *** jpr has quit IRC | 15:42 | |
| *** zeih has joined #openstack-infra | 15:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove gate-barbican-tox-bandit https://review.openstack.org/285487 | 15:42 |
| anteaya | AJaeger: okay thanks, I don't follow the i18n mailing list, thanks for helping me to understand | 15:42 |
| anteaya | AJaeger: and you're welcome | 15:42 |
| anteaya | fungi: thanks so much | 15:42 |
| *** ryanpetrello has quit IRC | 15:43 | |
| *** mrmartin has joined #openstack-infra | 15:43 | |
| *** jordanP has joined #openstack-infra | 15:43 | |
| *** esker has quit IRC | 15:43 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 15:43 |
| *** yamamoto has quit IRC | 15:43 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 15:43 |
| *** fhubik_brb is now known as fhubik | 15:44 | |
| *** icey has quit IRC | 15:44 | |
| *** ryanpetrello has joined #openstack-infra | 15:44 | |
| *** esker has joined #openstack-infra | 15:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove oslo.vmware bandit job https://review.openstack.org/285148 | 15:44 |
| openstackgerrit | Merged openstack-infra/project-config: Make ceph jobs non-voting until bug 1551305 is fixed https://review.openstack.org/286642 | 15:44 |
| openstack | bug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed] https://launchpad.net/bugs/1551305 | 15:44 |
| *** icey has joined #openstack-infra | 15:44 | |
| clarkb | odyssey4me: pabelanger our indexes are valid for a couple hours after new packages arrive iirc but that is still shorter than image refresh interval | 15:45 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Enable network isolation on all the jobs. https://review.openstack.org/285674 | 15:45 |
| *** amotoki has quit IRC | 15:46 | |
| smcginnis | mriedem: Thanks for working on that ceph issue. | 15:46 |
| *** zeih has quit IRC | 15:46 | |
| mriedem | o/ | 15:48 |
| openstackgerrit | Merged openstack-infra/project-config: Disable Liberty translations for Server projects https://review.openstack.org/285949 | 15:48 |
| pabelanger | clarkb: yup, that's what I do now | 15:49 |
| *** jaypipes has quit IRC | 15:50 | |
| *** jlanoux has quit IRC | 15:50 | |
| *** shardy has joined #openstack-infra | 15:51 | |
| *** yamahata has quit IRC | 15:51 | |
| fungi | yeah, so no clue why but nova boot in obh-gra1 is resulting in a lot of instances ending up in ERROR state | 15:52 |
| fungi | er, ovh-gra1 | 15:52 |
| clarkb | fungi: if you nova show their uuids sometimes you get more info | 15:52 |
| *** yamamoto has joined #openstack-infra | 15:52 | |
| fungi | yep, am about there | 15:52 |
| clarkb | anyone want to review (and hopfeully +A) https://review.openstack.org/#/c/285473/ ? I can start to work on booting a mirror in osic with that in | 15:53 |
| fungi | also bluebox is all "No more floating ips in pool external. (HTTP 404)" again. will clean up momentarily | 15:54 |
| *** kzaitsev_mb has quit IRC | 15:54 | |
| *** gokrokve has joined #openstack-infra | 15:54 | |
| anteaya | clarkb: I'm only +1 on system-config but you have mine | 15:55 |
| openstackgerrit | Merged openstack-infra/shade: Allow testing against Ansible dev branch https://review.openstack.org/285450 | 15:55 |
| fungi | by the time i can dig the failure and uuid out of the debug log, the instance is no longer there according to nova | 15:57 |
| anteaya | :( | 15:57 |
| *** maishsk has quit IRC | 15:58 | |
| pabelanger | fungi: AJaeger: Missed some backscroll over the last 24 hours, did we loop back to -bindep rollout discussion? | 15:59 |
| *** jsavak has quit IRC | 15:59 | |
| clarkb | ya nodepool can delete them pretty quickly, if you do a tail -f | grep ERROR and immediately run show on the output of that you might catch one | 15:59 |
| fungi | pabelanger: not yet. still trying to quell wildfires with a thimble brigade | 15:59 |
| anteaya | pabelanger: you know about eavesdrop.o.o yes? http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ | 15:59 |
| *** jsavak has joined #openstack-infra | 16:00 | |
| pabelanger | anteaya: indeed. I should start there first, before asking people to recap for me | 16:00 |
| pabelanger | fungi: sure, let me know if I can help | 16:00 |
| anteaya | pabelanger: no a recap is fine, sometimes you don't have time for the backscroll | 16:00 |
| anteaya | pabelanger: but you said you missed some, so wanted to make sure you had access to logs if you wanted | 16:00 |
| fungi | though at the moment the bluebox errors are so much more numerous than the ovh errors, i'm going to clean up the floating ip leak first | 16:01 |
| *** ilyashakhat has joined #openstack-infra | 16:01 | |
| openstackgerrit | Diana Whitten proposed openstack/requirements: Bump django-compressor to 2.0 https://review.openstack.org/286663 | 16:02 |
| anteaya | I'm hardly a role model at backscroll right now | 16:02 |
| fungi | grr, just realized i'm missing a conference call too | 16:03 |
| anteaya | :( | 16:03 |
| *** Guest67668 has joined #openstack-infra | 16:04 | |
| *** kevinbenton has quit IRC | 16:04 | |
| *** jaosorior is now known as jaosorior_away | 16:06 | |
| *** ajmiller_ has quit IRC | 16:06 | |
| *** ajmiller_ has joined #openstack-infra | 16:06 | |
| *** chlong_ has quit IRC | 16:10 | |
| *** jswarren__ is now known as jswarren | 16:11 | |
| pleia2 | clarkb: +Aed 285473 | 16:12 |
| clarkb | pleia2: woot thanks | 16:12 |
| *** abregman has quit IRC | 16:12 | |
| AndyU | @clarkb @nibalizer @jeblair @fungi etc : we had a sort conversation here on 1/29. I was asking about donating infrastructure. Have made some progress and now I have some more questions. Here's the potential scenario. The company I work for hosts a cloud for openstack which we would isolate from our network and expose to the internet. Potentially several hundred physical blade servers. These are machines that we'd be repurposi | 16:13 |
| AndyU | If one goes down we might not fix it and would just trash it. | 16:13 |
| *** e0ne has joined #openstack-infra | 16:13 | |
| *** Guest67668 is now known as annegentl_ | 16:13 | |
| AndyU | Questions: (1) How much support do you typically need from the cloud host? Can you give me an idea of the kinds of support and the frequency with which you tend to need it. | 16:13 |
| *** jaypipes has joined #openstack-infra | 16:13 | |
| AndyU | (2) Any concerns about hardware dying and not getting replaced? | 16:13 |
| *** jcooley_ has quit IRC | 16:14 | |
| *** arxcruz has quit IRC | 16:14 | |
| AndyU | (3) Does each node really need a public IP address?? That could be a real problem. Public IP's are getting to be in very short supply on my end I'm told. Any other options that you could foresee? | 16:14 |
| *** dizquierdo has quit IRC | 16:14 | |
| AndyU | Need info for a meeting in 2 hours time :-) | 16:14 |
| clarkb | AndyU: for your first message you cut off after "These are machines that we'd be repurposi" | 16:14 |
| *** dizquierdo has joined #openstack-infra | 16:15 | |
| pleia2 | AndyU: for 3, yes, it really needs a public ip | 16:15 |
| AndyU | oh sorry. I'll reformat and send again in smaller pieces. stand by ;-) | 16:15 |
| mordred | AndyU: we do support IPv6 - so public IPv6 works if there is not enough public IPv4 | 16:15 |
| pleia2 | AndyU: re: #2 - are you saying that over time the donation of nodes will decrease because you don't plan on replacing the hardware? | 16:16 |
| AndyU | completing the first post: These are machines that we'd be repurposing and are 2-3 years old. | 16:16 |
| cody-somerville | \o_ | 16:16 |
| pleia2 | oh, that's not so old | 16:17 |
| AndyU | ok, I'll investigate IP options on my end. | 16:17 |
| mordred | AndyU: for 1 - the main thing that we usually need is for things we physically can't do - we manage everything from baremetal provisioning on up | 16:17 |
| mordred | AndyU: so it's usually things related to switch config | 16:17 |
| *** Qiming has quit IRC | 16:17 | |
| mordred | that we are usually not allowed to touch where we need support | 16:17 |
| *** bpokorny has joined #openstack-infra | 16:17 | |
| mordred | but those usually only need to be set up once to start | 16:17 |
| mordred | (we also would probably nee to be able to report "dude, the entire network went away" to someone) | 16:17 |
| AndyU | Once switches, routers are configured do you find that they need to be changed? | 16:18 |
| clarkb | mordred: well wait | 16:18 |
| mordred | AndyU: not typically, no | 16:18 |
| clarkb | mordred: it depends on the type of donation, we can consume preexisting/managed cloud | 16:18 |
| mordred | clarkb: yes indeed | 16:18 |
| clarkb | mordred: or we can consume hardware more directly and provision the cloud ourselves | 16:18 |
| mordred | but if this is hardware donation for an infra-cloud region | 16:18 |
| clarkb | right so I think question one is what type of donation is this | 16:18 |
| clarkb | and that determines the type of support | 16:19 |
| AndyU | Do you prefer that we host a cloud and just provision you n number of nodes/tenants to do with as you please? | 16:19 |
| mordred | I got the idea from "If one goes down we might not fix it and would just trash it." that it was hardware - but that's an excellent question | 16:19 |
| fungi | right, we're still working through our very first hardware hosting donation for infra-cloud. we're pretty good at consuming donated cloud resources however | 16:19 |
| *** jlanoux has joined #openstack-infra | 16:20 | |
| anteaya | AndyU: I'm confident you were linked to http://docs.openstack.org/infra/system-config/contribute-cloud.html during an earlier conversation, just adding it here for completeness | 16:20 |
| mordred | AndyU: yah - that's easier - although the need for public IPs still exists | 16:20 |
| fungi | so preference at this point is still to have free accounts/quotas in a reachable openstack-based cloud | 16:20 |
| jeblair | i'm here, and supportive of this conversation, but am going to let other people talk | 16:20 |
| AndyU | We could potentially have a large numberof bare metal blade servers which we'd look to provide in whatever means is most advantagious to both sides. Want to eliminate the need for any kind of site access. Minimize support. | 16:20 |
| mordred | AndyU: yah. we also do not want to need site access or to bother you with a lot of support :) | 16:21 |
| AndyU | yes, I saw the link. Looking to dive deeper now | 16:21 |
| *** yamamoto has quit IRC | 16:21 | |
| anteaya | AndyU: great, thank you | 16:22 |
| *** yamamoto has joined #openstack-infra | 16:23 | |
| mtreinish | infra-root: we still need to restart the subunit gearman worker on subunit-worker01.o.o The fixes landed yesterday so hopefully it will stay up now | 16:24 |
| AndyU | Forgive my ignorance because I'm coming from a more management and less technical perspective on this. I just moved into this area a couple months ago. Knowing the resources potentially at our disposal, how would you optimally want them leveraged? | 16:24 |
| pleia2 | mtreinish: I don't know how to do that, but if one of my co-conspirators is willing to show me, we can take care of it | 16:24 |
| fungi | AndyU: ideally you would install and run a reachable openstack cloud on the hardware, give us accounts/quotas for the services we connect to it, and then be reachable somehow in case we run into issues with it | 16:25 |
| fungi | AndyU: where by "you" i of course mean someone in your organization | 16:25 |
| *** sorantis has quit IRC | 16:26 | |
| *** yamahata has joined #openstack-infra | 16:26 | |
| AndyU | ok. Does that openstack cloud need to be upgraded periodically, regularly by us? | 16:26 |
| openstackgerrit | mariam john proposed openstack/requirements: Add couchdb to global requirements https://review.openstack.org/285191 | 16:26 |
| mtreinish | pleia2: it should just be logging into the server and doing something like service subunit-gearman-worker-A start (or something like that) | 16:26 |
| fungi | probably. assuming you already run one or more openstack clouds anyway, you would probably just follow the same process you're doing elsewhere in your organization. i don't really know enough about how you're using openstack to guess | 16:27 |
| AndyU | ok, great | 16:27 |
| pleia2 | aha, someone as root has run /etc/init.d/jenkins-subunit-worker-A start | 16:28 |
| fungi | so far most of our donations have come from public cloud service providers, so the details around our needs are mostly from that perspective | 16:28 |
| dhellmann | what does it mean when the zuul status page reports a "queue length" of 2890 events? | 16:28 |
| pleia2 | but that's jenkins | 16:28 |
| anteaya | dhellmann: usually the gate is resetting | 16:28 |
| AndyU | So you don't really care how we necessarily host the cloud on our end, you just want a bunch of tenants and the keys to use them as you wish. Does that sound right? | 16:28 |
| *** _nadya_ has quit IRC | 16:28 | |
| fungi | AndyU: and reachable ipv4 or ipv6 addresses to be able to assign to the virtual instances we boot there, yes | 16:28 |
| dhellmann | anteaya : ok, thanks. that seemed like a large number, compared to what I'm used to seeing there | 16:29 |
| anteaya | dhellmann: 13 patches merging woot | 16:29 |
| *** _nadya_ has joined #openstack-infra | 16:29 | |
| mtreinish | pleia2: does that mean it's running? because we don't have anything being added to the db? maybe s/start/restart ? | 16:29 |
| anteaya | dhellmann: yes it spikes when zuul is recalculating | 16:29 |
| pleia2 | mtreinish: doesn't mean anything, I was just looking through the command history :) | 16:29 |
| mtreinish | ah, ok | 16:29 |
| *** ddecapit has joined #openstack-infra | 16:29 | |
| fungi | pleia2: i typically do `sudo service subunit-gearman-worker-A restart` so that probaly wasn't me | 16:30 |
| AndyU | ok, got it. And I hear you saying that support requests would likely be very infrequent and just related to hosting issues?? | 16:30 |
| jeblair | dhellmann: when zuul reloads its configuration, it drops its cache of gerrit changes, which slows it down a bit. | 16:30 |
| *** ddecapit is now known as DuaneDeC7 | 16:30 | |
| *** kevinbenton has joined #openstack-infra | 16:30 | |
| fungi | AndyU: yep, if we notice performance issues, or the environment goes offline, or something like that | 16:30 |
| *** armax has joined #openstack-infra | 16:31 | |
| fungi | AndyU: we're generally a pretty competent user, so tend not to reach out for assistance unless something is actually broken | 16:31 |
| pleia2 | mtreinish: ok, started -A | 16:31 |
| mtreinish | pleia2: cool, thanks | 16:31 |
| AndyU | cool. And on the question of hardare failures that might take down some nodes. I presume you could live with that? re-juggle on your end? It's unlikely but with that many servers it's bould to happen sometimes. | 16:32 |
| dhellmann | jeblair : somehow I seem to end up surprised by these sorts of cases and the change in behavior from what usually seems like "smooth" processing. I'll be happy when more of the release stuff is automated and I don't need to watch for jobs to finish. :-) | 16:32 |
| mtreinish | pleia2: I'll give it a min and see if anything is added, but we might need to do some debugging | 16:32 |
| pleia2 | mtreinish: sure, I'm around | 16:33 |
| *** vincentll has quit IRC | 16:33 | |
| *** _nadya_ has quit IRC | 16:33 | |
| *** fawadkhaliq has quit IRC | 16:33 | |
| jeblair | dhellmann: yeah; the cache is becoming enough of a problem that i think we're going to seriously rework it in zuulv3 | 16:33 |
| *** ajmiller__ has joined #openstack-infra | 16:33 | |
| jeblair | dhellmann: but just to be sure, i logged in and checked the log, and it is indeed busy querying gerrit as fast as it can | 16:34 |
| dhellmann | jeblair : oh, I wasn't complaining about zuul, just commenting on my own pattern of behavior | 16:34 |
| *** tiswanso has quit IRC | 16:34 | |
| *** yamamoto has quit IRC | 16:34 | |
| *** yamamoto has joined #openstack-infra | 16:34 | |
| jeblair | dhellmann: no worries. i'm just over-sharing. | 16:34 |
| *** yamamoto has quit IRC | 16:34 | |
| dhellmann | jeblair : np, I like learning | 16:35 |
| *** tiswanso has joined #openstack-infra | 16:35 | |
| AndyU | Let me explain further - we could have cases where say you were getting 200 nodes, something breaks and now we can only give you 196. Presumable you can cope with that? | 16:35 |
| *** ajmiller_ has quit IRC | 16:35 | |
| anteaya | wow that really cleared out the gate, yay merging patches | 16:35 |
| anteaya | AndyU: we can reset quota easily | 16:36 |
| anteaya | we just change a yaml file | 16:36 |
| *** yamahata has quit IRC | 16:36 | |
| AndyU | Ok. I expected that but I just wanted it said here. That's all I need for now! <fingers crossed> Thanks all ;-) | 16:37 |
| clarkb | AndyU: AndyU yup that is something we can handle fairly easily | 16:37 |
| anteaya | clarity is great, thank you | 16:37 |
| mtreinish | pleia2: ok is there anything in the log? I'd expect it to have added something by now (assuming there was a backlog and the previous fail state didn't drop everything) | 16:38 |
| *** cloudtrainme has quit IRC | 16:38 | |
| *** e0ne has quit IRC | 16:38 | |
| AndyU | The bigger hrudle on our end is probably less getting things set up and made available and more concerns over the ongoing support. The info you gave me is very helfull. Thanks again. | 16:38 |
| *** eharney has quit IRC | 16:39 | |
| *** apoorvad has joined #openstack-infra | 16:39 | |
| *** esikachev has quit IRC | 16:39 | |
| *** vgridnev has quit IRC | 16:40 | |
| *** e0ne has joined #openstack-infra | 16:40 | |
| *** eharney has joined #openstack-infra | 16:41 | |
| openstackgerrit | Yih Leong Sun proposed openstack-infra/infra-manual: Suggest to include a link to setting up gerrit on windows env. https://review.openstack.org/286703 | 16:41 |
| *** madorn has quit IRC | 16:41 | |
| pleia2 | mtreinish: checking | 16:41 |
| *** vgridnev has joined #openstack-infra | 16:41 | |
| openstackgerrit | Merged openstack/requirements: Add networking-nec to projects.txt https://review.openstack.org/280137 | 16:41 |
| openstackgerrit | Merged openstack-infra/system-config: Add OSIC clouds.yaml details https://review.openstack.org/285473 | 16:41 |
| pleia2 | mtreinish: heh, not running anymore, digging a bit more | 16:41 |
| mtreinish | haha | 16:42 |
| mtreinish | well, that would explain it :) | 16:42 |
| *** DuaneDeC7 has quit IRC | 16:42 | |
| *** DuaneDeC7 has joined #openstack-infra | 16:42 | |
| pleia2 | mtreinish: had to remove a stray pid file, seems better now | 16:42 |
| anteaya | AndyU: understood, do share if there are further quesions | 16:43 |
| anteaya | AndyU: and thanks for thinking of us! | 16:44 |
| mtreinish | pleia2: ok, it added 1 run to the db, but I don't see any others being added | 16:45 |
| mtreinish | is there anything in the log, or is it just being sluggish | 16:46 |
| pleia2 | mtreinish: seeing a lot of http://paste.openstack.org/show/488789/ | 16:46 |
| *** mrmartin has quit IRC | 16:46 | |
| *** fhubik has quit IRC | 16:47 | |
| *** exploreshaifali has joined #openstack-infra | 16:47 | |
| *** thorst_ is now known as thorst_afk | 16:47 | |
| mtreinish | pleia2: that's expected I think, although I have no idea why the gearman client is adding events with urls like that | 16:48 |
| *** pfallenop has quit IRC | 16:48 | |
| mtreinish | oh, actually it's probably because: https://review.openstack.org/#/c/281383/ hasn't landed | 16:48 |
| *** sridhar_ram1 has joined #openstack-infra | 16:48 | |
| *** pfallenop has joined #openstack-infra | 16:48 | |
| *** scheuran has quit IRC | 16:49 | |
| mtreinish | pleia2: but is there anything besides 404s being logged? | 16:49 |
| clarkb | I am debugging my mount_volume script additions and after attaching /dev/vdc via cinder then mkpart lvming on /dev/vdc no /dev/vdc1 device is mknod'd in /dev | 16:49 |
| clarkb | I shouldn't need to explicitly mknod right? | 16:49 |
| pleia2 | mtreinish: ok, also this http://paste.openstack.org/show/488792/ | 16:50 |
| *** jpr has joined #openstack-infra | 16:50 | |
| AndyU | @pleia2 Reading back I don't think I clearly addressed your question "[10:16] <pleia2> AndyU: re: #2 - are you saying that over time the donation of nodes will decrease because you don't plan on replacing the hardware?" - Yes, we might not replace them or it might be a long time before more servers are added to fill the gap. | 16:51 |
| mtreinish | pleia2: ah, that is more likely the real problem | 16:51 |
| pleia2 | mtreinish: but I guess I should only be looking for jobs with grenade-dsvm in them :) | 16:51 |
| pleia2 | AndyU: I think it came out in the end, thank you | 16:51 |
| mtreinish | pleia2: can you stop the worker so we don't exhaust the backlog and I'll push a fix for that | 16:51 |
| *** gokrokve has quit IRC | 16:52 | |
| pleia2 | mtreinish: ok, done | 16:52 |
| AndyU | Agree - again... just want tobe clear ;-) | 16:52 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job https://review.openstack.org/279218 | 16:52 |
| mtreinish | pleia2: although I'm not sure what the fix is :) | 16:52 |
| *** vgridnev has quit IRC | 16:52 | |
| *** ilyashakhat has quit IRC | 16:52 | |
| *** sridhar_ram1 has quit IRC | 16:53 | |
| *** vgridnev has joined #openstack-infra | 16:53 | |
| pleia2 | mtreinish: I can email you the debug log if you want to dig through for other things | 16:53 |
| *** sridhar_ram1 has joined #openstack-infra | 16:53 | |
| *** toabctl has quit IRC | 16:53 | |
| mtreinish | pleia2: sure, that might help | 16:54 |
| *** sridhar_ram1 has quit IRC | 16:54 | |
| *** sridhar_ram1 has joined #openstack-infra | 16:54 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 16:54 |
| pleia2 | mtreinish: sent | 16:54 |
| mtreinish | but I'm not sure what would be closing the IO object before we pass it into subunit2sql (which is what the exception is indicating is happening) | 16:54 |
| mtreinish | pleia2: thanks | 16:54 |
| *** rcernin has quit IRC | 16:55 | |
| *** jpr has quit IRC | 16:55 | |
| anteaya | clarkb: I don't know | 16:56 |
| *** kzaitsev_mb has joined #openstack-infra | 16:56 | |
| *** fawadkhaliq has joined #openstack-infra | 16:56 | |
| *** mikelk has quit IRC | 16:56 | |
| *** toabctl has joined #openstack-infra | 16:57 | |
| anteaya | wooot I got an out of office reply in Welsh | 16:57 |
| pleia2 | re: http://lists.openstack.org/pipermail/openstack-infra/2016-March/003941.html do we just drop the log file on the filesystem? (no bot magic will delete it?) | 16:58 |
| pleia2 | I'll fix up the filename too | 16:58 |
| anteaya | thats what you get for emailing someone from Cardiff University | 16:58 |
| pleia2 | hehe | 16:58 |
| anteaya | I think that is the process, yolanda has added logs successfully in the past, I believe | 16:59 |
| *** DuaneDeC7 has quit IRC | 17:00 | |
| fungi | anteaya: they're quite proud of their language, even if only something like 25% are fluent in it and 4% speak it as their first language (last time i looked at the statistics on it anyway) | 17:00 |
| anteaya | it is an awesome language | 17:00 |
| yolanda | yes, i did it my just creating the file in the right place, and ensuring perms | 17:00 |
| fungi | i spent a few years trying to learn welsh, but it's extremely opaque to anyone who doesn't already have some familiarity with celtic languages (which i most certainly do not) | 17:01 |
| anteaya | they have every reason to be proud | 17:01 |
| *** annegentl_ has quit IRC | 17:01 | |
| clarkb | partprobe does seem to add a /dev/vdc1 but pvcreate is still unhappy about it and blkid exits 2 on /dev/vdc1 | 17:01 |
| clarkb | so weird | 17:01 |
| anteaya | I met mhickey at neutron mid-cycle | 17:01 |
| anteaya | he is looking into information for me so that I might learn irish | 17:01 |
| anteaya | sounds like a lot of fun | 17:01 |
| fungi | exciting! | 17:01 |
| anteaya | I'm looking forward to finding out more | 17:02 |
| anteaya | not sure I'm ready for welsh yet | 17:02 |
| fungi | though that's a northern celtic language while welsh is a southern celtic language, so they're pretty different linguistically | 17:02 |
| anteaya | so much I have to learn | 17:02 |
| *** matrohon has quit IRC | 17:02 | |
| anteaya | clarkb: :( | 17:02 |
| *** gyee has joined #openstack-infra | 17:03 | |
| fungi | other northern celtic languages include scottish gaelic and manx | 17:03 |
| anteaya | I don't know manx at all | 17:03 |
| *** jsavak has quit IRC | 17:03 | |
| fungi | southern celtic languages besides welsh are basically all dead (gaulish) or an academic curiosity (cornish, brethonic) | 17:03 |
| *** annegentl_ has joined #openstack-infra | 17:04 | |
| anteaya | bring back the languages! | 17:04 |
| nibalizer | wooo more cloud donations, thanks AndyU | 17:04 |
| fungi | er, breton i meant | 17:04 |
| *** vgridnev has quit IRC | 17:04 | |
| *** jsavak has joined #openstack-infra | 17:05 | |
| *** ifarkas has quit IRC | 17:05 | |
| *** jistr has quit IRC | 17:05 | |
| *** yamamoto has joined #openstack-infra | 17:05 | |
| anteaya | fungi: you could just be making up words, I wouldn't know the difference | 17:05 |
| fungi | oh, and cumbric... i'd almost forgotten about that one | 17:05 |
| anteaya | how can you forget about cumbric | 17:05 |
| clarkb | fungi: do you know if http://docs.openstack.org/infra/system-config/sysadmin.html#cinder-volume-management is relying on an implicit reboot between volume attach and pvcreate? | 17:06 |
| fungi | clarkb: you should not need a reboot between those steps, no | 17:06 |
| fungi | clarkb: kernel hotplugging should cause the disk to just appear magically | 17:06 |
| *** maishsk has joined #openstack-infra | 17:06 | |
| clarkb | thats a good point the disk does and parted seems to wrok fine | 17:07 |
| clarkb | pvcreate however is one unhappy camper | 17:07 |
| *** pcaruana has quit IRC | 17:07 | |
| fungi | clarkb: usually i double-check dmesg and then make sure the kernel added the device to the /dev tree | 17:07 |
| iremizov | Hi guys. Could you please review this patch set https://review.openstack.org/#/c/284680/ | 17:07 |
| clarkb | fungi: yup I have it reliably added to /dev with partprobe | 17:08 |
| clarkb | fungi: http://paste.openstack.org/show/488795/ | 17:08 |
| fungi | thanks, was just about to ask what error it was throwing | 17:09 |
| *** Swami has joined #openstack-infra | 17:10 | |
| *** yamamoto has quit IRC | 17:10 | |
| *** achanda has joined #openstack-infra | 17:10 | |
| *** hashar has quit IRC | 17:10 | |
| *** jpr has joined #openstack-infra | 17:11 | |
| fungi | huh, i wonder if something is setting weirdness in the mbr? | 17:11 |
| fungi | how does parted describe /dev/vdc? | 17:11 |
| fungi | msdos not gpt hopefully | 17:12 |
| fungi | since you explicitly told it to create an msdos partition table | 17:12 |
| clarkb | http://paste.openstack.org/show/488797/ from parted --list | 17:13 |
| clarkb | yup msdos | 17:13 |
| fungi | try -vvvv with pvcreate to get more details? | 17:13 |
| fungi | weird device type maybe? | 17:14 |
| mtreinish | pleia2: so I'm still at a loss for what is closing the IO object before we parse it. My only thought is we're reuising the old object (because we close that after we're done) | 17:14 |
| mtreinish | but the python docs for Queue say get() will remove and return a queued item | 17:14 |
| pleia2 | :\ | 17:15 |
| clarkb | http://paste.openstack.org/show/488798/ that isn't much more info | 17:15 |
| *** ashleighfarnham has joined #openstack-infra | 17:15 | |
| clarkb | Iam about to attach a second device and rerun through steps manually to see if something in my script is wrong | 17:16 |
| fungi | /dev/vdc1: Skipping (regex) | 17:16 |
| *** maishsk_ has joined #openstack-infra | 17:16 | |
| fungi | odd | 17:16 |
| clarkb | is that sayin a regex somewhere says don't lvm this? | 17:17 |
| fungi | look in /etc/lvm.conf? | 17:17 |
| *** dims_ has joined #openstack-infra | 17:17 | |
| *** zeih_ has joined #openstack-infra | 17:17 | |
| *** maishsk has quit IRC | 17:17 | |
| *** maishsk_ is now known as maishsk | 17:17 | |
| fungi | maybe we have something weird in there? | 17:17 |
| *** dims has quit IRC | 17:19 | |
| *** salv-orl_ has joined #openstack-infra | 17:19 | |
| clarkb | http://paste.openstack.org/show/488800/ | 17:20 |
| clarkb | the filter should be fine but maybe global_filter is breaking us | 17:20 |
| *** bpokorny has quit IRC | 17:20 | |
| *** ashleighfarnham has quit IRC | 17:20 | |
| *** esikachev has joined #openstack-infra | 17:22 | |
| *** salv-orlando has quit IRC | 17:22 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Update Entity Events processing https://review.openstack.org/286723 | 17:23 |
| fungi | i mean, the block devices on rax are /dev/vdXY so maybe check the lvm.conf on static.o.o? | 17:24 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Update Entity Events processing https://review.openstack.org/286723 | 17:24 |
| mtreinish | pleia2: yeah I just checked the Queue code and confirmed what the docs say. It does a deque popleft() when you call get() | 17:24 |
| mtreinish | so back to the drawing board | 17:24 |
| fungi | clarkb: that might be the only other prover where we're using a xen domu | 17:24 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for volume-backed live migration abort bug 1524898 https://review.openstack.org/286725 | 17:25 |
| openstack | bug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,Confirmed] https://launchpad.net/bugs/1524898 | 17:25 |
| clarkb | fungi: they are xvd* on rax (I am on osic with vd*) but there is no global filter on the mirror host, also appears to be not the same verison of ubuntu | 17:25 |
| clarkb | er s/mirror/static/ | 17:26 |
| clarkb | let me check mirror.ord | 17:26 |
| clarkb | no global on mirror.ord | 17:26 |
| *** esikachev has quit IRC | 17:26 | |
| clarkb | # from devstack makes me really skeptical | 17:27 |
| clarkb | and I did run devstack here so going to start from cleaner state | 17:27 |
| *** erlon has joined #openstack-infra | 17:27 | |
| clarkb | that was it | 17:27 |
| fungi | oh, right | 17:27 |
| fungi | what was the reason for running devstack on it? | 17:28 |
| *** notmorgan is now known as morgan | 17:28 | |
| clarkb | it was my osic test box | 17:28 |
| clarkb | so tested devstack run time and now testing volume attach | 17:28 |
| fungi | aha, so not the mirror host, just testing out the steps before making the mirror host | 17:28 |
| clarkb | yup | 17:29 |
| clarkb | that'll learn me | 17:29 |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| clarkb | I am going to leave the partprobe there even though I don't think it is strictly necessary | 17:29 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 17:30 |
| *** tiswanso has quit IRC | 17:30 | |
| *** degorenko is now known as _degorenko|afk | 17:31 | |
| *** f1ller is now known as filler | 17:32 | |
| *** tiswanso has joined #openstack-infra | 17:32 | |
| *** bpokorny has joined #openstack-infra | 17:32 | |
| *** cloudtrainme has joined #openstack-infra | 17:32 | |
| openstackgerrit | Francesco Longo proposed openstack-infra/project-config: Added IoTronic project. https://review.openstack.org/286113 | 17:32 |
| fungi | yeah, i don't think i've ever needed to partprobe on a modern kernel | 17:33 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Test Puppet Parser Future - Do not merge https://review.openstack.org/286732 | 17:33 |
| *** cloudtrainme has quit IRC | 17:33 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Add more debug logging for closed file issues https://review.openstack.org/286733 | 17:33 |
| mtreinish | pleia2: ^^^ | 17:33 |
| *** kushal has quit IRC | 17:33 | |
| mtreinish | that should help narrow it down at least | 17:33 |
| pleia2 | wfm | 17:34 |
| *** sfinucan has quit IRC | 17:34 | |
| *** mrmartin has joined #openstack-infra | 17:34 | |
| mrmartin | morning | 17:35 |
| *** thorst_afk has quit IRC | 17:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add support to shade-launch-node for cinder attach https://review.openstack.org/285477 | 17:35 |
| *** thorst_afk has joined #openstack-infra | 17:36 | |
| *** thorst_afk is now known as thorst_ | 17:36 | |
| mtreinish | pleia2: once that lands and puppet applies it we can restart the worker and collect more data | 17:37 |
| pleia2 | sounds good | 17:37 |
| mtreinish | that should hopefully let us figure out why it's trying to read closed files | 17:37 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Added expand=location to events endpoints https://review.openstack.org/286735 | 17:38 |
| mtreinish | I still think reusing the old one is the most likely case, because it worked the first time but failed all the others after | 17:38 |
| *** annegentl_ has quit IRC | 17:38 | |
| mtreinish | I just have no idea why that would be happening | 17:38 |
| pleia2 | would be good to get this one in too so it's not so noisy https://review.openstack.org/#/c/281383/ | 17:38 |
| mtreinish | pleia2: yep, that's a good call | 17:39 |
| dougwig | looks like gerritbot has gone to lunch in #openstack-lbaas. known issue? | 17:39 |
| SpamapS | trash: ACK, I will take a look later today. THanks for the reminder. | 17:39 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Added expand=location to events endpoints https://review.openstack.org/286735 | 17:39 |
| fungi | it still seems to be working in here | 17:39 |
| fungi | dougwig: can you elaborate? | 17:40 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/project-config: Adjust acls for fuel-noop-fixtures https://review.openstack.org/286109 | 17:40 |
| dougwig | fungi: the last "proposed" message was at 4:40am, and i just submitted a bunch. nothing. | 17:40 |
| fungi | dougwig: is openstackgerrit in the channel? | 17:40 |
| dougwig | fungi: yes. | 17:41 |
| *** jlanoux has quit IRC | 17:41 | |
| fungi | i'll check recent config changes for it | 17:41 |
| dougwig | fungi: ok, i'll peek there too. | 17:41 |
| dougwig | gerritbot/channels.yaml looks fine. | 17:42 |
| *** ihrachys has quit IRC | 17:42 | |
| *** Jeffrey4l has quit IRC | 17:43 | |
| nibalizer | clarkb: pleia2 want to look at https://review.openstack.org/#/c/285740/ ? I think that will make pretty timing graphs for our puppet-ansible runs | 17:43 |
| *** ihrachys has joined #openstack-infra | 17:43 | |
| *** derekh has quit IRC | 17:43 | |
| fungi | dougwig: so give me an example change number you submitted which never got echoed in channel | 17:44 |
| fungi | i'll have a look in the gerritbot debug logs | 17:44 |
| dougwig | fungi: 286380 | 17:44 |
| *** harlowja_at_home has joined #openstack-infra | 17:45 | |
| dougwig | fungi: PS1 at 9pm did echo. PS2 at 9:36am did not. | 17:45 |
| *** tphummel has joined #openstack-infra | 17:45 | |
| zaro | morning | 17:46 |
| clarkb | new fail mode, neutronclient --insecure does not seem to work | 17:47 |
| anteaya | morning zaro | 17:47 |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard https://review.openstack.org/286194 | 17:47 |
| *** jsavak has quit IRC | 17:47 | |
| *** HeOS has quit IRC | 17:47 | |
| *** ihrachys has quit IRC | 17:47 | |
| beisner | fungi, hi from the new openstack/charm-* projects (ci). we've got our bot reviewing --verified N on ci-sandbox OK. but that verified data isn't hitting the stream when we do the same against our projects. i must be missing something. thoughts on what to check? | 17:48 |
| fungi | dougwig: 2016-03-01 17:36:48,780 INFO gerritbot: Sending "Doug Wiegley proposed openstack/neutron-lbaas: WIP - delete lbaasv2 agent driver https://review.openstack.org/286380" to #openstack-lbaas | 17:48 |
| *** jsavak has joined #openstack-infra | 17:48 | |
| *** claudiub|2 has quit IRC | 17:49 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard https://review.openstack.org/286194 | 17:49 |
| fungi | beisner: by default acls don't allow arbitrary accounts to leave a verified label vote | 17:49 |
| dougwig | fungi: peek here, i said "lunch" right about when that should've hit: http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2016-03-01.log.html | 17:49 |
| fungi | beisner: so the acl would need to be adjusted to add that permission to a group we can put that account into | 17:50 |
| fungi | beisner: look at, say, the openstack/cinder acl config for an example | 17:50 |
| beisner | fungi, ack, thank you | 17:50 |
| *** sdake_ has quit IRC | 17:51 | |
| *** dizquierdo has quit IRC | 17:52 | |
| *** piet has joined #openstack-infra | 17:54 | |
| fungi | dougwig: yeah, and i don't see it restarting around that time, or disappearing in a netsplit or anything | 17:54 |
| fungi | could it have possibly been devoiced on that channel? | 17:55 |
| *** sbelous_ has joined #openstack-infra | 17:55 | |
| *** kushal has joined #openstack-infra | 17:55 | |
| anteaya | I think I will have some lunch before the meetings begin | 17:55 |
| dougwig | fungi: not that was reflected in channel. i also don't see a netsplit. | 17:56 |
| fungi | yeah, trying to scarf down some food myself so i can spend a few minutes prepping to chair the meeting | 17:56 |
| fungi | dougwig: yeah, i'm not finding obvious errors in the log, but still looking | 17:56 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add release announcement job to django_openstack_auth https://review.openstack.org/286747 | 17:56 |
| dougwig | fungi: i think the only people besides infra that have admin is the original channel owner (me), but unless someone has hacked my freenode account, i haven't recovered admin there in a **long** time. | 17:57 |
| *** vgridnev has joined #openstack-infra | 17:58 | |
| *** baoli has quit IRC | 17:58 | |
| *** jamesmcarthur has joined #openstack-infra | 17:59 | |
| fungi | dougwig: fwiw, that patch is the _only_ comment i see gerritbot logging it sent after the one which is reflected in the channel log at 12:40 utc | 17:59 |
| fungi | so it may not be so much "gerritbot has gone silent" as "one message from gerritbot never made it to the channel" | 17:59 |
| fungi | you suggested there were others? | 17:59 |
| clarkb | https://bugs.launchpad.net/python-neutronclient/+bug/1538959 now affects infra /me checks the this bug affects me flag | 17:59 |
| openstack | Launchpad bug 1538959 in python-neutronclient "--insecure option did not take effect" [Undecided,In progress] - Assigned to Zhongcheng Lao (zlao) | 17:59 |
| dougwig | fungi: indeed, another that i just sent did show up. | 17:59 |
| dougwig | fungi: i have to disappear into a meatspace meeting for a bit. | 18:00 |
| *** krtaylor has quit IRC | 18:00 | |
| *** sc68cal has joined #openstack-infra | 18:00 | |
| fungi | the gerritbot debug log makes no mention of #openstack-lbaas between 12:40:51 and 17:36:48 utc | 18:01 |
| *** sbelous_ has quit IRC | 18:01 | |
| fungi | and the 17:36 logs were only for your 286380 patch upload | 18:01 |
| *** jsavak has quit IRC | 18:02 | |
| clarkb | for those following along if you downgrade neutronclient then everything works | 18:02 |
| dougwig | fungi: i'll keep an eye on it. i was getting complaints last week of some missing announcements, but i didn't start watching until today. | 18:02 |
| fungi | clarkb: backward-incompatible change in neutronclient? | 18:02 |
| fungi | dougwig: thanks, more examples will hopefully help narrow this down | 18:02 |
| *** jsavak has joined #openstack-infra | 18:03 | |
| clarkb | fungi: yes, --insecure is now a noop | 18:03 |
| clarkb | so if you hvae to not verify ssl as in case with osic then you can't use latest neutronclient | 18:03 |
| *** zeih_ has quit IRC | 18:03 | |
| fungi | clarkb: so we probably need to do something similar to what we've done for infra-cloud? | 18:03 |
| fungi | and if it's a self-signed cert, they may need to regenerate it with basic constranits set to let it act as a ca | 18:04 |
| *** BobBall is now known as BobBall_AWOL | 18:04 | |
| *** ajmiller__ is now known as ajmiller | 18:04 | |
| fungi | otherwise we just need to find out what ca it's signed by and add a trust for that | 18:05 |
| *** esikachev has joined #openstack-infra | 18:05 | |
| clarkb | fungi: we can't | 18:05 |
| clarkb | the issue is there is no DNS so the CN in the cert doesn't match | 18:05 |
| *** mriedem has quit IRC | 18:05 | |
| fungi | oh! | 18:05 |
| clarkb | we could hack /etc/hosts but meh | 18:06 |
| fungi | what's the cn on it? | 18:06 |
| *** sarob has joined #openstack-infra | 18:06 | |
| clarkb | cloud1.osic.rackspace.com it is self signed | 18:06 |
| *** mriedem has joined #openstack-infra | 18:06 | |
| clarkb | (I tested the infra-cloud hack and ran into ip vs cn mismatches and lack of dns records) | 18:06 |
| *** baoli has joined #openstack-infra | 18:07 | |
| *** yamamoto has joined #openstack-infra | 18:07 | |
| clarkb | ok security groups in both osic accounts should be working properly now | 18:07 |
| anteaya | hopefully | 18:07 |
| anteaya | also is now a good time to have the neutron default security groups chat? | 18:08 |
| clarkb | next step is boot a mirror but I need to run errands (that unfortunately overlap with our meeting today) | 18:08 |
| fungi | clarkb: yeah, so presumably we need to get them to add a dns record for that name | 18:08 |
| clarkb | fungi: yup if they did that we could use infra-cloud hack | 18:08 |
| anteaya | clarkb: oh so I guess not right now | 18:08 |
| clarkb | anteaya: no, not good for me right now | 18:08 |
| fungi | otherwise that's probably stretching the bounds of just-a-little-too-broken | 18:08 |
| *** e0ne has quit IRC | 18:08 | |
| clarkb | fungi: I mean the only things that can get compromised are the resources in the broken service :) | 18:09 |
| anteaya | clarkb: nod | 18:09 |
| clarkb | even the afs stuff is all read only which anyone on the planet has access to? | 18:09 |
| fungi | but also the infra-cloud trust hack still depends on the self-signed cert also having the right (non-default for openssl) basic constraints configuration | 18:09 |
| clarkb | I was trying to think of a situation where something else could be in trouble if we got man in the middled | 18:10 |
| *** kzaitsev_mb has quit IRC | 18:10 | |
| Shrews | so, https://review.openstack.org/285455 is not moving through the gate, and i'm a bit stumped why. who wants to be the first to point out where i'm being stupid?? :) | 18:10 |
| *** vgridnev has quit IRC | 18:10 | |
| Shrews | there will be prizes | 18:10 |
| clarkb | Shrews: if that was approved before the shade dependency was merged it will need to be approved again | 18:10 |
| cody-somerville | crinkle: Hey. Thanks for getting those requirements to Allison. Do you know if we have confirmation that the machines were shipped out in time to avoid being affected by the March 1st freeze? | 18:10 |
| clarkb | Shrews: zuul won't auto queue deps that don't share a gate pipeline | 18:10 |
| Shrews | clarkb: yeah, that's the case. a stuffed unicorn will be sent your way. can you re-approve for me? | 18:11 |
| clarkb | fungi: afs is read only, ansible ssh only exposes the public key, etc | 18:11 |
| clarkb | Shrews: yes I can | 18:11 |
| Shrews | danke | 18:11 |
| fungi | clarkb: worst case for the comprimised trust is probably that nodepool trusts it to sign certs for other clouds. so _if_ someone got their hands on it _and_ also could redirect nodepool's network traffic/dns lookups, then it could maybe be used to malicious ends for anything we run on nodepool-managed workers | 18:11 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Make debootstrap cache opt-in https://review.openstack.org/285886 | 18:11 |
| fungi | but if they can get at the private key for that cert, then they can probably already compromise our nodepool-managed workloads in their environment anyway | 18:12 |
| *** yamamoto has quit IRC | 18:12 | |
| clarkb | right | 18:12 |
| fungi | hard to defend against an inside actor, the ssl cert is not the weakest link there | 18:13 |
| clarkb | except in the case of not verifying ssl they don't need the private key | 18:13 |
| *** jed56 has quit IRC | 18:13 | |
| fungi | oh, i get you. i was talking about the risks of nodepool trusting the self-signed cert they're using | 18:13 |
| clarkb | fungi: that isn't an option so Iam not really worrying about it right now | 18:13 |
| fungi | or trusting their cloud-local certificate authority even if the api cert is not self-signed | 18:13 |
| *** baoli has quit IRC | 18:14 | |
| fungi | clarkb: right, sorry, i was thinking ahead to once they add a dns recored | 18:14 |
| fungi | record | 18:14 |
| clarkb | well it is if we want to edit /etc/hosts and yolo dns | 18:14 |
| clarkb | but I don't relaly want to do that either | 18:14 |
| openstackgerrit | Merged openstack-infra/project-config: Added new repository for fuel-plugin-murano https://review.openstack.org/269567 | 18:14 |
| clarkb | we could just offer to expense the real cert and dns record for them | 18:14 |
| clarkb | I can pay that out of my weekly beer budget | 18:14 |
| fungi | that might not be a terrible short-term workaround, but it's definitely not a long-term solution and i'd rather just press them to fix their dns | 18:15 |
| *** dizquierdo has joined #openstack-infra | 18:15 | |
| fungi | the /etc/hosts workaround i mean | 18:15 |
| *** cznewt has quit IRC | 18:15 | |
| clarkb | ya | 18:15 |
| *** tongli has joined #openstack-infra | 18:15 | |
| AJaeger | pabelanger: EmilienM 's change for puppet-lint to use ubuntu-trusty merged and I haven't seen him complaining yet, so assume it's fine. EmilienM can you confirm? | 18:16 |
| AJaeger | pabelanger: but that's all progress on the bindep front... | 18:16 |
| EmilienM | I always complain, is that what you say? :-P | 18:17 |
| EmilienM | all is fine for us AFIK :-) let me double check | 18:17 |
| openstackgerrit | Merged openstack-infra/storyboard: Change MySQL search mode to 'boolean' https://review.openstack.org/281890 | 18:17 |
| crinkle | cody-somerville: i have not seen confirmation yet | 18:17 |
| EmilienM | AJaeger, pabelanger: all looks fine: http://logs.openstack.org/82/282182/3/gate/gate-puppet-glance-puppet-lint/e343f6a/console.html | 18:17 |
| fungi | clarkb: apparently there are ns records for osic.rackspace.com to ns.rackspace.com and ns2.rackspace.com so in theory they were planning to have resource records under that subdomain | 18:17 |
| *** openstackgerrit has quit IRC | 18:18 | |
| EmilienM | AJaeger, pabelanger: wait, the job run on bare-trusty, is it expected? | 18:18 |
| *** openstackgerrit has joined #openstack-infra | 18:18 | |
| pabelanger | AJaeger: neat, let me look and see what is going on with it. | 18:18 |
| pabelanger | EmilienM: Ya, it should be using ubuntu-trusty | 18:18 |
| pabelanger | when did it merge? | 18:18 |
| fungi | EmilienM: did the job-template use node: {node} instead of node: ubuntu-trusty? if so, it was probably taking your default node: bare-trusty from the instantiating project entry | 18:18 |
| EmilienM | pabelanger: last night | 18:19 |
| EmilienM | https://review.openstack.org/285542 | 18:19 |
| fungi | EmilienM: looks right to me. i'll check the jenkins master where that job ran for indication it actually got the updated config | 18:20 |
| pabelanger | EmilienM: AJaeger: https://jenkins04.openstack.org/job/gate-puppet-heat-puppet-lint/79/consoleText is using bindep | 18:20 |
| AJaeger | this one on ubuntu-trusty: http://logs.openstack.org/76/281376/6/check/gate-puppet-pacemaker-puppet-lint/b2e28e1/console.html | 18:20 |
| EmilienM | pabelanger: when did tat run? | 18:21 |
| EmilienM | ok so we're good | 18:21 |
| pabelanger | I don't expect much issues with puppet-lint honestly, since they installed gem files themself | 18:21 |
| pabelanger | EmilienM: now, in the gate | 18:21 |
| EmilienM | fungi: sorry for the wrong link though the job ran this morning | 18:21 |
| AJaeger | pabelanger: my change failed wtih " Gems in the group system_tests were not installed." -is that ok? | 18:21 |
| pabelanger | 286717 | 18:21 |
| EmilienM | pabelanger: nice catch! thanks again | 18:21 |
| *** kzaitsev_mb has joined #openstack-infra | 18:21 | |
| pabelanger | AJaeger: where did you see that? | 18:22 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting shade job to test upstream Ansible https://review.openstack.org/285455 | 18:22 |
| AJaeger | pabelanger: http://logs.openstack.org/76/281376/6/check/gate-puppet-pacemaker-puppet-lint/b2e28e1/console.html - but error is something else | 18:22 |
| *** lucasagomes is now known as lucas-dinner | 18:22 | |
| AJaeger | pabelanger: I see it also on http://logs.openstack.org/82/282182/3/check/gate-puppet-glance-puppet-lint/cce794c/console.html - which succeeds | 18:23 |
| pabelanger | AJaeger: Ya, that is a pep8 failure for puppet | 18:23 |
| pabelanger | formatting issue on the patchset | 18:23 |
| AJaeger | So, we're fine on the bindep front, great! | 18:23 |
| pabelanger | yup | 18:23 |
| pabelanger | I'm going to add an experimental job for ansible roles now | 18:24 |
| *** flepied has quit IRC | 18:24 | |
| fungi | EmilienM: on jenkins02 where your example ran, the gate-puppet-glance-puppet-lint job is not updated to use bindep and ubuntu-trusty nodes | 18:24 |
| *** sputnik13 has joined #openstack-infra | 18:24 | |
| EmilienM | fungi: do I need to patch something else? | 18:25 |
| AJaeger | fungi: manual jjb run needed? | 18:25 |
| fungi | yeah, i think jjb updates are probably broken on at least one master again. i'll try to get them synced up while i prep for the meeting | 18:25 |
| AJaeger | thanks, fungi | 18:25 |
| *** fabio_ has quit IRC | 18:25 | |
| AJaeger | fungi, still ok to add an agenda item? | 18:26 |
| *** serverascode has quit IRC | 18:26 | |
| fungi | EmilienM: so basically, the update has propagated far enough that it looks like the new job works fine, we're just still running the old version of the job on some changes until i fix jjb updates | 18:26 |
| fungi | AJaeger: sure | 18:26 |
| * AJaeger adds at the end of a long list | 18:27 | |
| fungi | AJaeger: just be mindful that the agenda may be getting full so we could run out of time | 18:27 |
| fungi | yeah, that | 18:27 |
| *** zhiyan has quit IRC | 18:27 | |
| *** cznewt has joined #openstack-infra | 18:27 | |
| EmilienM | fungi: last question: puppet CI has some jobs runnong on devstack-trusty - should I switch them too? | 18:27 |
| *** weshay has quit IRC | 18:27 | |
| *** blogan_ is now known as blogan | 18:27 | |
| * AJaeger managed the captcha | 18:28 | |
| *** serverascode has joined #openstack-infra | 18:28 | |
| *** zhiyan has joined #openstack-infra | 18:28 | |
| clarkb | fungi Ya I will send them email asking if they plan to get that done soon | 18:28 |
| clarkb | fungi any thoughts on whether or not the mirror host and maybe a max-servers: 1 nodepool config shluld wait on that? | 18:29 |
| fungi | EmilienM: should be fine to, yes. the ubuntu-trusty nodes are supposed to be basically identical to devstack-trusty, except that we can't easily update ubuntu-trusty in rackspace while we still build devstack-trusty there via an older snapshot-based method | 18:30 |
| EmilienM | fungi: ack | 18:30 |
| *** kushal has quit IRC | 18:30 | |
| *** jaosorior_away is now known as jaosorior | 18:31 | |
| *** ashtokolov_ has joined #openstack-infra | 18:31 | |
| *** evgenyl_ has joined #openstack-infra | 18:31 | |
| anteaya | AJaeger: yay | 18:31 |
| *** roaet- has joined #openstack-infra | 18:31 | |
| *** jordanP has quit IRC | 18:33 | |
| *** sambetts is now known as sambetts|afk | 18:33 | |
| fungi | EmilienM: so the general risk with ubuntu-trusty nodes at the moment is that in rackspace they may lag behind updates in other providers due to our glance issues there | 18:33 |
| fungi | though maybe that's solved now? | 18:33 |
| fungi | clarkb: do you recall the most recent status on that front? | 18:33 |
| *** kzaitsev_mb has quit IRC | 18:34 | |
| EmilienM | fungi: ok good to know. I'll rune xperimental jobs first, like you did. | 18:34 |
| AJaeger | anteaya: could you put 286497 and 286527 on your review queue, please? One is a skip-rule addition for manila, the other resorts all of them. | 18:34 |
| anteaya | I will look after soup | 18:35 |
| clarkb | fungi still broken I pasted the error last night | 18:35 |
| fungi | k | 18:35 |
| clarkb | fungi no tenant specifird on swift token reup | 18:35 |
| fungi | thanks | 18:35 |
| clarkb | if you look at the builder debug log you can see traceback or findy paste link | 18:36 |
| *** roaet_ has quit IRC | 18:37 | |
| *** ashtokolov has quit IRC | 18:37 | |
| *** evgenyl has quit IRC | 18:37 | |
| AJaeger | is there a specific reason that django_openstack_auth does not use publish-to-pypi template? See https://review.openstack.org/#/c/286747/1/zuul/layout.yaml | 18:37 |
| *** ashtokolov_ is now known as ashtokolov | 18:37 | |
| *** evgenyl_ is now known as evgenyl | 18:37 | |
| AJaeger | thanks, anteaya . Enjoy your soup! | 18:39 |
| *** krtaylor has joined #openstack-infra | 18:42 | |
| *** BobBall_1WOL has joined #openstack-infra | 18:44 | |
| *** dims has joined #openstack-infra | 18:45 | |
| *** dmellado has quit IRC | 18:48 | |
| *** dkehn has quit IRC | 18:50 | |
| *** BobBall_AWOL has quit IRC | 18:50 | |
| *** itsuugo has quit IRC | 18:50 | |
| *** dkehn has joined #openstack-infra | 18:50 | |
| *** wolsen has quit IRC | 18:50 | |
| *** mdenny has quit IRC | 18:52 | |
| *** dims_ has quit IRC | 18:52 | |
| *** rbrndt has quit IRC | 18:52 | |
| *** pahuang has quit IRC | 18:52 | |
| *** rbrndt has joined #openstack-infra | 18:52 | |
| *** pahuang has joined #openstack-infra | 18:52 | |
| *** ociuhandu has quit IRC | 18:52 | |
| *** acabot has quit IRC | 18:52 | |
| *** afazekas has quit IRC | 18:52 | |
| *** _ody has quit IRC | 18:52 | |
| *** xiangxinyong has quit IRC | 18:54 | |
| *** aeng has quit IRC | 18:54 | |
| *** bpokorny has quit IRC | 18:54 | |
| anteaya | thanks | 18:54 |
| *** SpamapS has quit IRC | 18:54 | |
| *** openstack has joined #openstack-infra | 19:15 | |
| *** SpamapS has joined #openstack-infra | 19:15 | |
| anteaya | meetbot is still missing in #openstack-meeting-3 if someone has a moment | 19:15 |
| *** openstackstatus has joined #openstack-infra | 19:17 | |
| *** ChanServ sets mode: +v openstackstatus | 19:17 | |
| anteaya | meetbot is back in meeting-3 now, thank you | 19:18 |
| *** stevelle has left #openstack-infra | 19:18 | |
| *** sripriya_ has joined #openstack-infra | 19:18 | |
| *** esikachev has joined #openstack-infra | 19:19 | |
| *** openstackgerrit has joined #openstack-infra | 19:19 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on OR filtering https://review.openstack.org/286786 | 19:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix create_server() with a named network https://review.openstack.org/286787 | 19:23 |
| *** maximov_ has quit IRC | 19:25 | |
| *** bryan_att has quit IRC | 19:25 | |
| *** mgkwill has quit IRC | 19:25 | |
| *** markmcclain has quit IRC | 19:25 | |
| *** gnuoy has quit IRC | 19:25 | |
| *** clif_h has quit IRC | 19:25 | |
| *** odyssey4me has quit IRC | 19:25 | |
| *** mwhahaha has quit IRC | 19:25 | |
| *** msuriar has quit IRC | 19:25 | |
| *** briancurtin has quit IRC | 19:25 | |
| *** ikalnitsky has quit IRC | 19:25 | |
| *** Adri2000 has quit IRC | 19:25 | |
| *** sulo has quit IRC | 19:25 | |
| *** whoops has quit IRC | 19:25 | |
| *** StevenK has quit IRC | 19:25 | |
| *** lane_kong has quit IRC | 19:25 | |
| *** mhayden has quit IRC | 19:25 | |
| *** niska has quit IRC | 19:25 | |
| *** nibalizer has quit IRC | 19:25 | |
| *** alaski has quit IRC | 19:25 | |
| *** bauzas has quit IRC | 19:25 | |
| *** tdasilva has quit IRC | 19:25 | |
| *** johnthetubaguy has quit IRC | 19:25 | |
| *** maishsk has quit IRC | 19:26 | |
| *** sc68cal has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on OR filtering https://review.openstack.org/286786 | 19:28 |
| *** ikalnitsky has joined #openstack-infra | 19:31 | |
| *** maximov_ has joined #openstack-infra | 19:31 | |
| *** bryan_att has joined #openstack-infra | 19:31 | |
| *** mgkwill has joined #openstack-infra | 19:31 | |
| *** markmcclain has joined #openstack-infra | 19:31 | |
| *** gnuoy has joined #openstack-infra | 19:31 | |
| *** clif_h has joined #openstack-infra | 19:31 | |
| *** odyssey4me has joined #openstack-infra | 19:31 | |
| *** mwhahaha has joined #openstack-infra | 19:31 | |
| *** bauzas has joined #openstack-infra | 19:31 | |
| *** msuriar has joined #openstack-infra | 19:31 | |
| *** briancurtin has joined #openstack-infra | 19:31 | |
| *** Adri2000 has joined #openstack-infra | 19:31 | |
| *** sulo has joined #openstack-infra | 19:31 | |
| *** whoops has joined #openstack-infra | 19:31 | |
| *** StevenK has joined #openstack-infra | 19:31 | |
| *** lane_kong has joined #openstack-infra | 19:31 | |
| *** mhayden has joined #openstack-infra | 19:31 | |
| *** niska has joined #openstack-infra | 19:31 | |
| *** nibalizer has joined #openstack-infra | 19:31 | |
| *** alaski has joined #openstack-infra | 19:31 | |
| *** tdasilva has joined #openstack-infra | 19:31 | |
| *** johnthetubaguy has joined #openstack-infra | 19:31 | |
| *** mhayden has quit IRC | 19:31 | |
| *** abregman has joined #openstack-infra | 19:32 | |
| openstackgerrit | Merged openstack-infra/storyboard: Updated documentation for installing Storyboard https://review.openstack.org/286194 | 19:32 |
| *** mhayden has joined #openstack-infra | 19:33 | |
| *** geekinutah has joined #openstack-infra | 19:36 | |
| *** taron1 has quit IRC | 19:37 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:37 | |
| geekinutah | folks, reading http://docs.openstack.org/infra/system-config/contribute-cloud.html | 19:37 |
| geekinutah | specifically in the requirements, "A public IP address" | 19:37 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 19:38 |
| nibalizer | geekinutah: hi, we're actually in a meeting in a different channel right now | 19:38 |
| nibalizer | so we'll be more 'here' in about 20 minutes | 19:38 |
| geekinutah | does it matter where this IP address is, like preference for fixed IP to be public or floating or both | 19:38 |
| geekinutah | nibalizer: np, I'll lurk while you meet | 19:38 |
| *** ajmiller_ has joined #openstack-infra | 19:39 | |
| mordred | geekinutah: we _prefer_ fixed public, but can handle floating if that's what you can provide | 19:39 |
| nibalizer | #openstack-meeting is the channel if you want to lurk that | 19:39 |
| mordred | geekinutah: (managing floating ips requires more api calls and is more prone to failures than clouds with public fixed ips via dhcp - but we have a floating ip cloud in our set currently, and we also ran on hp for a few years which was floating) | 19:39 |
| pabelanger | So user Martcheap on the wiki emailed me asking why his account was blocked. He used in email system on wiki.o.o. | 19:40 |
| geekinutah | mordred: makes sense, thx | 19:40 |
| pabelanger | looking at his contrib log, he is clearly spamming | 19:40 |
| pabelanger | I told him to connect here and talk about it | 19:40 |
| mordred | pabelanger: hah | 19:40 |
| mordred | pabelanger: you're so nice | 19:40 |
| pabelanger | indeed | 19:41 |
| mordred | pabelanger: I would have said "you are blocked because you are spamming" | 19:41 |
| docaedo | pabelanger: wow - that's some nerve! | 19:41 |
| pabelanger | well, mostly curious if they do join! And see what they say | 19:41 |
| *** vgridnev has joined #openstack-infra | 19:41 | |
| *** ajmiller has quit IRC | 19:41 | |
| docaedo | yeah great response, I think I'd have done the same, will be fun to see if they join | 19:42 |
| *** vgridnev has quit IRC | 19:43 | |
| *** Sukhdev has joined #openstack-infra | 19:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add test for os_server Ansible module https://review.openstack.org/285424 | 19:44 |
| *** esikachev has quit IRC | 19:46 | |
| nibalizer | pabelanger: bizzare | 19:47 |
| *** esikachev has joined #openstack-infra | 19:47 | |
| pabelanger | Ya, I guess we should check if they really are an openstack contributor or not | 19:48 |
| *** gyee has quit IRC | 19:48 | |
| bkero | "What? This spam-bot signed a CLA too?" | 19:48 |
| *** hashar has joined #openstack-infra | 19:49 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:52 | |
| *** ajmiller_ is now known as ajmiller | 19:52 | |
| openstackgerrit | Francesco Longo proposed openstack-infra/project-config: Added IoTronic project. https://review.openstack.org/286113 | 19:53 |
| *** erikwilson has joined #openstack-infra | 19:53 | |
| *** sc68cal has joined #openstack-infra | 19:53 | |
| *** terryw is now known as otherwiseguy | 19:54 | |
| mtreinish | nibalizer, mordred, clarkb, jeblair: if you get a sec can you look at: https://review.openstack.org/286733 and https://review.openstack.org/281383 | 19:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add test for os_server Ansible module https://review.openstack.org/285424 | 19:54 |
| *** erikwilson has quit IRC | 19:54 | |
| mtreinish | we need the first 1 as a first step in debugging why the subunit worker is passing closed files into subunit2sql | 19:54 |
| *** rockyg has joined #openstack-infra | 19:56 | |
| *** esikachev has quit IRC | 19:57 | |
| anteaya | pabelanger: I want you to get from them who is telling them to do this | 19:58 |
| anteaya | pabelanger: obviously us blocking them is blocking their paycheque | 19:58 |
| *** maishsk has joined #openstack-infra | 19:58 | |
| *** david-lyle has quit IRC | 19:59 | |
| *** kgiusti has left #openstack-infra | 19:59 | |
| annegentl_ | hi fungi, had another question about cutoff date for patches for the Austin summit, do you know? | 20:00 |
| pabelanger | :) | 20:00 |
| AJaeger | We run out of time in the infra-meeting, so here's what I wanted to share. armax, mestery,dougwig this is for you as | 20:01 |
| annegentl_ | I'm the one people ask, I must be super accessible :) | 20:01 |
| AJaeger | well: | 20:01 |
| AJaeger | Constraints are enabled for nova, glance, cinder. Neutron still uses -constraints jobs, patch up at https://review.openstack.org/286777 and https://review.openstack.org/286778 to move them over. | 20:01 |
| AJaeger | Now waiting for post jobs to get constraint enabled - jesusaurus, have you made any progress on that one? | 20:01 |
| fungi | annegentl_: saw in scrollback but then you disappeared | 20:01 |
| AJaeger | What's our timeline? What will we get done for Mitaka? | 20:01 |
| AJaeger | I'd like to ask lifeless, to write an email telling projects what do to do if they want to use constraints... | 20:01 |
| *** vgridnev has joined #openstack-infra | 20:01 | |
| fungi | annegentl_: i'm planning to send the last batch on thursday | 20:01 |
| annegentl_ | fungi: yeah sorry :) online and offline lately | 20:01 |
| annegentl_ | fungi: ok, sounds good, so land by Thursday? | 20:01 |
| armax | AJaeger: ack | 20:02 |
| mtreinish | fungi, I knew I was forgetting someone in my ping about the puppet-subunit2sql patches... :) | 20:02 |
| fungi | annegentl_: yes, it could be as early as thursday depending on how my week shapes up | 20:02 |
| armax | ihrachys and I talked about this and we were thinking of taking care of this after M3 | 20:02 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs https://review.openstack.org/273424 | 20:02 |
| armax | AJaeger: is that ok? | 20:02 |
| fungi | annegentl_: so merging by thursday is guaranteed safe | 20:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Use isinstance() for result type checking https://review.openstack.org/286811 | 20:02 |
| dougwig | AJaeger: "if they want to use" <-- is it really helpful if it's voluntary? | 20:02 |
| *** dizquierdo has quit IRC | 20:02 | |
| annegentl_ | fungi: cool thanks | 20:03 |
| mrmartin | pabelanger: what is the situation with the wiki? | 20:03 |
| AJaeger | armax: That's fine with me. I can make 777 as WIP until then. M3 is this week, correct? So, this is next week? | 20:03 |
| armax | AJaeger: aye | 20:03 |
| jesusaur | AJaeger: sorry, I've been firefighting internal issues for a few weeks, I haven't been able to debug the issues with my zuul-cloner change | 20:03 |
| AJaeger | dougwig: let's see how lifeless writes it up;) But it's for projects to enable it. | 20:03 |
| AJaeger | jesusaur: Ah, thanks for the update | 20:04 |
| *** vgridnev has quit IRC | 20:04 | |
| *** taron1 has joined #openstack-infra | 20:04 | |
| *** amitgandhinz has quit IRC | 20:04 | |
| armax | AJaeger: thanks for beating us to it | 20:04 |
| greghaynes | ianw: Would you say that simple-init has been most of your new distro debugging? | 20:04 |
| *** sdake has quit IRC | 20:05 | |
| ianw | greghaynes: that was a big part of it | 20:05 |
| greghaynes | ianw: thats kind of the assumption I have been operating on since thats what its looked like to me | 20:05 |
| ianw | mostly it has been fallout from the switch to "-minimal" build | 20:05 |
| *** daemontool__ has joined #openstack-infra | 20:05 | |
| ianw | which is closely tied to simple-init, but also more | 20:05 |
| greghaynes | ah | 20:05 |
| *** amitgandhinz has joined #openstack-infra | 20:05 | |
| AJaeger | armax: ;) | 20:06 |
| fungi | mrmartin: apparently the update is that at least one spammer has reached out to us asking why we blocked them | 20:06 |
| mrmartin | wow | 20:06 |
| ianw | greghaynes: i want to add back that docker image build job as a separate job from the build functional tests, sound ok? | 20:06 |
| mrmartin | and why? | 20:06 |
| fungi | mrmartin: my guess is this means spammers may have compromised legitimate lp accounts | 20:06 |
| greghaynes | ianw: why separate? | 20:06 |
| greghaynes | ianw: I had no idea it went away | 20:06 |
| mrmartin | anyway, the LP openid implementation is broken | 20:06 |
| ianw | greghaynes: well, as you found it wasn't really running due to no docker | 20:06 |
| mrmartin | they are not handling properly a handler expiration, so it never expires | 20:07 |
| *** julim has quit IRC | 20:07 | |
| fungi | oh, neat | 20:07 |
| ianw | greghaynes: i mean in upstream. i just feel like the functional tests are doing more than enough, and this is really a separate thing | 20:07 |
| *** jsavak has quit IRC | 20:07 | |
| mrmartin | I realized that during the askbot openstackid.org integration, openstackid implementation is much better from this aspect. (openid assoc handler) | 20:07 |
| *** jsavak has joined #openstack-infra | 20:08 | |
| *** julim has joined #openstack-infra | 20:08 | |
| *** jcoufal_ has quit IRC | 20:08 | |
| greghaynes | ianw: oh, making it separate wont actually help failure rate (itll just fail in the other test at the same rate), it would only help run time at the expense of using a node, and the idea is that dib tests are really fast to add on since cache warmup is the slow part | 20:08 |
| fungi | yeah, it has crossed my mind that moving the wiki to openstackid sooner than lpanned might help matters, but i also don't want us to rush that migration since it's possible to end up in a worse state long-term if we don't figure out a way to try to associate accounts | 20:09 |
| *** daemontool_ has quit IRC | 20:09 | |
| greghaynes | ianw: https://review.openstack.org/#/c/177002/12 if you havent seen it | 20:09 |
| *** esikachev has joined #openstack-infra | 20:09 | |
| *** david-lyle has joined #openstack-infra | 20:09 | |
| *** maishsk has quit IRC | 20:09 | |
| greghaynes | ianw: conflicts after your test runner reworking, but that is what is needed for docker to work | 20:09 |
| mrmartin | yeah, account migration is a problem there. | 20:09 |
| ianw | greghaynes: yep, that's what i'm referring to | 20:09 |
| *** julim has quit IRC | 20:10 | |
| ianw | it really looks different to the other func tests to me | 20:10 |
| greghaynes | ianw: thats fine, theres nothing wrong with running a series of tests in serial | 20:10 |
| mrmartin | I can check how mediawiki works with openstackid.org, in an ideal world, it must work properly. | 20:10 |
| greghaynes | ianw: whether or not to do that isnt a matter of organization, its a matter of performance | 20:10 |
| *** jamesmcarthur has quit IRC | 20:10 | |
| hashar | mrmartin: I am not sure how well maintained is the mw OpenId extension though | 20:11 |
| ianw | greghaynes: alright, well either way i was going to rebase those changes on the newer test runner -- want me to do that? | 20:11 |
| greghaynes | ianw: if you could thatd be awesoome | 20:11 |
| *** gokrokve has joined #openstack-infra | 20:12 | |
| *** austin81 has quit IRC | 20:12 | |
| ianw | greghaynes: ok, that's pretty much bubbled to the top of my todo list (which is nice, because it means fires i know about are out, for now :) | 20:12 |
| *** jaosorior has quit IRC | 20:12 | |
| ianw | my main bubble is getting centos/f23-minimal functional test, but that's a sub-bubble :) | 20:13 |
| greghaynes | ianw: :) | 20:13 |
| *** abregman has quit IRC | 20:13 | |
| *** piet has quit IRC | 20:13 | |
| *** abregman has joined #openstack-infra | 20:13 | |
| *** flepied1 has joined #openstack-infra | 20:14 | |
| AJaeger | we have removed pandoc and dvipng from bindep (project-config) but not yet from system-config setup, could I get some review for https://review.openstack.org/#/c/286242/ and https://review.openstack.org/#/c/284371/ , please? | 20:14 |
| *** sdake has joined #openstack-infra | 20:16 | |
| *** flepied has quit IRC | 20:16 | |
| *** ihrachys has quit IRC | 20:18 | |
| *** maishsk has joined #openstack-infra | 20:19 | |
| openstackgerrit | Stephen Gordon proposed openstack-infra/project-config: Skip magnum functional test jobs on docs changes https://review.openstack.org/277892 | 20:19 |
| mrmartin | anyway, what's the next step with wiki? | 20:19 |
| *** |-paul-| has joined #openstack-infra | 20:20 | |
| *** yamahata has quit IRC | 20:21 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs https://review.openstack.org/286136 | 20:21 |
| *** xyang1 has quit IRC | 20:21 | |
| pabelanger | mrmartin: only 1 spammer today, I haven't done anything today to stop them | 20:26 |
| pabelanger | so, the changes we made yesterday have stopped them for the moment | 20:26 |
| mrmartin | sometimes they are going away, but it doesn't mean we solved the issue | 20:26 |
| fungi | though clearly leaving new account creation disabled is not a long-term fix | 20:26 |
| pabelanger | right | 20:26 |
| mrmartin | we should add a real captcha for the new account creation | 20:26 |
| pabelanger | getting things undercontrol was step 1 | 20:26 |
| fungi | and yeah, there is no long-term fix to spam really, just ever escalating barriers to usability until you find the sweet-spot between what spammers are willing to endure to use your system and what legitimate users are willing to endure. luckily the former is usually lower than the latter | 20:27 |
| openstackgerrit | Tim Buckley proposed openstack-infra/subunit2sql: Add API methods for getting tests by prefix https://review.openstack.org/283334 | 20:27 |
| clarkb | ya wasn't in meeting channel (haven't rejoined since my client died in ft collins) but had to see tax preparer | 20:28 |
| clarkb | will join now so that I am there though | 20:28 |
| clarkb | that is all done now so yay | 20:28 |
| mrmartin | I can check this captcha at new LP account issue in a dev environment. | 20:28 |
| fungi | captcha on new account creation probably solves this _if_ the spammers aren't using a captcha solver service to blow through them | 20:28 |
| clarkb | fungi: did you have an opinion on whether or not we should be booting the osic mirror and configuring a max server of one in nodepool before the ssl situation is better? | 20:28 |
| mrmartin | who provides a well-working captcha service? | 20:28 |
| pabelanger | I still don't think we have the properly tooling inplace to stop spammers. Things like spamblacklist title and others appear to be the current methods. Once setup and installed, then wiki admins should be able to deal with most of it out side of -infra | 20:29 |
| pabelanger | something we can work on moving forward | 20:29 |
| fungi | clarkb: i'm inclined to wait until they can add a dns entry--that seems like it should be quick and cheap | 20:29 |
| nibalizer | pabelanger: should we look at upgrading the blacklist module? | 20:30 |
| *** sdake has quit IRC | 20:31 | |
| *** _nadya_ has quit IRC | 20:31 | |
| fungi | nibalizer: i think upgrading things is going to be hit-or-miss until we can move wiki.o.o to a newer distro release so that we can install a newer mediawiki to support newer plugins | 20:31 |
| pabelanger | nibalizer: not sure, I need to read up on it more. I think the current one only deals with http links. Where most of our current spam is not using http links, just phone numbers and such | 20:31 |
| fungi | also, spam mitigation is one of those areas where you often end up in a continual upgrade cycle to get the new filtering/blocking features necessary to thwart spammers who have figured out ways around your previous solutions | 20:32 |
| clarkb | ok I can send mail to them | 20:32 |
| mrmartin | mod-security with a phone matching rule? | 20:32 |
| clarkb | see whether or not that is a thing | 20:32 |
| fungi | thanks clarkb! | 20:33 |
| *** hashar has quit IRC | 20:33 | |
| EmilienM | pabelanger: if ubuntu-trusty is the new node to use for ubuntu, what is the centos one? | 20:33 |
| fungi | mrmartin: well, general pattern blocks like "don't allow anything that looks like a phone number" would, for example, prevent us from updating https://wiki.openstack.org/wiki/Infrastructure/Conferencing because we document a legitimate phone service we use/run | 20:34 |
| *** austin81 has joined #openstack-infra | 20:34 | |
| mrmartin | nice | 20:34 |
| fungi | EmilienM: devstack-centos7 (we haven't renamed it) | 20:34 |
| EmilienM | fungi: ok. So I just need to use ubuntu-trusty instead of devstack-trusty, right? | 20:34 |
| *** tongli has quit IRC | 20:35 | |
| fungi | EmilienM: really either would probably work the same. basically we never had a bare-centos7 so there was no need to migrate jobs to a new name for those. we git rid of bare-centos6 when we dropped python 2.6 testing so never ended up using the centos-6 images i was working on | 20:35 |
| pabelanger | EmilienM: ubuntu-trusty and devstack-centos7 are the current dibs | 20:35 |
| *** jsavak has quit IRC | 20:35 | |
| pabelanger | infact, devstack-centos7 is not the final boss. I believe ianw is working on centos-7 dibs | 20:36 |
| fungi | er, got rid (finger memory always makes me want to type "git") | 20:36 |
| pabelanger | from centos-minimal | 20:36 |
| EmilienM | fungi: ok so I don't need to patch our CI to use ubuntu-trusty I guess | 20:37 |
| fungi | not sure what you mean there | 20:37 |
| EmilienM | our puppet-beaker & puppet-integration jobs use devstack-trusty nodes | 20:37 |
| pabelanger | ubuntu-trusty is what we are using for bindep | 20:37 |
| fungi | anyway, there will come a point when we move devstack-trusty jobs to ubuntu-trusty and devstack-centos7 jobs to centos-7 | 20:37 |
| fungi | EmilienM: yeah, no need to switch those now unless you simply want to | 20:38 |
| *** e0ne has joined #openstack-infra | 20:38 | |
| EmilienM | fungi: yeah that's why I'm asking so I can help you with the puppet jobs | 20:38 |
| *** Guest95751 is now known as Vivek | 20:38 | |
| EmilienM | but if no help is needed on that, I'll let the transition happen | 20:38 |
| *** Vivek has quit IRC | 20:38 | |
| *** Vivek has joined #openstack-infra | 20:38 | |
| *** kushal has quit IRC | 20:38 | |
| pabelanger | fungi: 286785 adds bindep for ansible jobs | 20:39 |
| fungi | the move off devstack-.* nodes will be much simpler to orchestrate en mass because the underlying configuration is basically the same. it's the move off bare-.* nodes which is taking a lot of extra care to get right without being too disruptive | 20:39 |
| pabelanger | experimental functional testing for ubuntu-trusty | 20:39 |
| *** yamahata has joined #openstack-infra | 20:39 | |
| EmilienM | fungi: once thing I noticed is that it takes a lot of time (lately at least) to get a devstack-trusty node comparing to a ubuntu-trusty node | 20:39 |
| fungi | EmilienM: that may simply be a demand issue, because not a lot of jobs are using ubuntu-trusty yet. i wouldn't count on it remaining that way for long | 20:40 |
| *** abregman is now known as abregman|nb | 20:40 | |
| EmilienM | fungi: ok good to know, so I won't patch that thing. i'll let you manage that en mass - thanks for this work btw | 20:41 |
| *** exploreshaifali has quit IRC | 20:42 | |
| *** maishsk_ has joined #openstack-infra | 20:42 | |
| *** maishsk has quit IRC | 20:43 | |
| *** sdake has joined #openstack-infra | 20:43 | |
| *** maishsk_ is now known as maishsk | 20:43 | |
| *** pcaruana has joined #openstack-infra | 20:44 | |
| annegentl_ | fungi: hey one area I wanted to understand, is infra able to consume an OSIC pop-up cloud? As in, would that work as an extra resource around release times? | 20:45 |
| trash | SpamapS: thanks | 20:45 |
| *** jamielennox|away is now known as jamielennox | 20:45 | |
| clarkb | annegentl_: I am currently trying to add in osic proper | 20:46 |
| clarkb | annegentl_: so I think the answer is yes, but I odn't know what an osic pop-up cloud is | 20:46 |
| *** e0ne has quit IRC | 20:47 | |
| annegentl_ | clarkb: oh just my term for "hey have some OSIC" | 20:47 |
| nibalizer | geekinutah: did ou get your questions answered? | 20:47 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Skip dsvm jobs on release note only tempest changes https://review.openstack.org/286831 | 20:47 |
| *** yamahata has quit IRC | 20:47 | |
| clarkb | annegentl_: in theory all you would need to do is bump our quota and reflect that in nodepool | 20:48 |
| *** jsavak has joined #openstack-infra | 20:48 | |
| annegentl_ | clarkb: pop-up, meaning, doesn't have to be there except at peak times | 20:48 |
| annegentl_ | clarkb: okay, good to know. | 20:48 |
| clarkb | (once I get it running, currently trying to sort out ssl) | 20:48 |
| *** bpokorny has joined #openstack-infra | 20:48 | |
| *** hashar has joined #openstack-infra | 20:49 | |
| *** gyee has joined #openstack-infra | 20:50 | |
| *** jpr has quit IRC | 20:51 | |
| openstackgerrit | Merged openstack-infra/shade: Fix heat create_stack and delete_stack https://review.openstack.org/276045 | 20:54 |
| *** gildub has joined #openstack-infra | 20:54 | |
| *** piet has joined #openstack-infra | 20:55 | |
| *** e0ne has joined #openstack-infra | 20:55 | |
| *** julim has joined #openstack-infra | 20:55 | |
| *** esikachev has quit IRC | 20:56 | |
| *** sdake has quit IRC | 20:56 | |
| *** amrith is now known as _amrith_ | 20:56 | |
| *** hichihara has quit IRC | 20:57 | |
| *** rguillebert has quit IRC | 20:57 | |
| clarkb | fungi: I did just confirm that /etc/hosts using the cert they present to veirfy the self signed cert works | 20:58 |
| clarkb | fungi: so I don't think they made a bad self signed cert at the very least | 20:58 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 20:58 |
| *** maishsk has quit IRC | 20:58 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 20:58 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:58 | |
| *** e0ne has quit IRC | 20:59 | |
| geekinutah | nibalizer: yeah, I'm sure I'll have more later :-) | 20:59 |
| *** sdake has joined #openstack-infra | 20:59 | |
| geekinutah | I'm crawling through nodepool and friends to try and answer some preemptively | 20:59 |
| anteaya | annegentl_: well given the amount of time it takes to figure out a cloud, it is optimal if we have the cloud purring along prior to the rush | 21:00 |
| anteaya | else we spend time debugging that cloud under load | 21:00 |
| anteaya | which does happen | 21:00 |
| nibalizer | what organizaion are you with? | 21:01 |
| fungi | annegentl_: yeah, basically osic is shaping up nicely from what clarkb has tested, minus a dns update request and the fact that they freaked out when we wanted more than 100 virtual machines and ip addresses for them | 21:01 |
| annegentl_ | IP addresses, the new gold rush | 21:01 |
| annegentl_ | anteaya: good point | 21:02 |
| *** maishsk has joined #openstack-infra | 21:02 | |
| clarkb | annegentl_: thankfully we have a new metal to make ip addresses out of that is dirt cheap | 21:02 |
| fungi | also we've repeatedly said we gladly use ipv6 instead of ipv4 for this, to which they've been completely silent (i'm willing to bet they don't want to figure out ipv6 in neutron) | 21:02 |
| anteaya | annegentl_: now we won't say no to new resources at any time | 21:02 |
| fungi | hah, yes, what clarkb just said | 21:02 |
| anteaya | but the notion that we have a bunch we bring online week prior to milestone is just going to mean a different kind of broken | 21:03 |
| *** mrmartin has quit IRC | 21:03 | |
| annegentl_ | anteaya: fungi: yeah I think it's interesting to not only have flexible cloud resources but entirely flexible whole clouds, and to do that, the flexible whole cloud parceling, they had to do network engineering. | 21:03 |
| clarkb | fungi: ok, email sent to osic about the dns thing I cc'd you | 21:03 |
| fungi | right, average time to production for consuming donated cloud environments has been improving, but is still probably a 1-month minimum | 21:03 |
| annegentl_ | parcelling? I dunno | 21:03 |
| anteaya | annegentl_: yup | 21:04 |
| anteaya | it is a utopia | 21:04 |
| anteaya | and I like it | 21:04 |
| fungi | thanks clarkb! | 21:04 |
| anteaya | but we have a whole bunch of work to get there | 21:04 |
| *** jsavak has quit IRC | 21:04 | |
| *** weshay has joined #openstack-infra | 21:04 | |
| fungi | i feel a lot better if they solve that than keeping a workaround in /etc/hosts on nodepool.o.o forever | 21:04 |
| clarkb | fungi: indeed | 21:04 |
| * clarkb is currently etc hosting on laptop and using "verified" connections | 21:05 | |
| anteaya | feel comfy and safe? | 21:05 |
| clarkb | I mean | 21:06 |
| *** david-lyle has quit IRC | 21:08 | |
| geekinutah | nibalizer: I work for Mirantis, but this research is unrelated to them | 21:08 |
| nibalizer | geekinutah: ok | 21:09 |
| nibalizer | let us know what information you need | 21:09 |
| nibalizer | we'll be happy to work with you | 21:09 |
| anteaya | geekinutah: and whenever you can reveal a group name we can reference among ourselves that would help us | 21:11 |
| geekinutah | thanks, I will for sure | 21:11 |
| anteaya | since right now it is, the thing geekinutah is asking about | 21:11 |
| anteaya | which is a bit long for a title | 21:11 |
| anteaya | geekinutah: thanks for helping :) | 21:11 |
| fungi | i don't mind not knowing who is working where, but being able to tell who is working together is useful | 21:11 |
| geekinutah | well, include AndyU in that long title | 21:11 |
| anteaya | true that | 21:11 |
| anteaya | geekinutah: ah wondered if this was the same topic | 21:12 |
| anteaya | thanks | 21:12 |
| geekinutah | we will try and be public soon, just working through internal approvals yada yada | 21:12 |
| fungi | yep, knowing this is related to something maybe already talked about helps avoid a lot of repetition when we don't know how briefed someone is on something | 21:13 |
| geekinutah | completely understood, will fix soon | 21:13 |
| *** korzen has joined #openstack-infra | 21:13 | |
| clarkb | fungi: see dns response? if I still worked at intel I coukd fix this | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| fungi | all of the jenkins masters except jenkins01 are caught up for configs now. 01 is hitting some proxy errors and likely could benefit from a shutdown/restart so i'll do that | 21:14 |
| anteaya | fungi: ack, thanks | 21:14 |
| anteaya | geekinutah: thank you | 21:15 |
| clarkb | I had root on this ns | 21:15 |
| fungi | clarkb: um, that's not the domain in question though? | 21:15 |
| clarkb | fungi: no sounds like they arent doing anything for current cn | 21:15 |
| clarkb | and are going to use that other domain | 21:15 |
| clarkb | I suppose we can ask for current cn | 21:16 |
| fungi | clarkb: so... they deployed with a cert for which they never intended to have working dns, and want to replace the cert and use a different dns name in a domain they don't control? | 21:16 |
| clarkb | ya | 21:16 |
| fungi | that just seems odd | 21:16 |
| fungi | hogepodge: ^ your osic friends and their choices | 21:17 |
| *** tphummel has quit IRC | 21:17 | |
| lifeless | AJaeger: I need to catch up on the status of the various discussions | 21:17 |
| *** david-lyle has joined #openstack-infra | 21:17 | |
| lifeless | AJaeger: but basically, just change tox, right? | 21:17 |
| clarkb | fungi: I will respond asking for the rax controlled domain to be used in interim if possible and see if I can't get the intel side expedited | 21:18 |
| clarkb | fungi: cloud1.osic.org does resolve but I guess they haven't been able to get a new cert for it yet | 21:18 |
| fungi | clarkb: cool. i guess there's some possible benefit in the near term to adding it to nodepool to take it for a spin even if we're going to need to change the endpoint name before it's in full swing | 21:20 |
| *** kzaitsev_mb has joined #openstack-infra | 21:20 | |
| clarkb | fungi: yup, the biggest risk is whether or not we feel to exposed without verifying any part of the connections | 21:20 |
| fungi | though if they're planning to not use a self-signed cert, maybe we don't need the extra hassle of adding the temporary trust for it | 21:21 |
| fungi | would have just been courteous of them to let us know up front that this wasn't a completely baked deployment | 21:21 |
| fungi | set better expectations | 21:21 |
| clarkb | I think they did mention it was a work in progress but ya I didn't realiez that they didn't even have dns working | 21:22 |
| *** abregman_ has joined #openstack-infra | 21:22 | |
| *** tphummel has joined #openstack-infra | 21:22 | |
| fungi | and didn't deploy it with the cert they were planning to use longer-term | 21:22 |
| fungi | jenkins01 does indeed appear to be suffering from the typical thread leak issue | 21:23 |
| *** abregman|nb has quit IRC | 21:24 | |
| *** annegentl_ has quit IRC | 21:25 | |
| anteaya | :( | 21:28 |
| *** annegentl_ has joined #openstack-infra | 21:28 | |
| clarkb | it seems to be a function of load/jobs run and not time | 21:28 |
| clarkb | I haven't heard from upstream since my last round of updating the bug with info on how we run into it | 21:28 |
| fungi | clarkb: i have a feeling our older jenkins masters (particularly 01 and 02 but also to a lesser extent 03 and 04) perform worse than more recent ones and may be hitting this with a proportionally greater frequency as a result | 21:29 |
| clarkb | ya 01 and 02 do seem to have the most trouble with large jjb updates | 21:30 |
| fungi | jjb run-times across each of them are dramatically graduated | 21:30 |
| *** sripriya__ has joined #openstack-infra | 21:31 | |
| *** dims_ has joined #openstack-infra | 21:31 | |
| *** sridhar_ram has joined #openstack-infra | 21:32 | |
| *** korzen has quit IRC | 21:32 | |
| *** yamahata has joined #openstack-infra | 21:32 | |
| clarkb | fungi: so here is what Ithink I will do any you can comment if it is terrible idea or not. Update puppetmaster's /etc/hosts to have entry for current osic cert CN. Launch mirror with the self signed cert verifying itself. This should get us "verified" connections when bringing up a mirror host and make sure all the infrastructure pieces are there, but hold off on adding to nodepool until we can do | 21:32 |
| clarkb | it properly | 21:33 |
| *** dims has quit IRC | 21:33 | |
| *** sridhar_ram1 has quit IRC | 21:33 | |
| fungi | no objection from me. we've seen no indication that they plan to tear down and rebuild the environment so should be fine to employ minimal workarounds like that to parallelize the setup work on our part and on their part | 21:33 |
| ianw | clarkb / jhesketh : so i guess images didn't get up to rax -- https://jenkins06.openstack.org/job/gate-tempest-dsvm-platform-fedora23-nv/18/console (doesn't have the timestamp at the top http://nodepool.openstack.org/dib.fedora-23.log <- build was ok) | 21:34 |
| *** sripriya_ has quit IRC | 21:34 | |
| clarkb | ianw: yes we are still failing to upload reliably there | 21:34 |
| fungi | as you pointed out earlier, the afs client mirrors are not holding any secret/privileged data we're transferring over that connection anyway | 21:34 |
| clarkb | ianw: we now attempt to reup the token but swiftclient complains about missing tenant id | 21:35 |
| *** [1]Thelo has joined #openstack-infra | 21:37 | |
| *** jpr has joined #openstack-infra | 21:39 | |
| *** Thelo has quit IRC | 21:40 | |
| *** [1]Thelo is now known as Thelo | 21:40 | |
| anteaya | clarkb: how is now for the neutron default security groups discussion? | 21:40 |
| clarkb | the error you get when doing a non insecure conection against unverifiable cloud is fun. Could not determine suitable url for plugin | 21:41 |
| clarkb | anteaya: works great | 21:41 |
| anteaya | I'm asking you first then will see how the timing is for neutron folks | 21:41 |
| anteaya | awesome let's see who I can round up from neutron | 21:41 |
| sc68cal | hello | 21:41 |
| *** derekh has joined #openstack-infra | 21:41 | |
| anteaya | sc68cal: awesome | 21:42 |
| anteaya | thank you | 21:42 |
| ianw | clarkb: in shade? | 21:42 |
| anteaya | let's just see if kevinbenton and dougwig are also available | 21:43 |
| kevinbenton | i am! | 21:43 |
| sc68cal | sounds good | 21:43 |
| anteaya | wonderful | 21:43 |
| anteaya | so to set the stage | 21:43 |
| anteaya | infra created an infra cloud | 21:43 |
| *** Sukhdev has quit IRC | 21:43 | |
| anteaya | and has some user stories from that experience | 21:43 |
| anteaya | one of the involves neutron default security groups | 21:43 |
| anteaya | and clarkb has more details | 21:43 |
| *** baoli has joined #openstack-infra | 21:43 | |
| *** rguillebert has joined #openstack-infra | 21:44 | |
| clarkb | ianw: it is mostly os-client-config and swiftclient I Think | 21:44 |
| clarkb | right so security groups | 21:44 |
| anteaya | I don't believe rcarrillocruz is around at this time unfortunately, though he also has some thoughts | 21:44 |
| clarkb | there are two general issues. The first is that the defaults don't let you do anything, and the other is that the defaults include broken rules by default (inter group rules) | 21:44 |
| *** baoli_ has joined #openstack-infra | 21:44 | |
| clarkb | so for every project we create I have to delete 2 rules and add 2 new ones. Always | 21:45 |
| kevinbenton | clarkb: what do you mean they are broken rules? | 21:45 |
| openstackgerrit | Ryan Beisner proposed openstack-infra/project-config: Enable verified label for charms https://review.openstack.org/286853 | 21:45 |
| sc68cal | the default rules IIRC are just to allow outbound | 21:45 |
| ianw | clarkb: this is what you mostly referring to? https://review.openstack.org/#/c/255623/ | 21:46 |
| clarkb | kevinbenton: intergroup rules eg default group members can talk to default are broken | 21:46 |
| dougwig | clarkb: fwiw, i agree and hate the defaults. | 21:46 |
| clarkb | kevinbenton: they put strain on the db or something and results in clarkb getting 2am phone calls from clouds | 21:46 |
| anteaya | yay dougwig is here too | 21:46 |
| clarkb | kevinbenton: so now as a rule I always delete them | 21:46 |
| dougwig | clarkb: #1 cause of user complaints. | 21:46 |
| clarkb | because I hate 2am phone calls | 21:47 |
| kevinbenton | clarkb: i don't understand. is it that there is a performance issue or is that they don't actually do what they are supposed to? | 21:47 |
| *** maishsk has quit IRC | 21:47 | |
| mtreinish | infra-root: I'm gonna keep bugging about: https://review.openstack.org/286733 and https://review.openstack.org/281383 so we can get to the bottom of why the subunit worker is getting stuck | 21:47 |
| mtreinish | the result collection has been down for ~2 weeks and I'd really like to sort this soon | 21:47 |
| nibalizer | mtreinish: ok | 21:47 |
| mtreinish | nibalizer: thanks | 21:48 |
| *** watersoul has quit IRC | 21:48 | |
| clarkb | kevinbenton: they do what they are supposed to, but you can apparently kill the clouds using them | 21:48 |
| clarkb | kevinbenton: which is why we now have quotas around rules? | 21:48 |
| *** baoli has quit IRC | 21:48 | |
| clarkb | kevinbenton: so we have attempted to work around the issue but haven't actually fixed it so 2am phone calls still a potential problem | 21:48 |
| *** watersoul has joined #openstack-infra | 21:49 | |
| kevinbenton | clarkb: hmmm. you may have stumbled onto a bug. the default rules should not result in killing the cloud | 21:49 |
| kevinbenton | clarkb: the quota is just because tons of rules will eventually choke iptables | 21:49 |
| *** |-paul-| has quit IRC | 21:49 | |
| kevinbenton | clarkb: but a basic query for members should not wipe out the server | 21:50 |
| sc68cal | ++ | 21:50 |
| *** sdague has quit IRC | 21:50 | |
| kevinbenton | clarkb: how many members are there of the group when it falls over? | 21:50 |
| clarkb | kevinbenton: it had to do wit hthe number of members in the group | 21:50 |
| kevinbenton | clarkb: :) | 21:50 |
| clarkb | kevinbenton: when I got the 2am call we had ~600 members | 21:50 |
| kevinbenton | clarkb: ok. i will see if i can repro | 21:50 |
| fungi | kevinbenton: it's possible this has been fixed in neutron. there was a time when having several hundred instances resulted in a combinatorial matrix of rules for every instance pairing in iptables | 21:50 |
| kevinbenton | fungi: yes, that should be long gone | 21:50 |
| clarkb | our/my response at the time was to remove the useless rules completely | 21:50 |
| fungi | kevinbenton: and then adding and deleting instances from the group rapidly would cause enough churn to bring the accounting to its knees | 21:50 |
| kevinbenton | fungi: ipset fixed that on the iptables side | 21:50 |
| kevinbenton | fungi: and sane queries fixed the server side (I thought) | 21:51 |
| kevinbenton | what version of neutron is this? | 21:51 |
| clarkb | it was whatever hpcloud was running | 21:51 |
| fungi | kevinbenton: kilo | 21:51 |
| fungi | oh, hpcloud | 21:51 |
| fungi | not what we deployed | 21:51 |
| clarkb | right 2am phone call was hpcloud | 21:51 |
| *** abregman has joined #openstack-infra | 21:52 | |
| fungi | i mean, from our perspective those are irrelevant default rules anyway, because we want default allow everything to and from everywhere, so additional rules allowing more specific sources/destinations are not doing anything | 21:52 |
| *** jpr has quit IRC | 21:52 | |
| kevinbenton | fungi: ok | 21:52 |
| sc68cal | The defaults already allow half that - the outbound piece | 21:52 |
| sc68cal | so really it should just be two rules you add, to allow inbound on v4 and v6 ? | 21:53 |
| kevinbenton | fungi: so unfortunately the thread in the past about operator-configurable defaults did not end well | 21:53 |
| kevinbenton | fungi: because it results in different experiences per cloud | 21:53 |
| fungi | we assume we are representative of at least a significant slice of the userbase who prefer to do traffic filtering on individual instances interfaces and drive that via configuration management, rather than centrally in a fake network firewall in the cloud provider's network | 21:53 |
| kevinbenton | fungi: fake firewall? | 21:54 |
| *** fawadkhaliq has quit IRC | 21:54 | |
| sc68cal | fungi: I can think of a significant openstack deployment where that is not allowed, by policy, for compliance reasons | 21:54 |
| kevinbenton | fungi: what would you do it with on the instance? | 21:54 |
| fungi | but i get that there are likely others who would prefer a default block everything ruleset in the security rules in neutron/nova | 21:54 |
| kevinbenton | (nevermind, not fighting about whether or not iptables is fake) :) | 21:54 |
| kevinbenton | may i suggest just setting port_security_enabled to False on the network these VMs are attaching to? | 21:54 |
| clarkb | right so the other problem is literally on every cloud I have to update the rules | 21:55 |
| clarkb | this tells me that our defaults are broken | 21:55 |
| fungi | kevinbenton: fake network firewall (e.g. not dedicated hardware) vs host firewall on the instance | 21:55 |
| fungi | fake was probably not the word i wanted there | 21:55 |
| kevinbenton | it sounds like you don't want any port security | 21:55 |
| fungi | virtual network firewall perhaps | 21:55 |
| *** rhallisey has quit IRC | 21:55 | |
| *** austin81 has left #openstack-infra | 21:55 | |
| clarkb | ideally ( and this may not be possible ) our defaults should capture a reasonable subset of what users need so that they don't have to customize everything | 21:55 |
| fungi | kevinbenton: we use iptables on the hosts for port security | 21:55 |
| fungi | on the instances i mean | 21:55 |
| *** abregman_ has quit IRC | 21:55 | |
| fungi | within each instance's operating system | 21:56 |
| kevinbenton | ack, so are the hosts 100% trusted in these cases? | 21:56 |
| sc68cal | clarkb: if the security group API extension is not useful, it is just an extension and you can disable it in your neutron deployment. I don't know anything about infra cloud so I don't know if you can make that decision | 21:56 |
| clarkb | sc68cal: I cannot disbale it in most of the clouds I use | 21:56 |
| clarkb | sc68cal: I am not talking infra cloud | 21:56 |
| *** annegentl_ has quit IRC | 21:56 | |
| clarkb | sc68cal: I am talking generally this is a problem on every single cloud I have used | 21:56 |
| clarkb | (except for rackspace | 21:56 |
| sc68cal | ok, then what about kevinbenton 's suggestion about disabling port security | 21:57 |
| clarkb | and it occurs to me that if users have to "fix" this every time they use a new cloud we probably aren't doing what the users need | 21:57 |
| kevinbenton | clarkb: we match the defaults of AWS IIRC | 21:57 |
| *** annegentl_ has joined #openstack-infra | 21:57 | |
| anteaya | oh sorry perhaps I set the inital story incorrectly | 21:58 |
| kevinbenton | clarkb: (i think that's the history of where they came from) | 21:58 |
| anteaya | I apologize if I created confustion | 21:58 |
| anteaya | or confusion | 21:58 |
| *** rhallisey has joined #openstack-infra | 21:59 | |
| dougwig | kevinbenton: so we value uniform awfulness over operator configurable maybe not awfulness? | 21:59 |
| * dougwig hides. | 21:59 | |
| anteaya | okay so perhaps doing what AWS has passed the best before date? | 21:59 |
| *** abregman is now known as abregman|nb | 21:59 | |
| dougwig | nova-net by default is open, right? | 21:59 |
| sc68cal | no it is not | 21:59 |
| clarkb | I also field a lot of questions on the Internet about "I am using neutron and cannot ping or ssh to my instance" | 21:59 |
| kevinbenton | dougwig: so we can change it, but I'm surprised to hear this coming from you since it creates insecure VMs by default :) | 21:59 |
| clarkb | most of the time the problem there is "edit your security groups" | 22:00 |
| kevinbenton | dougwig: i recall a signnificant bikeshed around DNSMASQ leaking queries... | 22:00 |
| kevinbenton | so how was this dealt with with nova-net? | 22:00 |
| clarkb | kevinbenton: I am not sure that more open security groups implies insecure VMs | 22:00 |
| dougwig | kevinbenton: i am never opposed to handing operators a gun and a lot of ammo. it becomes their choice which foot they shoot. | 22:00 |
| sc68cal | clarkb: these issues are basically the difference between people who used only bare metal and expect certain behaviors versus cloud | 22:00 |
| *** pcaruana has quit IRC | 22:01 | |
| sc68cal | I've had this conversation multiple times - the issue is the security group default behavior is usually part of a security policy dictated by other parts of the organiation, and also matches a "secure by default" behavior | 22:01 |
| kevinbenton | dougwig: it's not the operators, it's the users | 22:01 |
| kevinbenton | dougwig: i use one cloud, i get protected VMs by default | 22:01 |
| kevinbenton | dougwig: i use another, i get exposed VMs by default | 22:01 |
| sc68cal | ^ this too | 22:01 |
| clarkb | kevinbenton: uh more like "possibly protected but hard to confirm at all times" | 22:01 |
| clarkb | which is another separate issue that seems to get fixed every 6-12 months | 22:02 |
| kevinbenton | clarkb: i'm not following | 22:02 |
| clarkb | kevinbenton: there have been several vulnerabilities where security groups don't actually apply | 22:02 |
| clarkb | kevinbenton: which is a big reason for using instance local iptables | 22:02 |
| *** dims_ has quit IRC | 22:02 | |
| sc68cal | that's FUD, and trolling | 22:02 |
| clarkb | it is not | 22:02 |
| dougwig | kevinbenton: so, at AWS it's explicit, and at DO it's open by default. and never have I, as a user, been confused. | 22:02 |
| anteaya | let's stick with details | 22:03 |
| clarkb | as a cloud user you have little insight into whether or not a seucrity group rule is actually working | 22:03 |
| clarkb | you can test a point in time | 22:03 |
| clarkb | however as a cloud user running your own local firewall you can inspect state | 22:03 |
| sc68cal | clarkb: if a security group rule is applied to a group, and a VM is part of that group, it will be applied, Hence we return the apropriate HTTP code to the API request | 22:03 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: update django_openstack_auth to use publish-to-pypi jobs https://review.openstack.org/286747 | 22:04 |
| *** fawadkhaliq has joined #openstack-infra | 22:04 | |
| *** Sukhdev has joined #openstack-infra | 22:04 | |
| kevinbenton | ok. as i understand it, we don't trust neutron to do what the API says | 22:04 |
| kevinbenton | so shut off port security and call it a day | 22:04 |
| kevinbenton | problem solved | 22:04 |
| mtreinish | ugh, connreset on centos apply job: http://logs.openstack.org/33/286733/1/gate/gate-openstackci-beaker-centos7-dsvm/0e5f1e0/console.html | 22:04 |
| clarkb | kevinbenton: how do you turn it off as a user? | 22:05 |
| mtreinish | that just added like another 2 hrs until that patch can merge now.. :( | 22:05 |
| kevinbenton | neutron net-create mynet --port-security-enabled=False | 22:05 |
| clarkb | kevinbenton: and if I cannot create my own networks? | 22:05 |
| sc68cal | or do a port-update on a port to set port security to false | 22:05 |
| kevinbenton | clarkb: well if you don't own the networks then you can't do this | 22:05 |
| sc68cal | or create, iirc - kevinbenton can correct me if wrong | 22:05 |
| kevinbenton | this won't work for networks you don't own | 22:06 |
| clarkb | right in several of our clouds we get the the networks we get | 22:06 |
| kevinbenton | because it turns off all kinds of filtering | 22:06 |
| dougwig | that nonsense being the burden of end users is... nonsense. bleh. | 22:06 |
| openstackgerrit | Merged openstack-infra/system-config: Add a job filter for old side subunit files https://review.openstack.org/281383 | 22:06 |
| anteaya | dougwig: can you expand? | 22:06 |
| kevinbenton | clarkb: ok. so if you don't control the cloud, how do you propose the default rules be changed? | 22:06 |
| kevinbenton | clarkb: are you suggesting for Neutron to change it's default rules for everyone? | 22:07 |
| clarkb | kevinbenton: yes | 22:07 |
| clarkb | at the very least I think it is reasonable to pass through protocols like ssh | 22:07 |
| fungi | part of the challenge, i think, is that because default security groups are configurable by the provider, every provider decides to configure them differently, so step 1 in hooking up to a new cloud provider is to figure out what random things they've decided to block on your instances | 22:07 |
| clarkb | because without that you aren't doing much with your instances | 22:07 |
| fungi | for example, no gre in ovj | 22:07 |
| fungi | ovh | 22:07 |
| kevinbenton | fungi: they can't with neutron though | 22:07 |
| clarkb | fungi: with neutron you can't change the defaults that new projects get | 22:08 |
| *** dprince has quit IRC | 22:08 | |
| clarkb | fungi: I think the gre thing is a weird outlier | 22:08 |
| clarkb | (which probably should get solved, it just isn't directly related) | 22:08 |
| sc68cal | allowing inbound SSH to every instance is .... very aggressive | 22:08 |
| fungi | kevinbenton: that seems counter to our experience. i think internap is using neutron and they just recently adjusted their global default security groups at our recommendation | 22:08 |
| sc68cal | I think someone from a security would flip out | 22:08 |
| fungi | because they were allowing all tcp and udp, as if it were the internet | 22:08 |
| kevinbenton | fungi: they changed code then | 22:08 |
| *** thorst_ has quit IRC | 22:09 | |
| fungi | kevinbenton: i wouldn't be surprised, though mgagne probably has more details on that | 22:09 |
| clarkb | sc68cal: there is a balance between conservative rules to protect instances and useable so that users don't have to fight their clouds | 22:09 |
| dougwig | clarkb, fungi: how is the neutron experience different (and worse) than the nova-net default closed rules? | 22:09 |
| clarkb | dougwig: nova-net allowed providers to change the defaults and every nova net cloud I used allowed ssh | 22:10 |
| clarkb | dougwig: neutron does not allow this and I have to edit the rules in every new project | 22:10 |
| kevinbenton | fungi: https://github.com/openstack/neutron/blob/master/neutron/db/securitygroups_db.py#L169-L185 | 22:10 |
| *** dims has joined #openstack-infra | 22:10 | |
| sc68cal | is that good though? different clouds having different defaults? | 22:10 |
| kevinbenton | clarkb: but i thought you didn't control the clouds? In this case would you ask the operator to change their defaults? | 22:10 |
| clarkb | sc68cal: not necessarily, I think the goodness is in having a default that is sane for users | 22:10 |
| fungi | right, i'm not convinced configurable defaults is necessarily good since it creates interoperability rifts | 22:10 |
| sc68cal | clarkb: ok - and I guess I think allowing inbound everything by default is not sane. | 22:11 |
| clarkb | kevinbenton: no I think my goal would be to make a default ruleset that is good for users | 22:11 |
| clarkb | sc68cal: I am not suggesting that | 22:11 |
| *** jamesmcarthur has joined #openstack-infra | 22:11 | |
| clarkb | (I mean that would be even better for us specifically but I can see the arguments against it too) | 22:11 |
| anteaya | let's stay focused on finding some common ground here | 22:11 |
| anteaya | I think it is possible | 22:11 |
| fungi | right, where the previous discussions ended up at was that nobody could agree what ports were reasonable for initial ingress by default, so the decision was to just allow no ingress at all and make everyone change it before they could reach anything | 22:12 |
| kevinbenton | fungi: +1. this is where it ultimately lands whenever we repaint this shed | 22:12 |
| dougwig | what do the operators want and/or do? do they just edit neutron to specify defaults anyway, regardless of our ideology on this? | 22:12 |
| *** tiswanso has quit IRC | 22:12 | |
| fungi | basically there's no one config that's good for everyone, so install a config that's consistently bad for everyone | 22:12 |
| anteaya | I don't know as we have asked the operators community | 22:12 |
| sc68cal | Is there a reason those rules need to be in the default security group? Why not create a security group with the rules that you need, and create them - heat template/provisioning system/ etc | 22:13 |
| sc68cal | whatever you use | 22:13 |
| anteaya | I just thought some benefit might come out of infra sharing their use cases | 22:13 |
| *** abregman|nb has quit IRC | 22:13 | |
| clarkb | sc68cal: because I think we should provide useable defaults to the majority of our uesrs without fiddling extra knobs | 22:13 |
| kevinbenton | The issue I see now is that if we change this it's a big behavior change on upgrade | 22:13 |
| *** dkranz has joined #openstack-infra | 22:13 | |
| sc68cal | clarkb: believe me, I am a fan of that idea, but the usable default we've got right now is the best compromise I think we have, allowing outbound and blocking inbound | 22:13 |
| clarkb | current situation is as fungi says consistently bad for everyone so none of your users win | 22:13 |
| sc68cal | and educating users to bring a security group and rules along with images and whatever other stuff they need for the cloud, holistic solution | 22:14 |
| fungi | because making something easier for a majority of users is unfair if you can't solve it for everyone, so punish them all | 22:14 |
| sc68cal | it's not just "upload your image and off we go" | 22:14 |
| *** shardy has quit IRC | 22:15 | |
| sc68cal | fungi: I also do not believe we are "punishing all" users | 22:15 |
| anteaya | okay so we might not come up with a solution today | 22:15 |
| kevinbenton | We're sort of backed into a corner. I don't see how we can swap the default because it's a major change in the end-user expectations of filtering | 22:15 |
| fungi | hyperbole, yes. i apologize | 22:15 |
| anteaya | but I think we are gaining something by sharing our thoughts with each other | 22:15 |
| dougwig | AWS puts this closure in your face initially, forcing its discovery. other big public clouds (DO, rax) are open by default. the magic inability to ping without knowledge is indeed kinda ... not good. | 22:15 |
| kevinbenton | dougwig: so you are proposing a change to horizon :) | 22:16 |
| dougwig | it's not that i want it open by default, per se. | 22:16 |
| clarkb | dougwig: right icmp and a small set of services that are consistently necessary to use the cloud seem like a good compromise | 22:16 |
| anteaya | I think we have consencous that the current situation isn't ideal | 22:16 |
| *** sridhar_ram has quit IRC | 22:16 | |
| fungi | i also disagree and think icmp echo should be allowed to any host which is participating on the global internet, but i get that there are people who still fear ping-of-death will rear its ugly head 20 years later, or think that security by obscurity is good for breakfast | 22:16 |
| anteaya | and that as of yet we don't have agreement on a clear way forward | 22:16 |
| anteaya | but it sounds like we would like one | 22:16 |
| sc68cal | I think icmp echo inbound default is something I'd be willing to give ground on | 22:17 |
| kevinbenton | i think the next step is for clarkb or fungi to file an RFE on neutron so these complaints are visible to the rest of the neutron cores | 22:17 |
| kevinbenton | and so they can see it's coming from infra | 22:17 |
| anteaya | use cases maybe instead of complaints? | 22:17 |
| fungi | well, all icmp really. any operating system which has insecure handling of any icmp echo type is downright silly in this day and age | 22:17 |
| fungi | and blocking arbitrary control and messaging types leads to all sorts of terrible black-hole effects | 22:18 |
| kevinbenton | I think allowing ICMP and blocking others may even be more misleading if users don't understand that there is filtering between them and their VM | 22:18 |
| *** sarob has quit IRC | 22:18 | |
| clarkb | it is also particularly detrimental for fragmentation when dealing with smaller mtus which is a side effect of using neutron in the first place | 22:18 |
| kevinbenton | at least now nothing inbound works | 22:18 |
| fungi | grr, why did i write icmp echo type? icmp type | 22:18 |
| *** Sukhdev has quit IRC | 22:19 | |
| *** sridhar_ram has joined #openstack-infra | 22:19 | |
| *** dimtruck is now known as zz_dimtruck | 22:20 | |
| clarkb | in other news I have a clouds.yaml that works on one host but not another | 22:20 |
| fungi | anyway, yes, all ic protocols and ports for egress, all ip protocol 1 (icmp) types ingress, tcp ports 22 (ssh) and 3389 (rdp), and make it known that base images with password auth are discouraged | 22:20 |
| fungi | gah, ip protocols | 22:20 |
| fungi | i should give up on keyboards today | 22:21 |
| kevinbenton | http://docs.openstack.org/developer/neutron/policies/blueprints.html#neutron-request-for-feature-enhancements | 22:21 |
| anteaya | apparently | 22:21 |
| sc68cal | whoa whoa no 22 or 3389 | 22:21 |
| fungi | sc68cal: how do you reach your instances? | 22:21 |
| clarkb | then the other semi related item we have run into is that other tcp types seem ot be forgotten and are not reliable (eg GRE) | 22:21 |
| *** keedya has quit IRC | 22:21 | |
| sc68cal | fungi: I don't allow 22 or 3389 from the whole internet. | 22:21 |
| fungi | out of band console? vpn tunnel? | 22:22 |
| clarkb | s/tcp/ip/ | 22:22 |
| sc68cal | fungi: I provide a cidr of known good addresses. | 22:22 |
| sc68cal | and only that cidr | 22:22 |
| fungi | sc68cal: the good news is, you can fix that in your security groups before you nova boot! | 22:22 |
| *** alivigni has quit IRC | 22:22 | |
| fungi | i hear neutron makes that pretty easy | 22:22 |
| sc68cal | I am absolutley not a fan of allowing 22 or 3389 by default - that is way out of bounds | 22:23 |
| fungi | those seem like the bare minimum to me | 22:23 |
| clarkb | most of your users are going to need that by default | 22:23 |
| *** achanda has quit IRC | 22:23 | |
| hashar | fungi: I dont buy the ICMP flood argument either. IIRC lot of ISP on internet handle them on a best effort basis so a huge flood of ICMP should see a nice packet loss on the way to the target | 22:23 |
| sc68cal | Having those by default was a great way to get your instances deleted for compliance reasons where I used to work | 22:23 |
| fungi | hashar: yep, icmp response throttling is pretty commonplace for decades now | 22:24 |
| fungi | sc68cal: this is probably a clash between people who want to participate in the internet (and need to be this tall to ride) vs people who want to run private enterprise networks and get to them through something internet-connected | 22:25 |
| *** sarob has joined #openstack-infra | 22:25 | |
| sc68cal | This was not private enterprise networks. | 22:25 |
| sc68cal | you got a v6 address, by default. You were on the internet. | 22:25 |
| sc68cal | so the whole enterprise vs "public cloud" is a red herring | 22:26 |
| fungi | equating to traditional non-virtualized environments, when i paid for a colo i got an internet drop from a stateless, non-filtering routed gateway. all decisions from the end of that cambe down were mine and the responsibility for securing the environment from outside threats was mine alone | 22:27 |
| *** sarob has quit IRC | 22:27 | |
| fungi | i also ran colocation facilities for many years, and that's precisely how we operated | 22:27 |
| sc68cal | cloud != colo | 22:27 |
| *** krtaylor has quit IRC | 22:27 | |
| fungi | cloud: somewhere to run my virtual servers; colo: somewhere to run my physical servers | 22:27 |
| *** sripriya has joined #openstack-infra | 22:27 | |
| anteaya | okay I'm starting to feel like this discussion is less productive than it might be | 22:28 |
| anteaya | we seem to have some emotional attachment to things that I wasn't aware of, sorry about that | 22:28 |
| anteaya | thanks for taking some time to chat about this, I appreciate it | 22:28 |
| anteaya | perhaps we should maybe come back to this again some time? | 22:29 |
| anteaya | thanks for the additional details on both sides | 22:29 |
| fungi | anteaya: it likely boils down to there being extremes of opinion between network engineers, some who feel the internet should be open by default, others who feel the internet should be closed by default | 22:29 |
| anteaya | I certainly learned some things I didn't know before | 22:29 |
| anteaya | fungi: yup, we seem to have some differences here | 22:29 |
| *** david-lyle has quit IRC | 22:29 | |
| anteaya | and I apologize for not knowing in advance how large the gap was | 22:30 |
| anteaya | sorry about that folks | 22:30 |
| anteaya | I value you all | 22:30 |
| anteaya | you are amazing people | 22:30 |
| *** yamahata has quit IRC | 22:30 | |
| anteaya | thank you for your time today | 22:30 |
| sc68cal | fungi: that viewpoint is not uniform across network engineers. | 22:30 |
| *** sripriya__ has quit IRC | 22:31 | |
| fungi | i'm sure there is a gradient of opinions, and possibly middle ground somewhere | 22:31 |
| sc68cal | agreed. | 22:31 |
| anteaya | great | 22:32 |
| fungi | i was more defining the spectrum | 22:32 |
| anteaya | so let's come back to this another day | 22:32 |
| anteaya | and see if we can make more progress finding that middle ground | 22:32 |
| *** annegentl_ has quit IRC | 22:32 | |
| sc68cal | agreed - middle ground would be good | 22:33 |
| fungi | i mean, i'm thrilled that neutron doesn't also block all egress by default. but since i always have to change the security groups anyway, i guess it wuold be adjustable at the same time as opening up ingress | 22:33 |
| ianw | clarkb / greghaynes : i'm trying to get some insight into the rax upload issue at a low level, without nodepool on top. i'm doing something like http://paste.openstack.org/show/488860/ to simulate uploading several images. am i going to see the issue, or is it only when timeouts occur? | 22:33 |
| *** verdurin has quit IRC | 22:33 | |
| ianw | or do i need to do the uploads in parallel threads? | 22:34 |
| clarkb | ianw: you should see it break by doing that if you have a long period between uplodas (longer than the token expiry) | 22:34 |
| clarkb | ianw: basically we need swiftclient to get a new token after the old one expires and that isn't reliable yet | 22:34 |
| ianw | clarkb: how long is token expiry? | 22:34 |
| *** verdurin has joined #openstack-infra | 22:34 | |
| clarkb | ianw: it varies between clouds I think rax is 24 hours? | 22:34 |
| clarkb | don't quote me on that, it is long enough that first upload after a service restart works but second upload a day later may not | 22:35 |
| clarkb | ianw: you might be able to game it by updating time.time to be a day later | 22:35 |
| clarkb | not sure how much of the verification is on the server vs client side for expiration checking | 22:35 |
| fungi | okay, second pass of jjb update with jenkins01 in prepare for shutdown managed to complete without further error | 22:36 |
| jroll | pretty sure rackspace is 24h | 22:36 |
| *** achanda has joined #openstack-infra | 22:36 | |
| *** angdraug has quit IRC | 22:36 | |
| fungi | and one job left to finish before i can clean up and restart jenkins01 now | 22:36 |
| clarkb | sc68cal: kevinbenton: right so understanding that there is a spectrum of opinions here I think the thing to think about is whether or not there is a position that can address the default needs of your "default" users and maybe make people that need fancy to do the fancy themselves. This is in contrast to current setup where everyone must do the fancy regardless | 22:37 |
| *** hashar has quit IRC | 22:38 | |
| sc68cal | clarkb: I agree with the sentiment of everyone having to do all the fancy themselves, and yes I want to reduce the tedium of getting a new tenant up and running | 22:38 |
| *** aysyd has quit IRC | 22:38 | |
| sc68cal | I just think ssh and rdp inbound by default is a non-starter | 22:39 |
| fungi | the other feature request we might want to write up (and i don't know how deeply ingrained in neutron's design this is to even be feasible) is an option to disable filtering _and_ state tracking for a network | 22:39 |
| *** dizquierdo has joined #openstack-infra | 22:40 | |
| fungi | because really, while we can say neutron i want to allow "connections" to and from a range of ports and all protocols, whatever, if the hosts aren't configured to properly handle state for obscure protocols they'll still end up breaking | 22:40 |
| clarkb | right thats the GRE problem | 22:41 |
| fungi | e.g. loading appropriate conntrack modules on the hosts | 22:41 |
| fungi | _if_ the kernel weren't explicitly told to cram this through stateful iptables rules, that would cease to be an issue | 22:41 |
| *** jtomasek has quit IRC | 22:41 | |
| sc68cal | I know a couple instances where state tracking kills a usecase or project that was going to use neutron | 22:42 |
| sc68cal | the issue will be, to get rid of conntrack, you have to disable port security | 22:42 |
| fungi | which can be done on a per-network basis? | 22:42 |
| sc68cal | correct | 22:42 |
| *** piet has quit IRC | 22:43 | |
| fungi | i'm saying disable all filtering and also disable state tracking, so that sounds possible i guess | 22:43 |
| *** rhallisey has quit IRC | 22:43 | |
| clarkb | the altnerative to that would be to explicitly grok more than icmp, tcp, and udp right? | 22:43 |
| sc68cal | fungi: yeah I think disabling state tracking is a side-effect of disabling filtering at this point in time | 22:43 |
| fungi | i get that it's pretty hard to deal with stateless filtering rulesets (i did it for more years than i care to think, but do not wish to relive that) | 22:43 |
| *** annegentl_ has joined #openstack-infra | 22:44 | |
| sc68cal | fungi: yeah I don't think stateless filtering would be fun to go back to :) | 22:45 |
| *** zz_dimtruck is now known as dimtruck | 22:45 | |
| fungi | yay! jenkins01 idle. cleaning up and restarting now | 22:45 |
| anteaya | yay! | 22:45 |
| fungi | sc68cal: so is "disabling port security" (e.g. running a neutron network with no packet filtering and no state tracking) already a feature we can make use of in neutron, or is that a feature request/spec? | 22:46 |
| kevinbenton | fungi: you can use it if you own the network | 22:47 |
| kevinbenton | fungi: but not if you don't | 22:47 |
| fungi | okay, that's a start | 22:47 |
| kevinbenton | fungi: because it shuts off everything including anti-spoofing features | 22:47 |
| sc68cal | fungi: I think we'd just need to verify that no state is being put into conntrack on the hypervisor host, my suspicion is that it it already does. | 22:47 |
| fungi | and by "own the network" you mean neutron network create blah | 22:47 |
| fungi | not completely control the neutron deployment in the openstack cloud | 22:47 |
| sc68cal | correct | 22:47 |
| *** amitgandhinz has quit IRC | 22:48 | |
| fungi | that seems like something we may want to give a shot | 22:48 |
| sc68cal | I forget what release we added the port security api ext | 22:48 |
| clarkb | gah new problem in osic | 22:49 |
| sc68cal | but it's optional... sooooo we're still kind of at square one | 22:49 |
| fungi | though provider networks are probably going to fall in the category of "not controlled by us" | 22:49 |
| kevinbenton | kilo or juno | 22:49 |
| clarkb | the catalog uses the IP address | 22:49 |
| anteaya | clarkb: :( | 22:49 |
| clarkb | which means even /etc/hosts update is almost useless | 22:49 |
| clarkb | there goes that idea | 22:49 |
| fungi | clarkb: oh wow | 22:49 |
| clarkb | fungi: I think our option is to not verify or to wait | 22:49 |
| fungi | that's a complete non-starter | 22:49 |
| fungi | right | 22:49 |
| kevinbenton | fungi: are you seeing the rules interfere with GRE even if you add GRE rules bi-directionally? | 22:49 |
| clarkb | kevinbenton: we add "allow all ip rules" | 22:49 |
| clarkb | kevinbenton: and that isn't sufficient | 22:49 |
| fungi | kevinbenton: does neutron have a list of other ip protocols that have to be worked around in such a fashion? | 22:50 |
| fungi | or the kernel's iptables/conntrack documentation covers this maybe? | 22:50 |
| sc68cal | GRE is udp - what exactly is your "allow all IP rule" | 22:51 |
| fungi | gre is not udp | 22:51 |
| fungi | gre is ip protocol 47 | 22:51 |
| sc68cal | sorry, brainfart | 22:51 |
| fungi | udp is ip protocol 17 | 22:51 |
| fungi | they do share a digit1 | 22:52 |
| sc68cal | So, what protocol is your security group rule | 22:52 |
| fungi | s/1$/!/ | 22:52 |
| sc68cal | for allow all ip | 22:52 |
| *** mriedem has quit IRC | 22:52 | |
| clarkb | sc68cal: whatever the rule create command does for not specigying a type | 22:52 |
| kevinbenton | fungi: so there was a change at some point to make sure allow rules were hit before the ALLOW ESTABLISHED rule that punted to conntrack | 22:53 |
| *** rbrndt has quit IRC | 22:53 | |
| *** dingyichen has joined #openstack-infra | 22:53 | |
| fungi | kevinbenton: oh, so this may simply be that provider not running a new enough neutron to have that? | 22:53 |
| sc68cal | clarkb: ok, I don't remember that off the top of my head - but if you can give us the actual security group rule, it'll list the protocol | 22:54 |
| *** baoli_ has quit IRC | 22:54 | |
| ianw | clarkb: fyi, resetting the system clock into the future does not seem to expire the token | 22:54 |
| *** baoli has joined #openstack-infra | 22:54 | |
| kevinbenton | fungi: i'm not 100% sure of the behavior of iptables when it's missing a module to process a particular protocol | 22:54 |
| kevinbenton | fungi: but it may fix it | 22:54 |
| kevinbenton | fungi: let me find the commit so i can get a release it's present in | 22:54 |
| *** jamesmcarthur has quit IRC | 22:55 | |
| *** andymaier has quit IRC | 22:55 | |
| fungi | this was observed in ovh's environment, which drove us to switch from gre to vxlan for the overlay in devstack multi-node scenarios | 22:55 |
| *** dims has quit IRC | 22:55 | |
| kevinbenton | fungi: i mispoke, it changed it so dropping INVALID packets would happen after user-defined rules | 22:56 |
| kevinbenton | fungi: but it may still apply | 22:56 |
| fungi | which also has potential issues since broadcast over vxlan ends up being multicast ip which also won't really work correctly | 22:56 |
| kevinbenton | fungi: this change is in 8.0.0.0b2 8.0.0.0b1 7.0.3 7.0.2 7.0.1 7.0.0.0rc3 7.0.0.0rc2 7.0.0.0rc1 7.0.0 | 22:56 |
| kevinbenton | fungi: https://github.com/openstack/neutron/commit/0a258afc7ee3c03974dffa2c0dd0b7b367034cc7#diff-abf220de4c2165d9e5bfd6dde12b3f4f | 22:56 |
| fungi | kevinbenton: cool, so liberty or later? | 22:57 |
| kevinbenton | fungi: looks like it | 22:57 |
| fungi | thanks--that's useful information | 22:57 |
| kevinbenton | fungi: do you have control over the agents running vxlan? | 22:57 |
| fungi | kevinbenton: yeah, they're in devstack-running nova instances | 22:57 |
| kevinbenton | fungi: i take it you are running linux bridge? | 22:58 |
| *** baoli has quit IRC | 22:58 | |
| fungi | they form the fake lan between devstack hosts since we don't have control of the actual lan in our providers | 22:58 |
| *** baoli has joined #openstack-infra | 22:59 | |
| kevinbenton | fungi: well i ask because that multicast behavior can be stopped | 23:00 |
| *** sc68cal has quit IRC | 23:00 | |
| kevinbenton | fungi: and flood to unicast tunnels instead | 23:00 |
| *** harlowja_at_home has quit IRC | 23:00 | |
| *** baoli has quit IRC | 23:00 | |
| fungi | kevinbenton: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt | 23:00 |
| *** david-lyle has joined #openstack-infra | 23:00 | |
| *** sridhar_ram has quit IRC | 23:00 | |
| *** dims has joined #openstack-infra | 23:00 | |
| anteaya | kevinbenton: thanks for the ml post | 23:01 |
| fungi | clarkb: do we have a pending change to s/gre/vxlan/ on that? | 23:01 |
| *** baoli has joined #openstack-infra | 23:01 | |
| kevinbenton | anteaya: no prob | 23:01 |
| *** sridhar_ram has joined #openstack-infra | 23:02 | |
| *** asalkeld has left #openstack-infra | 23:02 | |
| clarkb | fungi: we do not, but we should ! | 23:02 |
| clarkb | I am going to take the password file lock now | 23:02 |
| fungi | kevinbenton: so anyway, we don't use the "linuxbridge" driver for neutron to set that up (or neutron at all to set that up), but we do use the bridge driver in the linux kernel | 23:02 |
| fungi | clarkb: i can update that, just need to know if anything more substantial than encapsulation changed when we switched the setup to vxlan | 23:03 |
| *** dingyichen has quit IRC | 23:03 | |
| clarkb | fungi: no that was it | 23:05 |
| fungi | cool, patch on the way in moments before i forget | 23:05 |
| fungi | kevinbenton: so sounds like there is an option we can tweak in the vxlan kernel config to switch from multicast to unicast flood? | 23:06 |
| fungi | if i understand what you were suggesting | 23:06 |
| *** tpsilva has quit IRC | 23:07 | |
| *** ianw has quit IRC | 23:07 | |
| kevinbenton | fungi: sorry, got pulled away for a sec | 23:08 |
| kevinbenton | fungi: yeah, if you are using linux bridge agent, clear the setting for vxlan_group | 23:09 |
| *** doug-fis_ has joined #openstack-infra | 23:09 | |
| *** dingyichen has joined #openstack-infra | 23:09 | |
| *** doug-fis_ has quit IRC | 23:09 | |
| kevinbenton | fungi: that will ensure the 'group' parameter is not passed to the vxlan links created | 23:09 |
| *** doug-fis_ has joined #openstack-infra | 23:09 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on Blowfish Password test https://review.openstack.org/286877 | 23:09 |
| kevinbenton | fungi: then the traffic should be flooded to all tunnels when it's broadcast/multicast | 23:10 |
| *** rfolco_ has quit IRC | 23:11 | |
| *** baoli has quit IRC | 23:11 | |
| *** zhurong has joined #openstack-infra | 23:11 | |
| *** smarcet has joined #openstack-infra | 23:11 | |
| clarkb | ok releasing lock on passwords file | 23:11 |
| clarkb | taking hiera lock now | 23:12 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Update multinode setup doc to VXLAN https://review.openstack.org/286880 | 23:12 |
| *** doug-fish has quit IRC | 23:13 | |
| *** chlong_ has joined #openstack-infra | 23:13 | |
| *** zhurong has quit IRC | 23:14 | |
| *** doug-fis_ has quit IRC | 23:14 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use project_name not _id with OSIC https://review.openstack.org/286881 | 23:14 |
| fungi | kevinbenton: oh! openvswitch actually, looking at the code. i misremembered | 23:14 |
| kevinbenton | fungi: oh, well that should be easy then. openvswitch doesn't support targeting multicast for its vxlan tunnels :) | 23:15 |
| clarkb | fungi: ^ 286881 fixes an osic clouds.yaml thing | 23:15 |
| fungi | kevinbenton: great! theoretical problem averted. thanks! ;) | 23:15 |
| clarkb | fungi: it was linuxbridge but then when we got DVR stuff running there were people that felt strongly it should be ovs... | 23:15 |
| kevinbenton | fungi: np | 23:15 |
| clarkb | fungi: reality is it probably didn't matter all that much when using GRE but with vxlan maybe things like this are different | 23:16 |
| fungi | sounds like yes, they are | 23:16 |
| *** krtaylor has joined #openstack-infra | 23:16 | |
| fungi | at least its default behavior avoided us spewing multicast into our providers' networks they were never going to forward | 23:16 |
| *** Sukhdev has joined #openstack-infra | 23:18 | |
| kevinbenton | this is one of those cases where ovs's shortcomings are a feature! | 23:18 |
| *** pcrews_ has joined #openstack-infra | 23:18 | |
| kevinbenton | OVS overlays at scale are a nightmare because it lacks translation to multicast for encapped broadcast/multicast | 23:18 |
| *** dingyichen has quit IRC | 23:19 | |
| pleia2 | hah | 23:19 |
| *** pcrews__ has quit IRC | 23:19 | |
| *** salv-orl_ has joined #openstack-infra | 23:19 | |
| *** Jeffrey4l has joined #openstack-infra | 23:21 | |
| fungi | yeah, i can see that being undesirable in a large production environment | 23:22 |
| fungi | thinking back to the number of times i used multicast-enabled protocols | 23:22 |
| *** salv-orlando has quit IRC | 23:22 | |
| *** dingyichen has joined #openstack-infra | 23:22 | |
| *** kzaitsev_mb has quit IRC | 23:23 | |
| *** annegentl_ has quit IRC | 23:24 | |
| *** annegentl_ has joined #openstack-infra | 23:25 | |
| kevinbenton | yeah, if you don't block it, a tenant could easily saturate a network by dumping a nice 100mbps multicast stream to a network with lots of instances | 23:26 |
| *** kzaitsev_mb has joined #openstack-infra | 23:26 | |
| *** jpr has joined #openstack-infra | 23:26 | |
| *** erikwilson has joined #openstack-infra | 23:27 | |
| fungi | sounds absolutely crippling | 23:27 |
| *** annegentl_ has quit IRC | 23:30 | |
| *** erikwilson has quit IRC | 23:30 | |
| *** Jeffrey4l has quit IRC | 23:31 | |
| * clarkb does a few more hiera edits | 23:31 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on Blowfish Password test https://review.openstack.org/286877 | 23:33 |
| *** darrenc is now known as darrenc_afk | 23:33 | |
| *** gildub has quit IRC | 23:34 | |
| *** ashleighfarnham has joined #openstack-infra | 23:34 | |
| clarkb | and done releasing hiera lock | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add vexxhost cloud credentials https://review.openstack.org/286895 | 23:37 |
| clarkb | fungi: ^ | 23:37 |
| clarkb | mnaser: ^ you too may be interested in that. It is our first step in bringing in a cloud | 23:38 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenstackId relase 1.0.13 https://review.openstack.org/286896 | 23:38 |
| clarkb | mnaser: once that is in we can work on editing security groups (if they are in place), getting quotas to be what you expect, build a cloud local mirror host, and run some test runs on the initial hosts | 23:39 |
| fungi | clarkb: mnaser: thanks!!! | 23:39 |
| anteaya | is the ca in vexxhost canada or california? | 23:40 |
| clarkb | I think canada | 23:40 |
| fungi | smarcet: 1.0.13 is working well on openstackid-dev i take it? | 23:40 |
| *** david-lyle has quit IRC | 23:41 | |
| clarkb | anteaya: your country is well represented :) | 23:41 |
| *** Qiming has joined #openstack-infra | 23:41 | |
| anteaya | yay! | 23:41 |
| smarcet | fungi: i test it locally | 23:42 |
| anteaya | were doing something | 23:42 |
| anteaya | we're | 23:42 |
| smarcet | we found a security hole | 23:42 |
| clarkb | anteaya: ovh has a region in your CA too | 23:42 |
| smarcet | and we need to release this fix asap | 23:42 |
| *** mriedem has joined #openstack-infra | 23:42 | |
| clarkb | anteaya: with this up and running something like 2/8 regions will be in canada | 23:42 |
| anteaya | near montreal | 23:42 |
| anteaya | yay | 23:42 |
| anteaya | we're doing something useful | 23:42 |
| *** dimtruck is now known as zz_dimtruck | 23:44 | |
| *** annegentl_ has joined #openstack-infra | 23:46 | |
| *** doug-fish has joined #openstack-infra | 23:46 | |
| *** sdake has quit IRC | 23:47 | |
| *** doug-fish has quit IRC | 23:47 | |
| mnaser | anteaya it is in montreal :) | 23:47 |
| clarkb | mnaser: I am going to go ahead and do a quick tempest baseline on the 8GB 8vcpu flavor | 23:47 |
| mnaser | clarkb: +1 on that review, and sure let me know if you run into anything | 23:48 |
| clarkb | mnaser: will do, thanks a gain | 23:49 |
| *** dkranz has quit IRC | 23:49 | |
| fungi | smarcet: given the urgency of that security update, i kicked 286896 straight into the gate pipeline so it doesn't have to wait for available check resources | 23:49 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenstackId relase 1.0.13 https://review.openstack.org/286896 | 23:49 |
| fungi | oh, you just updated it anyway | 23:49 |
| smarcet | fungi: cool tnx :) | 23:49 |
| mnaser | no problem, i'll be up a bit late today as we have some maint stuff to do (but nothing service impacting) | 23:50 |
| *** salv-orlando has joined #openstack-infra | 23:50 | |
| *** salv-orlando has quit IRC | 23:50 | |
| fungi | smarcet: thanks for adding more detail in the commit message! | 23:50 |
| *** salv-orl_ has quit IRC | 23:50 | |
| *** darrenc_afk is now known as darrenc | 23:50 | |
| smarcet | fungi: yes i updated commit message :) | 23:50 |
| *** salv-orlando has joined #openstack-infra | 23:50 | |
| anteaya | mnaser: awesome | 23:51 |
| anteaya | mnaser: may I come for a tour sometime? | 23:52 |
| *** dingyichen has quit IRC | 23:52 | |
| mnaser | sure, we have 2 facilities right now (two az). quite busy in the current few days as we're deploying a bunch of private clouds but drop me an email and we can organize something | 23:52 |
| mnaser | email = my nick @ company name dot com :) | 23:53 |
| *** gordc has quit IRC | 23:53 | |
| *** dingyichen has joined #openstack-infra | 23:54 | |
| fungi | so... many... clouds.yamls | 23:54 |
| fungi | next oscc feature request: composite configuration files | 23:55 |
| fungi | clarkb: i know you didn't start the trend, but why do we hide our project names away in hiera? | 23:57 |
| anteaya | mnaser: nice, thank you | 23:57 |
| clarkb | fungi: usernames too, I honestly don't know | 23:57 |
| clarkb | fungi: maybe keystone treats them as privileged? | 23:57 |
| fungi | right, i can almost see not disclosing the username (though really, no, i'd be fine disclosing that) but even the project name? seems extreme to keep it secret | 23:58 |
| clarkb | fungi: we should ask mordred | 23:58 |
| *** dizquierdo has quit IRC | 23:58 | |
| anteaya | mnaser: sent | 23:59 |
| *** kzaitsev_mb has quit IRC | 23:59 | |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Add more debug logging for closed file issues https://review.openstack.org/286733 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!