Tuesday, 2016-03-01

*** bpokorny_ has joined #openstack-infra00:00
anteayadoug-fish: yes00:00
doug-fishfantastic. Thanks!00:00
*** flepied has joined #openstack-infra00:00
*** otsuka has joined #openstack-infra00:00
mordredsmarcet: well... I can think of a few ways we might address that, all of them seem like terrible ideas00:00
anteayadoug-fish: hopefully a patch to resolve the issue will merge soon00:00
*** thorst has quit IRC00:01
mordredsmarcet: let me go read some of your deploy code real quick and see if I have better non-suck ideas00:01
anteayadoug-fish: and we hope it resolves the issue00:01
anteayadoug-fish: thanks00:01
jamesmcarthurok - i have to take off for a bit. Thanks to anteaya: mordred: and jeblair: for the assist.00:02
jamesmcarthurI’ll check back in a bit after the patch has merged.00:02
*** jamesmcarthur has quit IRC00:02
anteayajamesmcarthur: thanks for helping us understand the issue00:02
*** ZZelle_ has quit IRC00:02
*** rbrndt_ has quit IRC00:02
*** bpokorny has quit IRC00:03
*** zz_dimtruck is now known as dimtruck00:08
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Make lbaasv2-minimal job voting in check and gate  https://review.openstack.org/28632800:09
*** cloudtrainme has quit IRC00:12
anteayapabelanger: how are you deleting spam?00:13
anteayapabelanger: can I do anything to help?00:13
*** tiswanso has joined #openstack-infra00:14
*** sdake has quit IRC00:14
*** rhallisey has joined #openstack-infra00:16
pabelangeranteaya: currently, I've been manually looking at logs and blocking / deleting by hand00:16
ianwSpamapS: so i guess your point is, if it doesn't have CI, is it really supported?00:17
pabelangermoving forward, we should look at automating the clean up00:17
*** Qiming has quit IRC00:17
pabelangersince there is about 1GB of PDF files00:17
jpmaxmanahh I thought you'd automated00:17
pabelangernot yet00:17
pabelangerwill look into that in the morning00:17
jpmaxmanso, just to be clear, we still don't know the attack vector but we're effectively tracking spam accounts and blocking them?00:17
pabelangerand use existing tooling from mediawiki to combat it00:17
jpmaxmanwell the qeustyquest from mediawiki tooling should block it completely00:18
jpmaxmanbut isn't00:18
pabelangerFrom what I see, I don't think we have a security issue.  Just authenticated accounts spamming our wiki00:18
jpmaxmanI'd like to further investigate why that is00:18
*** dalgaaf has quit IRC00:18
jpmaxmanbut how are they getting past captcha question?00:18
*** abitha has quit IRC00:18
jpmaxmando we need to make it harder? or is it broken?  or is there a vulnerability?00:18
pabelangerfrom what I see and read, they broke our captcha00:19
pabelangereither ORC or a human did it00:19
pabelangerquestions aren't that hard00:19
jpmaxmanI'd like some verification on that00:19
*** abitha has joined #openstack-infra00:19
jpmaxmanlike making the questions harder as a start00:19
*** ashleighfarnham has joined #openstack-infra00:19
*** ashleighfarnham has quit IRC00:19
jpmaxmanwe never saw even a momentary break from the spam00:19
jpmaxmanwhen we enabled the captcha00:19
jpmaxmanI'd think it would have taken them at least one minute00:19
jpmaxmanto adjust00:19
*** ybathia has quit IRC00:20
*** _joes_ has quit IRC00:20
pabelangernot sure about that, easy to check however, if we did some stats on logs00:20
jpmaxmanI was watching it on the recently updated special page00:20
pabelangerSo, I'll stop blocking accounts for tonight and wait until the spam starts again00:20
jpmaxmanmaybe we can start by making the question more difficult?00:20
pabelangerwe'll ask an infra-root to block out captcha and see what happens00:21
*** arxcruz has quit IRC00:21
*** _joes_ has joined #openstack-infra00:21
jpmaxmanan impossible question would be ultimately telling, but even making it more difficult might provide some indication00:21
pleia2I can change them back to being unanswerable00:21
pleia2if we want to test whether they're bypassing entirely00:21
pabelangerpleia2: the issue now is, we need spammers :)00:22
pleia2did they disappear? :)00:22
jpmaxmanpleia2: yes, that was my suggestion earlier.  But I agree with pabelanger let's wait until tomorrow00:22
*** Sukhdev has quit IRC00:22
pabelangerhttps://wiki.openstack.org/wiki/Special:NewPages and https://wiki.openstack.org/wiki/Special:RecentChanges are void of them currently00:22
pleia2bots don't know about february 29th00:22
jpmaxmanpabelanger has been going into hand to hand combat with them00:22
jpmaxmanand so far is winning ;)00:22
* pleia2 nods00:22
pabelangerOur current changes have stopped the attack, tomorrow is a new day :)00:23
pleia2tomorrow is fine, but we do want to do it when dev-time is quiet (since it makes the wiki effectively read only for everyone)00:23
jpmaxmanso maybe we can reconvene here at 5:00pm UTC tomorrow00:23
pabelangerpleia2: good idea00:23
jpmaxmansure whenever that is, I can be here, let's just have people online so we can apply the change then undo it quickly00:24
*** abitha has quit IRC00:24
pleia22100 utc or later is probably best00:24
fungiyeah, i would not be at all surprised to see pretty much any solvable captcha bypassed. mechanical turk is a thing00:24
* pleia2 nods00:25
fungiapparently the hip term now is "captcha farmers"00:26
jeblairi generally think of spammers as bot-assisted humans00:26
jpmaxmanIt's possible but we've blocked a lot of spammers there's so many opportunities out there that generally the "captcha farmers" are used for more valuable targets than updating a wiki page.00:28
*** tiswanso has quit IRC00:28
jpmaxmanthese seem to be bots that once you figure out what they're doing they just move on to the next undprotected mediawiki00:28
fungiyes, spam evasion is sort of like outrunning a bear. you don't need to be faster than the bear...00:28
jpmaxmanexacly fungi00:28
*** tiswanso has joined #openstack-infra00:28
fungihowever, being a higher-traffic target, we'll be preferred by ranking-savvy spammers, so we need to be less convenient to spam than other high-traffic wikis out there00:30
jpmaxmanyup you probably got some google juice00:31
jpmaxmanthat they like :)00:31
jpmaxmanI could be wrong but my gut says this isn't human assisted - I think it would be easy enough to figure out00:32
jpmaxmanwhat is the fourth word in this sentence is pretty easily scriptable00:32
fungii'm just glad they've finally lost interest in spamming usenet. in another 10 years e-mail will be mostly spam-free and in 20 years so will wikis ;)00:32
SpamapSianw: yes that's my point. :)00:33
*** boris-42 has quit IRC00:34
anteayapabelanger: was having some dinner, seems like the response is see what happens tomorrow, and if we have time try to automate deletion00:34
*** Keedya_ has joined #openstack-infra00:35
*** Keedya_ has quit IRC00:36
mtreinishinfra-root: when you get a sec to fix the subunit gearman worker we need: https://review.openstack.org/286304 and then to restart the worker after that's applied00:36
mtreinishit'll also be good to land: https://review.openstack.org/285560 if we need to restart it00:36
*** markvoelker has quit IRC00:38
*** Sukhdev has joined #openstack-infra00:38
*** thiagop has quit IRC00:38
openstackgerritSpencer Krum proposed openstack-infra/system-config: Run cacti's node generation from cron  https://review.openstack.org/28446600:38
*** bpokorny_ has quit IRC00:39
*** bpokorny has joined #openstack-infra00:39
nibalizermtreinish: ok i got you00:40
mtreinishnibalizer: cool, thanks00:41
openstackgerritMerged openstack-infra/puppet-subunit2sql: Fix bug on missing subunit2sql data  https://review.openstack.org/28630400:46
*** sridhar_ram1 has joined #openstack-infra00:47
nibalizerinfra-root the puppetmaster isn't applying again00:51
nibalizerbecause the ansible-inventory is failing because the west region isn't there00:51
fungiit won't just skip?00:51
*** baoli has joined #openstack-infra00:52
fungii have a feeling we need to remove it from nodepool.yaml and then from clouds.yaml to avoid the issue jhesketh ran into00:52
nibalizerthe weird thing is I thought jhesketh fixed this last night00:52
*** Sukhdev has quit IRC00:52
jheskethfungi, nibalizer: yes, I removed it from clouds.yaml... maybe I missed something or it reverted00:53
funginodepoold was still trying and failing to reach infra-cloud west earlier today when i was looking in its debug.log00:53
nibalizerfungi: yea when we reference a cloud that doesn't exist it blows up our inventory which aborts00:53
jeblairjhesketh: yeah, you removed it from the wrong clouds.yaml00:53
jeblairjhesketh: you removed it from the one nodepool uses00:54
nibalizerjhesketh: it was working at 2016-03-01 00:17:49,58900:54
jheskethah, and the inventory is cached00:54
jeblairthat's why nodepool broke00:54
jheskethright, I see00:54
jheskethsorry about that :-(00:54
*** smarcet has quit IRC00:54
nibalizeroh so we need a new patch to /etc/openstack/clouds.yaml on the puppetmaster?00:54
jeblairjhesketh: we have too mayn clouds.yamlses :)00:54
fungioh, yes, cached inventory means we have up to 24 hours to see it start breaking00:54
jeblairnibalizer: i think so, if no one else has done it00:54
jheskethnibalizer: yes, and we'll need to probably apply it manually to the puppetmaster00:55
nibalizerim on it00:55
jheskethokay, I'll do a revert for my bad one00:55
nibalizerno i think we need yours too00:55
jeblairyeah, i think nodepool is fine now00:55
mordredjhesketh: oh - also - did you see my response to your patch to ansible?00:56
jeblairjhesketh: nodepool was fixed by a followup change that removed infra-cloud00:56
jheskethmordred: yes, haven't had time to look at it yet though sorry00:56
jheskethjeblair: correct, but we should have the credentials there for when infra-west is back00:56
mordredjhesketh: I wholeheartedly agree with you about the patch and think it's great00:56
jheskethor at least ready for review00:56
mordredjhesketh: I think we need to plumb a config thing through occ- because there isn't really a way to pass parameters to the inventory script in normal operation00:57
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove infracloudwest from ansible-clouds.yaml  https://review.openstack.org/28633700:57
mordredjhesketh: but we've got an example of one other config setting that the inventory reads already, so cargo-culting should be easy enough00:57
fungioh, the nodepool debug errors are about failing to image-delete because it can't find infracloud-west defined, so we need to manually clean up the images00:57
openstackgerritMerged openstack-infra/puppet-subunit2sql: Use first test from subunit_stream for run_at value  https://review.openstack.org/28556000:57
jheskethmordred: yeah I noticed that limitation... but I was thinking about the more general use case of shade/the inventory where you probably want to know when your operation isn't working on a particular cloud00:58
nibalizerinfra-root 28633700:58
jheskethmordred: so I wanted the flag to be set on list_hosts rather than in the config00:58
nibalizerim going to apply that manually as well since we can't auto un-wedge00:58
*** baoli has quit IRC00:58
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Revert "Infra-cloud-west is currently offline"  https://review.openstack.org/28633800:58
jheskethnibalizer: yep, that's what I did yesterday00:58
*** esikachev has joined #openstack-infra00:59
*** zhurong has joined #openstack-infra00:59
*** chenli has joined #openstack-infra00:59
*** ajmiller has quit IRC00:59
*** arif-ali has joined #openstack-infra01:00
*** flepied has quit IRC01:00
*** thorst_ has quit IRC01:00
asselin_hi, I'm trying to figure out why a job couldn't clone from a zuul merger. I see this 404 in the apache logs: [25/Feb/2016:14:34:10 +0000] "GET /p/xyz/abcdef/info/refs?service=git-upload-pack HTTP/1.1" 404 279 "-" "git/1.9.1"01:01
*** ybathia has joined #openstack-infra01:01
*** amotoki has joined #openstack-infra01:01
*** thorst has joined #openstack-infra01:01
*** asselin__ has joined #openstack-infra01:01
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Revert "Remove infracloud-west"  https://review.openstack.org/28634001:02
anteayaasselin_: can you offer more of the log, perhaps in a paste?01:02
asselin_I saw another failed clone with this 500 error in the apache access log: "POST /p/xyz/abcdef/git-upload-pack HTTP/1.1" 500 841 "-" "git/1.9.1"01:02
jheskethnibalizer: do you have the correct cloud.yaml change up for review?01:02
asselin_anteaya, it's just that one line01:03
nibalizerua 28633701:03
anteayaasselin_: interesting01:03
anteayaasselin_: those errors aren't ringing any bells for me01:03
asselin_anteaya, well...this is apache log file...so one line per request. the other lines aren't related. they're 200 OK responses.01:03
anteayaasselin_: ah okay01:03
*** esikachev has quit IRC01:04
fungiasselin_: i wouldn't expect a job to clone from a zuul merger, only fetch zuul refs from one01:04
asselin_fungi, you're right, they are git fetch: git fetch01:05
*** asselin__ has quit IRC01:06
*** apoorvad has quit IRC01:06
*** aeng has quit IRC01:06
jeblairnibalizer: comment on 33701:06
nibalizerjeblair: ack01:07
jeblairnibalizer: i lean toward leaving the cert in place01:07
jeblairsince everything else is in place01:07
jeblairbut it's kind of a toss-up01:07
*** pvaneck has quit IRC01:07
nibalizerjeblair: so my thinking is when we get the servers back we will not call them 'west' or 'east' so anything with that name (including dns) should be wiped out01:07
fungiasselin_: so... depending on the job and the repos in use, it's not guaranteed that the zuul merger will have an appropriate ref for any given repo01:08
jeblairnibalizer: yeah, if you want to go the other way (remove from all-clouds) as a first step in that direction, i'll be +2 on that too01:08
nibalizerya im gonn remove it from all-clouds then01:08
fungiasselin_: for example, devstack-gate will try to retrieve a zuul ref for basically ever source repo it's integrating, but very few of those are actually expected to have that ref01:09
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove infracloudwest from ansible-clouds.yaml  https://review.openstack.org/28633701:09
*** sputnik13 has quit IRC01:09
fungiasselin_: and git doesn't have a "check if this ref exists in the remote" feature, other than to try (and potentially fail) to fetch it01:09
*** thorst has quit IRC01:09
asselin_fungi, true, except that the ref is there and needs to be there. Theses failures are failing the job.01:09
*** kzaitsev_mb has quit IRC01:09
asselin_(very intermittently)01:10
*** jamesmcarthur has joined #openstack-infra01:10
jeblairjhesketh: i -1d your changes; now that it's out of nodepool, better to just leave it that way01:10
jheskethjeblair: yep, I think I agree :-)01:10
jeblairjhesketh: also, as nibalizer says above, we probably won't call them west anyway :)01:10
fungivanilla openstack!01:10
*** AndyU has joined #openstack-infra01:11
jeblairjhesketh: (we will probably have multiple regions in the same data center, and ... yeah, vanilla :)01:11
nibalizerstrawberry openstack!01:11
asselin_fungi, I'm hoping that 500 & 404 errors with thos URLs means something to someone so I can dig deeper.....01:11
jeblairchocolate is my favorite!01:11
anteayaha ha ha01:11
jheskethwhy not use gps coords?01:11
jheskeththat would be safe from physical attacks...01:11
anteayathen someone moves a rack01:12
jeblairfungi: infracloud-pod46row7b01:12
anteayaand we have to redo the config01:12
fungiextreeeeeeemly precise gps coordinates, at floor tile granularity01:12
nibalizerjeblair: jhesketh 286337 reupped01:12
fungiasselin_: so, the mergers should be getting gearman work requests to create the merges, and then signal back to the scheduler that they succeeded, before it will issue a work request for the jobs which use that merge01:13
jheskethnibalizer: +w'd01:13
*** Qiming has joined #openstack-infra01:13
fungiasselin_: are you able to fetch the refs mentioned? do they actually exist but the jobs are not succeeding in getting them? or are they really not there?01:14
asselin_fungi, the refs exists01:14
nibalizerokay i am reapplying the manual application of 337 because puppet put it back and broke itself again01:14
asselin_fungi, let me get you a more complete paste of log files with my analysis....01:15
fungiasselin_: so your jobs are occasionally getting a 404 trying to retrieve a ref which actually exists when you try to fetch it yourself from the same zuul merger?01:15
asselin_actually I didn't try to fetch it...I just checked it was there in the git filesystem01:16
asselin_but that's a good idea01:16
*** ybathia has quit IRC01:16
*** apoorvad has joined #openstack-infra01:17
*** sigmavirus24 is now known as sigmavirus24_awa01:17
asselin_fungi, http://paste.openstack.org/show/488667/01:18
openstackgerritMerged openstack-infra/system-config: OpenstackId relase 1.0.12  https://review.openstack.org/28631501:20
*** pfallenop has quit IRC01:20
anteayaasselin_: that is a great paste01:20
*** angdraug has quit IRC01:20
*** yamamoto_ has quit IRC01:23
*** Daisy has joined #openstack-infra01:23
*** aeng has joined #openstack-infra01:23
fungiasselin_: i agree the debug log concurs with what you found on the filesystem01:23
asselin_anteaya, fungi full paste of the 500 error issue: http://paste.openstack.org/show/488668/01:26
*** yamamoto has joined #openstack-infra01:26
chenlihello, anyone has comment on this : http://lists.openstack.org/pipermail/openstack-dev/2016-February/087522.html01:27
chenliCan ovs repository added to \$PROJECTS variable in the job definition ?01:27
*** Daisy has quit IRC01:27
*** sripriya has quit IRC01:31
fungiasselin_: also the zm debug log looks similar to our production mergers, which i have not seen any indication of experiencing the issue you've seen on yours01:31
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Remove QA health link from status page  https://review.openstack.org/28635001:32
asselin_fungi, any insight as to what those urls are doing?01:35
*** tphummel has quit IRC01:35
*** [1]Thelo has joined #openstack-infra01:37
asselin_info/refs?service=git-upload-pack 404 ?01:37
asselin_GET ^^01:38
fungiasselin_: you're using distro packages for your git, not any custom-compiled git? unlikely but checking to be thorough01:38
*** markvoelker has joined #openstack-infra01:38
asselin_I can't imagine we're using any custom-compiled anything....01:39
*** Thelo has quit IRC01:39
*** [1]Thelo is now known as Thelo01:39
fungijust digging through various problems with git-upload-pack 404 and 500 responses01:40
asselin_what exactly? I'm interested to know where to look. just google?01:41
*** sam_wan has joined #openstack-infra01:41
fungiyeah, web searches01:43
fungiyou've got apache's mod_cgi enabled?01:43
*** bgaifullin has quit IRC01:43
fungii'm not really finding much anything that could cause this to be intermittent, but i suppose flawed fallback behaviors could provide an explanation01:44
*** apoorvad has quit IRC01:44
nibalizeransible is chugging along nicely now01:45
*** thorst has joined #openstack-infra01:45
*** jamielennox is now known as jamielennox|away01:45
fungithanks, nibalizer!01:45
*** yamamoto has quit IRC01:45
asselin_fungi, http://paste.openstack.org/show/488671/01:45
nibalizerfungi: can you peek at the emergency file and remove aanything that doesn't need to be there01:45
*** Keedya_ has joined #openstack-infra01:46
*** Jeffrey4l has joined #openstack-infra01:46
*** kingia has quit IRC01:47
fungiasselin_: yeah, looks similar to ours http://paste.openstack.org/show/48867301:48
asselin_as expected :)01:48
*** watanabe_isao has joined #openstack-infra01:50
funginibalizer: i don't see anything in there i know for sure should be removed, except maybe controller00.hpuseast.ic.openstack.org if we've also removed that region from clouds.yaml01:51
nibalizerfungi: ok01:51
funginibalizer: clarkb probably knows why logstash-worker20.openstack.org is in there01:51
*** sarob has quit IRC01:51
nibalizerwe can take out cacti right?01:51
clarkbbecause it is running logstash 2.001:51
nibalizerit was just disabled because we wanted to GoFast(TM)01:52
clarkband I haven't been able to get reviews on the changes to switch everything to 2.001:52
anteayanibalizer: what is the emergency file?01:52
funginibalizer: cacti.openstack.org is in there until we stop snmp reindexing from puppet exec01:52
funginibalizer: and i think puppet on afstest.openstack.org is probably just plain broken01:52
nibalizerhttps://review.openstack.org/284466 is a stab at that01:52
*** andreykurilin__ has quit IRC01:53
prometheanfireSpamapS: why didn't you +w this? https://review.openstack.org/28196001:53
fungianteaya: the non-puppeted skip file at /etc/ansible/hosts/emergency on puppetmaster.o.o01:53
anteayafungi: thanks01:53
*** thorst has quit IRC01:53
clarkbnibalizer: https://review.openstack.org/#/c/285473/4 adds osic creds01:53
clarkbnibalizer: if you are in a reviewing mood01:54
fungianteaya: it's our workaround for timing and/or catch-22 issues with disabling puppet for select hosts through git01:54
anteayaprometheanfire: noone is obliged to +w anything they don't want to01:54
nibalizerclarkb: got the hiera keys set?01:54
fungiwhich means we should probably move these entries to the proper host list if they're going to stay disabled for a while01:54
prometheanfireanteaya: I know, but it'd be nice to know why, know if I need to do anything01:54
anteayaprometheanfire: just because they can +A after another core +2'd a patch doesn't mean they are obligated to01:54
nibalizerthey don't have dns?01:54
*** mtanino has quit IRC01:55
anteayaprometheanfire: well asking for a reason is fine, but it is the reviewer's choice01:55
prometheanfireanteaya: I understand, but can I not ask for clarification? or am I not allowed to talk to them?01:55
anteayafungi: that makes sense, thank you01:55
anteayaprometheanfire: you certainly may ask questions as may anyone01:56
fungiasselin_: i'm at a loss. did you try fetching the same refs remotely? i can't recall if you had an example of doing that to see whether a given ref was only temporarily failing or indefinitely failing01:56
*** yamamoto has joined #openstack-infra01:57
*** yamamoto has quit IRC01:58
*** sarob has joined #openstack-infra01:58
*** doug-fish has quit IRC01:58
asselin_fungi, both work01:59
fungiyeah, i'm out of ideas. looked in syslog/dmesg for filesystem issues?02:00
asselin_good idea02:00
openstackgerritTristan Cacqueray proposed openstack-infra/zuul: Add support for layout configuration split  https://review.openstack.org/15229002:01
fungiasselin_: is it a network-attached filesystem maybe?02:01
asselin_not sure: http://paste.openstack.org/show/488674/02:01
asselin_no...openstack base filesystem02:02
asselin_no cinder volumes or anything like that02:02
*** aeng has quit IRC02:02
clarkbnibalizer: yes keys are all set02:02
pleia2sprint blogged http://princessleia.com/journal/?p=11335 (it'll go to openstack planet too)02:03
jamesmcarthuropenstackID is back up02:03
Keedya_anteaya: hello02:03
*** sarob has quit IRC02:04
*** rhallisey has quit IRC02:04
fungiasselin_: dmesg -T is probably more helpful, to get timestamps translated from relative to absolute, but none of those entries look related anyway02:04
*** dims has quit IRC02:04
* asselin_ learned something new02:05
*** reed_ has quit IRC02:05
Keedya_anteaya, clarkb I am hoping to push the new project (shovel plugin) to Openstack02:05
asselin_feb 16 & 17....so nothing for quite a few days02:06
fungiyeah, don't completely trust the times reported by -T either since it's still based on kernel ticks since boot and can drift pretty significantly from localtime (at least on on-systemd platforms)02:06
jamesmcarthurfungi: mordred: jeblair: jpmaxman: smarcet: I’d love to talk about how to make this easier for all parties moving forward. I realize Infra isn’t available to jump at our every request. At the same time, having to wait 3 hours for a fix is madenning when a website is down.02:06
clarkbfungi: if you ever suspend dmesg -T is very wrong02:07
clarkbfungi: because kernel isn't running while suspended02:07
*** rguillebert has quit IRC02:07
fungiyep, hopefully servers don't get suspended too often02:07
fungibut clouds _do_ actually do that to virtual machines02:08
*** markvoelker has quit IRC02:08
fungiluckily usually not for days at a time02:08
*** markvoelker has joined #openstack-infra02:08
*** Sukhdev has joined #openstack-infra02:08
*** dims has joined #openstack-infra02:08
*** baoli has joined #openstack-infra02:11
asselin_fungi, ok...well git fetch results in the GET ...git-upload-pack 200 request02:11
fungijamesmcarthur: i agree, we can encourage more integrated participation from both directions, and find a way to improve the current configuration management for those systems to be more robust and better tested. with automated deployment, we should be able to make deployment validation and functional testing of these changes possible so we avoid as many emergencies02:13
anteayapleia2: nice post02:13
anteayapleia2: bunnies!02:13
asselin_actually, trying the other I got a GET followed by a POST02:13
*** dims has quit IRC02:13
pleia2anteaya: thanks :)02:14
*** Sukhdev has quit IRC02:14
fungiasselin_: are those systems heavily loaded when you hit these errors? maybe the git backend is trying to create packfiles but take too long to complete and end up returning an error at the apache layer?02:14
fungithat would explain why you only issue a get later when you retry (packfile exists at that point and is returned directly)02:15
nibalizerpleia2: awesome!02:15
asselin_fungi, can you elaborate what you mean by 'systems' and 'git backend'? I'm not familiar with the flow. It is possible the system was loaded at the time...I don't know how to check for that historically though.02:16
fungiasselin_: the server where your zuul-merger's apache is running02:17
*** dims has joined #openstack-infra02:17
fungiasselin_: the git backend cgi called from apache will serve up packfiles if they exist, or create them first if they don't02:17
asselin_fungi, in theory if they were loaded, what would be the fix? have another zuul merger?02:18
*** baoli has quit IRC02:18
*** baoli has joined #openstack-infra02:18
asselin_ok and the packfiles are on the local filessystem?02:18
fungiasselin_: well, if you see your zuul mergers under significant load, it's at least designed to be broadly scalable by simply booting more mergers02:18
*** aeng has joined #openstack-infra02:18
fungiyeah, packfiles are in the git repos. they're basically aggregations of refs02:19
fungithey're an abstraction which will be created within a copy of the repo, so in this case by the git-smart-http backend cgi in response to requests for some unpacked refs02:19
anteayaI have to go to bed now in order to get up in the middle of the night02:19
fungig'night, anteaya!02:20
asselin_anteaya, good night02:20
*** zhurong has quit IRC02:20
*** zhurong has joined #openstack-infra02:21
*** baoli has quit IRC02:22
*** baoli has joined #openstack-infra02:23
fungiasselin_: maybe someone else has alternative ideas, but i'm wiped. need to kick back and get some sleep. good luck tracking it down--this one seems more elusive than usual02:24
asselin_fungi, thanks for you help02:24
asselin_good night02:24
*** bpokorny_ has joined #openstack-infra02:26
*** fawadkhaliq has quit IRC02:27
*** Keedya_ has quit IRC02:28
*** bpokorny has quit IRC02:29
*** bpokorny_ has quit IRC02:31
*** chenli has quit IRC02:32
*** blogan has quit IRC02:32
*** ptoohill has quit IRC02:32
*** Keedya_ has joined #openstack-infra02:33
*** x00350071 is now known as xiangxinyong02:34
*** flepied has joined #openstack-infra02:34
*** andymaier has quit IRC02:34
*** sridhar_ram1 has quit IRC02:35
*** armax has quit IRC02:35
*** kzaitsev_mb has joined #openstack-infra02:35
*** yamamoto has joined #openstack-infra02:42
*** Keedya_ has quit IRC02:45
*** Keedya_ has joined #openstack-infra02:46
openstackgerritEmilien Macchi proposed openstack-infra/project-config: fix zuul/layout for puppet-heat  https://review.openstack.org/28636202:48
*** Keedya_ has quit IRC02:51
*** thorst has joined #openstack-infra02:51
*** kingia has joined #openstack-infra02:54
openstackgerritMerged openstack-infra/system-config: Remove infracloudwest from ansible-clouds.yaml  https://review.openstack.org/28633702:55
openstackgerritTristan Cacqueray proposed openstack-infra/gerritbot: Add change-created event type  https://review.openstack.org/28636602:56
openstackgerritTristan Cacqueray proposed openstack-infra/shade: Fix heat create_stack and delete_stack  https://review.openstack.org/27604502:57
*** thorst has quit IRC02:58
*** esikachev has joined #openstack-infra02:59
*** baoli has quit IRC03:00
*** rockyg has joined #openstack-infra03:01
*** sripriya has joined #openstack-infra03:01
*** dimtruck is now known as zz_dimtruck03:01
*** esikachev has quit IRC03:04
*** kzaitsev_mb has quit IRC03:05
*** rockyg has quit IRC03:05
openstackgerritClark Boylan proposed openstack-infra/system-config: Add OSIC clouds.yaml details  https://review.openstack.org/28547303:07
clarkbnow with less merge conflict03:07
*** ajmiller has joined #openstack-infra03:08
craigeping jhesketh or any other infra root folk that may be lurking.03:10
jheskethcraige: pong03:10
* craige is in need of another copy storyboard.sql03:10
craigePretty please :-D03:10
jheskethhmm I haven't done that before and not sure what might need scrubbing etc03:11
craigeoh, I thought you'd been shown :-/03:11
jheskethno, sorry03:11
*** zz_dimtruck is now known as dimtruck03:12
craigeIIRC there was a script that dumps it to a location I can snavel it form03:12
* craige checks system-config/tools03:12
jheskethhmm, might be best to wait for somebody with experience?03:12
craigePerhaps but jeblair was clear anyone can run it.03:12
* craige can't seeit though.03:13
craigeso I may have it all wrong :-D03:13
*** Keedya_ has joined #openstack-infra03:15
*** gildub has joined #openstack-infra03:15
*** woodster_ has quit IRC03:16
*** Keedya_ has quit IRC03:17
*** dims has quit IRC03:19
*** sripriya has quit IRC03:19
*** yuanying has joined #openstack-infra03:21
*** yuanying_ has quit IRC03:23
*** Keedya_ has joined #openstack-infra03:23
*** baoli has joined #openstack-infra03:25
clarkbcraige: maybe it was in puppet-storyboard/03:26
*** Keedya_ has quit IRC03:27
*** yuanying has quit IRC03:27
*** kencjohnston has joined #openstack-infra03:29
craigePerhaps clarkb03:31
*** Nakato has quit IRC03:31
* craige looks03:31
clarkbjhesketh: if you get a moment 285473 has been +2'd by mordred and nibalizer at different times but run into rebase fun, is first step in using osic03:32
*** rossella_s has quit IRC03:32
*** rossella_s has joined #openstack-infra03:33
clarkbwith that in I can use https://review.openstack.org/#/c/285477/4 to boot a mirror node in osic then we can point nodepool at it03:34
*** sarob has joined #openstack-infra03:38
*** sarob has quit IRC03:42
*** Nakato has joined #openstack-infra03:45
*** watanabe_isao has quit IRC03:46
ianwclarkb: no known issues with uploading images that you know of?  trying to figure out a f23 failure that only seems to replicate in upstream and want to make sure i'm not running env too different ...03:46
*** watanabe_isao has joined #openstack-infra03:46
clarkbianw: not tht I know of but havent really looked in a week or so03:46
jheskethclarkb: looking now03:46
ianwi know the builds are ok, which is half-way :)03:47
ianwotherwise, i've got some issue on my hand that only replicates in upstream and involves some intersection of polkit, systemd & dbus.  any of those are unfun to debug, all three together...03:48
*** rlandy has quit IRC03:48
*** baoli has quit IRC03:49
ianwif any system-config people feel like looking at -> https://review.openstack.org/#/c/285876/ (fix pypi install of requests) that would be great too03:49
*** links has joined #openstack-infra03:49
craigeno luck finding to anywhere else, clarkb, jhesketh.03:50
craigeI believe jeblair had it on the storyboard host...03:50
*** thorst has joined #openstack-infra03:56
clarkbianw: this isn't something that can be fixed in fedora I take it?03:58
clarkb(it has been a known issue for a couple years now I think(03:58
*** chenli has joined #openstack-infra03:59
ianwclarkb: i think mostly the problem is that the un-vendored directories are turned into symlinks03:59
jheskethcraige: righto.. unfortuantely I'm not really comfortable with doing that sorry. Might be best to ask fungi or jeblair when they are around03:59
craigeYep, understood jhesketh03:59
ianwclarkb: and rpm barfs at overwriting a symlink with a directory from a package03:59
clarkbianw: the underlying issue is that fedora/centos/rhel don't pip install to /usr/local03:59
clarkbso yum/dnf and pip fight over the same paths04:00
ianwclarkb: well, yeah, that ... but that's not going to change quickly04:00
ianweven just shipping the files list with packages would allow pip to uninstall them correctly & overwrite04:00
*** kzaitsev_mb has joined #openstack-infra04:02
*** chlong_ has quit IRC04:03
*** kencjohnston has quit IRC04:03
*** _amrith_ is now known as amrith04:03
*** thorst has quit IRC04:03
ianwin *most* cases we have an uneasy truce ... but when directories are replaced with symlinks, things just start getting confused04:04
*** kzaitsev_mb has quit IRC04:07
*** bpokorny has joined #openstack-infra04:07
*** chenli has quit IRC04:07
*** yuanying has joined #openstack-infra04:10
openstackgerritIan Wienand proposed openstack-infra/project-config: Add a dib-builddate file  https://review.openstack.org/28637404:12
ianwclarkb / jhesketh : ^ i think this would allow me to not bother you when i'm checking what image version tests are running on04:13
*** armax has joined #openstack-infra04:15
*** yamahata has joined #openstack-infra04:16
openstackgerritIan Wienand proposed openstack-infra/project-config: Add a dib-builddate file  https://review.openstack.org/28637404:16
jheskethianw: cool, lgtm04:16
*** fawadkhaliq has joined #openstack-infra04:18
clarkbianw: +A04:19
*** dimtruck is now known as zz_dimtruck04:22
*** sam_wan has quit IRC04:32
*** chenli has joined #openstack-infra04:36
*** sam_wan has joined #openstack-infra04:45
*** amrith is now known as _amrith_04:46
clarkbianw: image uplaods in rax look staleish04:46
clarkbOpenStackCloudException: Failed at action (create_container) [No tenant specified]04:46
clarkblooks like we are attempting to reauth now with newer oscc but aren't providing enough information to do so?04:47
*** tiswanso has quit IRC04:47
clarkbpreviously we would just fail on our auth being invalid so this is progress I suppose04:47
*** sridhar_ram1 has joined #openstack-infra04:50
ianwclarkb: ok, good (?) it would explain things for me ...04:58
ianwif i can help...04:58
*** fedexo has joined #openstack-infra04:59
clarkbianw: mostly trying to figure out why the tenant/project would be missing and not seeing it05:00
clarkbthis is a trickyish thing becuase it involves token expiration05:01
clarkbagainst a real cloud05:01
clarkbthat is the only cloud that does image uploads in this manner05:01
*** kingia_ has joined #openstack-infra05:01
*** thorst has joined #openstack-infra05:01
*** kzaitsev_mb has joined #openstack-infra05:04
*** kingia has quit IRC05:05
openstackgerritMerged openstack-infra/project-config: Add a dib-builddate file  https://review.openstack.org/28637405:07
*** thorst has quit IRC05:08
clarkbjhesketh: good point on fs detection, I mostly just modified what was already there but ther eprobably is a better way to determine that "this isn't just a raw block device we should leave it be"05:09
*** kzaitsev_mb has quit IRC05:09
clarkbre 28547705:09
*** ajmiller has quit IRC05:16
*** blogan_ has joined #openstack-infra05:18
*** salv-orl_ has joined #openstack-infra05:19
*** ptoohill has joined #openstack-infra05:20
*** bpokorny has quit IRC05:21
*** salv-orlando has quit IRC05:22
*** jamesmcarthur has quit IRC05:28
*** abregman has joined #openstack-infra05:35
*** jogo has quit IRC05:38
*** oomichi_ has joined #openstack-infra05:40
*** oomichi_ has quit IRC05:40
*** jogo has joined #openstack-infra05:41
*** fawadkhaliq has quit IRC05:42
*** kushal has joined #openstack-infra05:43
*** chenli has quit IRC05:45
clarkbianw: at this point probably the best thing would be to incrase the verbosity of shade, swiftclient, and os-client-config logging when running within the nodepool builder05:46
clarkbianw: I can work on that patch in the morning (builders have their own config now so yay)05:46
*** jaosorior has joined #openstack-infra05:49
*** esker has quit IRC05:49
*** esker has joined #openstack-infra05:50
*** Sukhdev has joined #openstack-infra05:55
*** chenli has joined #openstack-infra05:55
*** fawadkhaliq has joined #openstack-infra05:58
*** esikachev has joined #openstack-infra05:59
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/28639306:01
*** Daisy_ has joined #openstack-infra06:02
*** lucasagomes has quit IRC06:02
*** lucasagomes has joined #openstack-infra06:02
*** Daisy_ has quit IRC06:03
*** esikachev has quit IRC06:04
*** kzaitsev_mb has joined #openstack-infra06:05
*** thorst has joined #openstack-infra06:06
*** Kiall has quit IRC06:06
*** Kiall has joined #openstack-infra06:08
*** rcernin has joined #openstack-infra06:09
*** kzaitsev_mb has quit IRC06:10
*** sdake has joined #openstack-infra06:11
*** sridhar_ram1 has quit IRC06:12
*** thorst has quit IRC06:13
tobiash_jhesketh, jeblair: I need to add ProxyCommand support for connecting a zuul instance to our gerrit. What option would be favoured by you? Create a config option for specifying this in zuul. conf or add support for ~/.ssh/config?06:17
clarkbtobiash_: this comes up fairly frequently and my typical suggestion is to not solve this in zuul directly06:18
clarkbyou can typically run the proxy setup external to zuul and foraard through it for zuul06:19
tobiash_clarkb: thx, I'll try this06:20
clarkbunlike say http(s) there aren't cobsistent and reliable methods for proxying ssh so it can get complicated to handle all the cases06:20
*** e0ne has joined #openstack-infra06:23
*** lucasagomes has quit IRC06:24
tobiash_I use corkscrew for proxying it over an http proxy06:25
tobiash_works well with normal ssh and proper ssh config06:25
jhesketh+1 to what clarkb said06:26
tobiash_does zuul manage a single connection to gerrit?06:26
*** lucasagomes has joined #openstack-infra06:27
*** sdake has quit IRC06:27
*** e0ne has quit IRC06:31
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/28590106:31
ianwclarkb: ok, ... i do have a rax account so i guess i could setup things to upload and see, but it would take me a fair bit to context switch in06:32
*** sdake has joined #openstack-infra06:33
*** e0ne has joined #openstack-infra06:33
*** e0ne has quit IRC06:34
*** dmellado has quit IRC06:34
*** dmellado has joined #openstack-infra06:39
*** roxanaghe has quit IRC06:39
*** Sukhdev has quit IRC06:45
*** harlowja_at_home has quit IRC06:49
*** asalkeld has joined #openstack-infra06:52
jamespageAJaeger, morning - this may be a ignorant question but what happens next with regards https://review.openstack.org/#/c/232705/ ?06:52
*** bgaifullin has joined #openstack-infra06:53
*** maishsk has joined #openstack-infra06:53
pleia2jamespage: the commit message leads me to believe the change in question is being moved forward with, but it's been Abandoned, can you adjust the commit message for accuracy?06:54
pleia2jamespage: I *believe* you're simply moving Juju charms into the namespace, but not yet going for TC approval06:55
pleia2aside from that, you just need another +2 and approval from one of us infra-core people06:55
pleia2(or project-config core)06:55
jamespagepleia2, that's correct - TC wanted us to move dev first before assessing as a formal project06:56
*** Daisy has joined #openstack-infra06:56
* pleia2 nods06:56
jamespagepleia2, the project change is currently abandoned/deffered for now06:57
*** Daisy has quit IRC06:58
jamespagepleia2, but i will be re-ssurecting that once we move into the openstack namespace...06:58
*** achanda has quit IRC06:58
jamespagepleia2, do you still need me to amend the commit message? I think the reference is still pertinent06:58
nibalizerjamespage: happy to help you out07:00
pleia2I suppose it's ok07:00
pleia2having a look at the rest of the change now07:01
nibalizerjamespage: so are you super super dooper sure those names are right07:01
nibalizerbecause renaming them is a royal pain07:01
jamespagenibalizer, yup07:01
pleia2I'm checking all the github links too ;)07:01
jamespageoh one of the checks does that07:02
pleia2I meant, matching with naming, etc07:03
nibalizerpleia2: this lgtm07:03
*** maishsk has quit IRC07:03
nibalizerill only +2 if you're going to vote tonight07:03
pleia2nibalizer: +2ed07:03
*** korzen has joined #openstack-infra07:03
pleia2I should go to BED tonight07:03
nibalizerorrrrrr we could stay up all night breaking things!07:03
jamespagenibalizer, pleia2: thankyou!07:04
nibalizerjamespage: its been approved so you'll have tasty new repos in a bit07:04
nibalizernow the trick is to make sure you don't get pull requests and development on the github07:04
*** scheuran has joined #openstack-infra07:05
nibalizerso maybe make those read only or delete all the code or something07:05
nibalizer(dont do this until our stuff has time to grab though)07:05
jamespagenibalizer, they are transitional only and will be dropped post migration07:05
nibalizerdid you have them in lp before?07:05
jamespagenibalizer, yup07:05
jamespageunder bzr07:05
*** oanson has quit IRC07:06
*** kzaitsev_mb has joined #openstack-infra07:06
*** Daisy_ has joined #openstack-infra07:06
*** korzen_ has joined #openstack-infra07:07
*** korzen has quit IRC07:08
*** sdake has quit IRC07:10
jamespagenibalizer, do I get setup automatically with perms for review in gerrit on those repos? I need to setup the current set of committers as well07:10
pleia2an initial member of the groups will need to be added by one of us07:10
nibalizerno what we'll do is add you to the group then you can add remove folk07:10
*** kzaitsev_mb has quit IRC07:11
*** thorst has joined #openstack-infra07:11
nibalizerjamespage: do you have a twitter07:11
*** Daisy_ has quit IRC07:11
pleia2he's teh @javacruft07:12
*** sdake has joined #openstack-infra07:12
* nibalizer tweets07:12
jamespagenibalizer, i do and i am javacruft07:12
nibalizerpleia2: you'll love this07:12
nibalizeri had my ubuntu phone for like 24 hours before I briked it07:13
pleia2my Nexus 7 is running a dev iso of it, but I never bought an actual ubuntu phone07:14
openstackgerritYuriy Taraday proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests  https://review.openstack.org/28562007:15
jamespagenibalizer, pleia2: i had a bq for a while but found it to hard to travel with07:15
jamespageonly being dual band...07:15
* pleia2 nods07:15
*** achanda has joined #openstack-infra07:15
*** mikelk has joined #openstack-infra07:16
openstackgerritMerged openstack-infra/project-config: Add Juju Charms for OpenStack  https://review.openstack.org/23270507:17
*** thorst has quit IRC07:18
*** mrmartin has joined #openstack-infra07:27
AJaegergood morning infra07:28
AJaegerjamespage: congrats to your new repos - and thanks pleia2 and nibalizer for approving.07:28
AJaegerianw: http://docs.openstack.org/infra/manual/creators.html#decide-status-of-your-project is one place that documents official/unofficial projects07:28
AJaegerlifeless: are you still awake and around? I have a pip question regarding constraints...07:29
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack-infra/zuul: Keep py3.X compatibility for urllib  https://review.openstack.org/26117307:31
*** bnemec has quit IRC07:36
*** bnemec has joined #openstack-infra07:39
*** roxanaghe has joined #openstack-infra07:39
yolandagood morning07:40
yolandayay, jamespage, charms!07:40
max_loburGood Morning All07:40
*** vgridnev has joined #openstack-infra07:41
max_loburCan someone take a look at the project-config patch please https://review.openstack.org/#/c/281301/  It needs another +207:41
jamespageAJaeger, thanks - and thanks for your patient reviews :-)07:42
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/28639307:42
openstackgerritMerged openstack-infra/project-config: Publish bashate docs  https://review.openstack.org/28630507:43
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Remove check requirements from Kingbird"  https://review.openstack.org/28535607:43
lifelessAJaeger: ish; go on?07:44
*** roxanaghe has quit IRC07:44
AJaegerlifeless: cinder is using constraints now in their tox.ini and they run into problems with older pip07:44
AJaegerlifeless: I added you to an email as well for async communication07:45
AJaegerlifeless: http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2016-02-29.log.html#t2016-02-29T20:58:1607:45
AJaegerI assume they need pip 7.1, correctly?07:45
* AJaeger just wonder why it now pops up...07:45
lifelessAJaeger: yes; we don't support older pips07:46
lifelessAJaeger: there used to be an infra manual that said to upgrade pip and tox and stuff07:46
lifelessAJaeger: its moved now, and there's a pending patch to overhaul recommendations07:46
AJaegerlifeless: do you have a link handy to share with the cinder folks?07:47
*** flepied has quit IRC07:47
lifelesspython.html from memory07:48
*** sdake has quit IRC07:48
* AJaeger can't find it...07:50
*** esker has quit IRC07:50
*** jtomasek has joined #openstack-infra07:50
openstackgerritMerged openstack-infra/project-config: fix zuul/layout for puppet-heat  https://review.openstack.org/28636207:51
AJaegerlifeless: http://docs.openstack.org/project-team-guide/project-setup/python.html ;)07:52
*** maishsk has joined #openstack-infra07:52
lifelessAJaeger: that actually has different prose to the older doc; there's a pending patch under discussion to get it updated07:52
*** esker has joined #openstack-infra07:54
AJaegeryolanda: could you review https://review.openstack.org/#/c/284371/ as well, please?07:55
AJaegerlifeless: yes https://review.openstack.org/264398 - send out via email.07:55
AJaegerlifeless: thanks, I'm sure the discussion will continue, let's see in which form later today (or tomorrow for you).07:56
*** sam_wan has quit IRC07:57
*** k4n0 has joined #openstack-infra07:57
AJaegerlifeless: just to double check: 7.1 has oldest support version but 8.x recommended?07:58
*** sam_wan has joined #openstack-infra07:58
*** sam_wan has quit IRC07:59
openstackgerritMerged openstack-infra/project-config: Add smaug-dashboard and python-smaugclient to gerritbot  https://review.openstack.org/28600507:59
*** sam_wan has joined #openstack-infra08:00
*** achanda_ has joined #openstack-infra08:02
AJaegeryolanda: https://jenkins04.openstack.org/job/gate-openstack-chef-repo-chef-rake-integration-nv/6/console is running enelessly, can you kill 284730. jklare that's your patch08:03
AJaegeryolanda, jklare that change is running for 18 hours and on top of check queue now http://status.openstack.org/zuul/08:04
jklareAJaeger just saw this, how can i kill it?08:04
openstackgerritMerged openstack-infra/project-config: Remove obsolete dvipng install  https://review.openstack.org/28624108:04
*** achanda has quit IRC08:04
*** achanda_ has quit IRC08:04
jklareAJaeger it looks like it ran into some jenkins/java exception, but did not error out08:04
anteayasubmitting a new patchset should reset the jobs in the check queue08:05
*** k4n0 has quit IRC08:05
AJaegerjklare: please keep an eye open on it - and feel free to come back to ask for debugging help (I can't do it but an infra-root might be) if this continues08:06
AJaegermorning, anteaya. Thanks!08:06
*** slagle has quit IRC08:06
anteayaAJaeger: morning and welcome08:06
jklareAJaeger i think this is due to the bridging of eth1... we need to change that in the recipe08:06
*** kzaitsev_mb has joined #openstack-infra08:07
anteayaif the problem is with the patch, resetting the check jobs won't help08:07
anteayaas the same issue will keep happening08:08
*** achanda has joined #openstack-infra08:08
jklareanteaya i will try to patch the patch :D08:08
jklareanteaya give me a second08:08
*** fawadkhaliq has quit IRC08:08
anteayatake all the time you need08:08
anteayaand good luck08:08
*** armax has quit IRC08:08
jklareanteaya do you know which interface is used to connect to the jenkins master? eth0 or eth1 ?08:08
*** pcaruana has joined #openstack-infra08:09
openstackgerritMerged openstack-infra/project-config: Added new projects for the OSA role break out  https://review.openstack.org/28451208:09
anteayaI do not know08:09
*** xiangxinyong has quit IRC08:10
*** yamahata has quit IRC08:11
*** kzaitsev_mb has quit IRC08:12
AJaegerjklare: you might have more success on #openstack-qa - and/or later ;(08:12
*** aviau has quit IRC08:13
*** aviau has joined #openstack-infra08:15
*** thorst has joined #openstack-infra08:16
*** [HeOS] has quit IRC08:17
*** amotoki has quit IRC08:18
*** HeOS has joined #openstack-infra08:18
*** vgridnev has quit IRC08:19
*** amotoki has joined #openstack-infra08:19
*** andymaier has joined #openstack-infra08:19
openstackgerritJan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty  https://review.openstack.org/28616108:20
*** sshnaidm has quit IRC08:22
*** thorst has quit IRC08:23
*** flepied has joined #openstack-infra08:24
*** hichihara has quit IRC08:24
openstackgerritMerged openstack-infra/project-config: Add bareon-ironic project  https://review.openstack.org/28130108:24
*** achanda has quit IRC08:27
yolandaAJaeger, taking a look08:28
yolandaah it was aborted already08:29
*** ifarkas has joined #openstack-infra08:30
*** vincentll has joined #openstack-infra08:31
*** ociuhandu has quit IRC08:31
*** jlanoux has joined #openstack-infra08:32
*** kushal has quit IRC08:33
*** watanabe_isao has quit IRC08:34
AJaegeryolanda: yeah, jklare "fixed" it - thanks08:34
*** dizquierdo has joined #openstack-infra08:36
*** daemontool__ has joined #openstack-infra08:38
*** arxcruz has joined #openstack-infra08:39
*** _nadya_ has joined #openstack-infra08:39
*** yaume has joined #openstack-infra08:40
*** roxanaghe has joined #openstack-infra08:41
*** daemontool_ has quit IRC08:41
*** matrohon has joined #openstack-infra08:45
*** roxanaghe has quit IRC08:46
*** salv-orl_ has quit IRC08:47
*** openstackgerrit has quit IRC08:48
*** openstackgerrit has joined #openstack-infra08:48
trashmordred, SpamapS: Can you please review https://review.openstack.org/#/c/280178 again?08:48
*** salv-orlando has joined #openstack-infra08:48
*** dingyichen has quit IRC08:50
*** zeih has joined #openstack-infra08:51
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Add pylint, coverage, py34 to sahara-tests  https://review.openstack.org/28469308:55
*** fedexo has quit IRC08:58
abregmanhi. where can I find the images the gates use?08:59
*** andymaier has quit IRC09:01
*** _degorenko|afk is now known as degorenko09:02
*** arxcruz has quit IRC09:04
*** arxcruz has joined #openstack-infra09:06
*** kzaitsev_mb has joined #openstack-infra09:07
*** ildikov has quit IRC09:08
rcarrillocruzabregman: check it out projects.yaml on openstack-infra/project-config09:08
openstackgerritJan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty  https://review.openstack.org/28616109:09
*** asettle has joined #openstack-infra09:10
abregmanrcarrillocruz: what exactly am I looking for?: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml09:10
*** jcoufal has joined #openstack-infra09:10
rcarrillocruzabregman: what test are you looking for09:11
*** salv-orlando has quit IRC09:11
rcarrillocruzthe 'node' tells you the image used09:11
rcarrillocruzbare-trusty => bare ubuntu trusty09:11
rcarrillocruzbare-precise => bare ubuntu precise09:11
rcarrillocruzso on and so forth09:11
abregmanrcarrillocruz: I want to download the image this gate job used -> http://logs.openstack.org/74/286074/1/check/gate-neutron-dsvm-api/9c25e1d/console.html09:11
rcarrillocruzdsvm stand for 'devstack vm'09:12
abregmanyeah I know09:12
rcarrillocruzin your case09:12
rcarrillocruz2016-02-29 18:04:55.236 | Building remotely on devstack-trusty-ovh-bhs1-8379741 (devstack-trusty) in workspace /home/jenkins/workspace/gate-neutron-dsvm-api09:12
*** kzaitsev_mb has quit IRC09:12
*** ihrachys has joined #openstack-infra09:13
abregmanrcarrillocruz: where the 'devstack-trusty' image is stored? is it this?: https://cloud-images.ubuntu.com/trusty/09:13
*** hashar has joined #openstack-infra09:13
abregmanrcarrillocruz: where can I download the exact image the job is using?09:14
*** amotoki has quit IRC09:14
*** jistr has joined #openstack-infra09:14
*** scheuran has quit IRC09:15
*** andymaier has joined #openstack-infra09:18
rcarrillocruzyou can't download, they are built by the infra team09:18
rcarrillocruzcheck it out ^ docs to build your own09:18
*** mattt has joined #openstack-infra09:19
*** amotoki has joined #openstack-infra09:19
*** sshnaidm has joined #openstack-infra09:19
matttquite likely i'm being thick ... but is there a clear way to see a dependency tree for dependent reviews in gerrit?09:19
matttthis used to be a lot more obvious in the old interface09:19
*** thorst has joined #openstack-infra09:22
markus_zmtreinish: You created the "cat-pip.txt" file, so this might be interesting for you https://bugs.launchpad.net/devstack/+bug/154254509:23
openstackLaunchpad bug 1542545 in devstack "devstack is broken" [Undecided,Confirmed]09:23
markus_zmtreinish: I'm wondering why the gate doesn't complain09:23
markus_zdtroyer: ^ It looks like you reviewed that file too (http://git.openstack.org/cgit/openstack-dev/devstack/commit/tools/cap-pip.txt?id=75446deea06107fa63a7f08990f0de26e5761833)09:26
openstackgerritFausto Marzi proposed openstack/requirements: Add modules for freezer Mitaka release  https://review.openstack.org/27107209:27
*** scheuran has joined #openstack-infra09:27
*** chenli has quit IRC09:28
*** e0ne has joined #openstack-infra09:28
*** ikalnitsky has joined #openstack-infra09:28
*** thorst has quit IRC09:28
*** keedya has quit IRC09:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list  https://review.openstack.org/28606209:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043109:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of brctl show from hosting node  https://review.openstack.org/28627909:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job  https://review.openstack.org/27342409:29
*** sbelous_ has joined #openstack-infra09:32
*** dtantsur|afk is now known as dtantsur09:33
*** ildikov has joined #openstack-infra09:35
*** ociuhandu has joined #openstack-infra09:35
*** korzen_ is now known as korzen09:37
*** e0ne has quit IRC09:38
*** derekh has joined #openstack-infra09:39
AJaegermattt: it's still there in the upper right corner09:40
AJaegermattt: if you prefer cli, use gertty, it can show dependencies nicely09:40
matttAJaeger: yeah i see that, but at least for me there's no clear indication what the relationship is09:40
AJaegermattt: which change?09:40
*** roxanaghe has joined #openstack-infra09:42
AJaegermattt: if it says "RElated chagnes (2)", then the lower item is bottom of stack and the one above is top of stack09:43
openstackgerritJan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty  https://review.openstack.org/28616109:43
AJaegerhttps://review.openstack.org/#/c/286242/ - dvipng change is on top of pandoc one09:43
matttAJaeger: ok, so it's the ordering that is key ... the reviews i was looking at were dependent on one-another i think so it was super confusing what was what09:44
matttAJaeger: thanks for clearing up!  and i really should look at gerrty09:45
*** dtardivel has joined #openstack-infra09:45
*** vgridnev has joined #openstack-infra09:46
*** roxanaghe has quit IRC09:46
lucasagomeshi all, I've this small patch in devstack gate fixing an hardcoded assumption that is preventing the ipmitool jobs in gate to run https://review.openstack.org/#/c/284036/ , if you have some time mind taking a look at it ? thank you09:46
*** zhurong has quit IRC09:47
AJaegermattt: ;)09:47
AJaegermattt: yes, ordering is key. They're stacked on top of each other09:48
openstackgerritOlivier Lemasle proposed openstack-infra/jeepyb: Update projects on GitHub  https://review.openstack.org/27717509:48
*** aarefiev has quit IRC09:48
*** aarefiev has joined #openstack-infra09:49
*** vgridnev has quit IRC09:54
*** sorantis has joined #openstack-infra09:56
*** jordanP has joined #openstack-infra09:59
*** ociuhandu has quit IRC09:59
*** vgridnev has joined #openstack-infra10:00
*** gnuoy_ has joined #openstack-infra10:03
*** gnuoy_ has quit IRC10:08
*** rguillebert has joined #openstack-infra10:09
openstackgerritJens proposed openstack-infra/git-review: Make it possible to configure draft as default push mode  https://review.openstack.org/22042610:09
*** e0ne has joined #openstack-infra10:11
dtantsurmorning folks! is gerrit terribly slow for everyone or just me?10:13
*** jcooley_ has joined #openstack-infra10:15
*** fabio_ has joined #openstack-infra10:19
openstackgerritJan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty  https://review.openstack.org/28616110:20
*** andymaier has quit IRC10:25
*** sambetts|afk is now known as sambetts10:26
*** ildikov has quit IRC10:26
*** thorst has joined #openstack-infra10:26
jamespagemorning (again) - please can I be added to the charms-core and charms-release groups created under https://review.openstack.org/#/c/232705/10:30
*** Qiming has quit IRC10:30
yolandahi jamespage10:30
yolandai can do it10:30
jamespageyolanda, thanks muchly10:30
*** lucasagomes has quit IRC10:30
openstackgerritFrancesco Longo proposed openstack-infra/project-config: Added IoTronic project.  https://review.openstack.org/28611310:30
yolandagerrit is super slow for me today10:31
jamespageyolanda, I'm assuming that will bootstrap me to add other members to both of those groups?10:31
yolandajamespage yes, i will add you and you can other people10:31
jamespageyolanda, +1 awesome10:31
dtantsuryolanda, gerrit is crazily slow, yeah10:31
*** lucasagomes has joined #openstack-infra10:31
dtantsurand it becomes slower and slower10:31
yolandadtantsur, going to check, it's not usable for me now10:31
*** lucasagomes has quit IRC10:32
*** lucasagomes has joined #openstack-infra10:32
*** thorst has quit IRC10:33
yolanda#status alert Gerrit is going to be restarted due to poor performance10:33
openstackstatusyolanda: sending alert10:33
-openstackstatus- NOTICE: Gerrit is going to be restarted due to poor performance10:36
*** ChanServ changes topic to "Gerrit is going to be restarted due to poor performance"10:36
*** daemontool has joined #openstack-infra10:36
*** lucasagomes has quit IRC10:37
*** lucasagomes has joined #openstack-infra10:37
yolandajamespage, added10:38
ikalnitskyHey folks! I'm a core in fuel-plugins repo. Recently I landed the patch that should help me to publish releases to PyPI https://review.openstack.org/#/c/283683/ However, yesterday I pushed a new tag but PyPI sdist wasn't uploaded. openstackci user is added on PyPI as owner. Could someone help me to understand what's wrong?10:38
yolandadtantsur, gerrit was already restarted, but statusbot looks slow on announcing it10:38
openstackstatusyolanda: finished sending alert10:39
yolanda#status ok gerrit finished restartign10:40
openstackstatusyolanda: sending ok10:40
*** ildikov has joined #openstack-infra10:40
*** daemontool__ has quit IRC10:40
*** lucasagomes has quit IRC10:41
jamespageyolanda, thankyou10:42
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"10:42
-openstackstatus- NOTICE: gerrit finished restartign10:42
* yolanda is happy to see charms in openstack10:42
*** roxanaghe has joined #openstack-infra10:42
*** lucasagomes has joined #openstack-infra10:43
openstackstatusyolanda: finished sending ok10:45
AJaegerikalnitsky: check the log files to see what the error is10:46
AJaegerikalnitsky: which version did you push?10:46
ikalnitskyAJaeger: 4.0.010:46
*** kzaitsev_mb has joined #openstack-infra10:47
ikalnitskyAJaeger: how to check logs? I've sent 4.0.0 tag to gerrit and that's it.10:47
*** roxanaghe has quit IRC10:47
*** ihrachys has quit IRC10:48
*** exploreshaifali has joined #openstack-infra10:49
AJaegerikalnitsky: git show-ref  4.0.010:49
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-cfgdb'  https://review.openstack.org/28613710:49
AJaegerAnd then look on logs.openstack.org for it10:49
AJaegerIt's 6abd3371d870cc5e90ce72aa8cf6103b641f0e42 -> so, look for 6a/6abd3371d870cc5e90ce72aa8cf6103b641f0e4210:50
ikalnitskyAJaeger: didn't know about that! let me try. thanks!10:50
AJaegerjamespage: seems you have no setup.cfg file. ARe you using pbr?10:52
jamespageAJaeger, not yet10:52
jamespageis that causing issues?10:52
AJaegerjamespage: sorry, I meant ikalnitsky and somehow did not notice that wrong tab completion10:53
ikalnitskyAJaeger: nope, no pbr10:53
AJaegerikalnitsky: that's your problem, AFAIK we require pbr for it10:54
ikalnitskyAJaeger: yeah, i found that your job relies on setup.cfg10:54
ikalnitskyAJaeger: well, I'll move to pbr then. Thanks :)10:54
AJaegerand then release a 4.0.1 ;)10:55
ikalnitskyAJaeger: however, I'd recommend you guys to use `python setup.py --name` to retrieve project name :)10:55
ikalnitskyAJaeger: besides, we have obsolete branches. any chance to remove them? or they will be there forever?10:55
AJaegerikalnitsky: didn't I ask you during review whether you want all of them? That's my standard question...10:56
AJaegerikalnitsky: they can only be removed manually and that's expensive - it should have done before the import ;(10:56
*** pfallenop has joined #openstack-infra10:56
AJaegeryou need to bribe ;) one of the admins to do it while there'S no fire ongoing10:57
openstackgerritJan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty  https://review.openstack.org/28616110:57
ikalnitskyAJaeger: heh.. any contacts? or, wait,  don't you have such permissions?  :)10:58
*** bexelbie has quit IRC10:58
*** bexelbie has joined #openstack-infra10:58
*** oanson has joined #openstack-infra10:58
AJaegerikalnitsky: I don't have such permissions - come back during US time, please10:59
*** rvba` is now known as rvba10:59
openstackgerritTom Barron proposed openstack-infra/project-config: Skip dsvm jobs on manila docs/reno/unit test changes  https://review.openstack.org/28649711:00
ikalnitskyAJaeger: hehe. ok :) thanks a lot for your help!11:00
*** chlong_ has joined #openstack-infra11:02
*** pfallenop has quit IRC11:03
abregmanrcarrillocruz: thanks for the help!11:03
*** dims has joined #openstack-infra11:03
openstackgerritMerged openstack-infra/tripleo-ci: Source undercloud environment variable from a file  https://review.openstack.org/27566711:05
openstackgerritMerged openstack-infra/tripleo-ci: Split the deploy script into its own file  https://review.openstack.org/27566811:06
*** amotoki has quit IRC11:06
*** fhubik has joined #openstack-infra11:11
openstackgerritSergey Lukjanov proposed openstack/requirements: Bump to final Mitaka python-saharaclient  https://review.openstack.org/28650311:12
openstackgerritTom Barron proposed openstack-infra/project-config: Skip dsvm jobs on manila docs/reno/unit test changes  https://review.openstack.org/28649711:13
tbarronvponomaryov: thanks, that woke me up11:14
tbarronvponomaryov: patch set #2 is up, as you specified11:14
*** yamamoto has quit IRC11:15
*** |-paul-| has joined #openstack-infra11:15
*** pfallenop has joined #openstack-infra11:16
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Archive all of the delorean logs  https://review.openstack.org/27141611:16
*** xiangxinyong has joined #openstack-infra11:17
*** claudiub has joined #openstack-infra11:25
*** jlanoux_ has joined #openstack-infra11:30
abregmanis there a way to connect the gate job machine?11:30
openstackgerritFausto Marzi proposed openstack/requirements: Add Freezer modules and projects for Mitaka release  https://review.openstack.org/27107211:30
*** jlanoux has quit IRC11:30
*** thorst has joined #openstack-infra11:31
*** ldnunes has joined #openstack-infra11:32
*** rossella_s has quit IRC11:32
*** rossella_s has joined #openstack-infra11:33
*** ildikov has quit IRC11:36
*** Qiming has joined #openstack-infra11:36
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043111:36
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list  https://review.openstack.org/28606211:36
*** gildub has quit IRC11:37
*** [1]Thelo has joined #openstack-infra11:37
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job  https://review.openstack.org/27342411:37
*** _amrith_ is now known as amrith11:38
*** thorst has quit IRC11:38
*** aysyd has joined #openstack-infra11:39
*** ihrachys has joined #openstack-infra11:39
*** Thelo has quit IRC11:40
*** [1]Thelo is now known as Thelo11:40
*** esker has quit IRC11:40
*** ociuhandu has joined #openstack-infra11:40
*** esker has joined #openstack-infra11:40
*** salv-orlando has joined #openstack-infra11:41
*** tpsilva has joined #openstack-infra11:44
*** dizquierdo is now known as dizquierdo_afk11:44
*** aysyd has quit IRC11:46
*** aysyd has joined #openstack-infra11:46
*** sam_wan has quit IRC11:47
*** jlanoux_ has quit IRC11:50
*** Jeffrey4l has quit IRC11:51
*** sdague has joined #openstack-infra11:51
*** jlanoux has joined #openstack-infra11:51
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Skip dsvm jobs on manila docs/reno/unit test changes  https://review.openstack.org/28649711:54
*** sarob has joined #openstack-infra11:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Sort projects in big skip list  https://review.openstack.org/28652711:57
*** salv-orlando has quit IRC11:58
*** amotoki has joined #openstack-infra11:58
*** sorantis has quit IRC11:59
*** sarob has quit IRC12:00
*** yamamoto has joined #openstack-infra12:02
AJaegersdague: could you review https://review.openstack.org/285949 https://review.openstack.org/#/c/285487/ and https://review.openstack.org/#/c/285148/ , please? Those are project-config changes that remove jobs12:04
*** fhubik is now known as fhubik_brb12:06
*** daemontool_ has joined #openstack-infra12:07
*** rfolco_ has joined #openstack-infra12:07
*** amotoki has quit IRC12:07
*** Jeffrey4l has joined #openstack-infra12:09
*** jpr has joined #openstack-infra12:10
*** daemontool has quit IRC12:10
*** dizquierdo_afk is now known as dizquierdo12:11
openstackgerritMerged openstack-infra/project-config: Cleanup specs repos setup  https://review.openstack.org/28288912:13
*** fhubik_brb is now known as fhubik12:13
*** dtardivel has quit IRC12:28
*** exploreshaifali has quit IRC12:29
*** |-paul-| has quit IRC12:29
*** andymaier has joined #openstack-infra12:30
*** amrith is now known as _amrith_12:34
*** kushal has joined #openstack-infra12:34
*** daemontool__ has joined #openstack-infra12:34
*** jaosorior has quit IRC12:36
*** jaosorior has joined #openstack-infra12:37
*** daemontool_ has quit IRC12:37
*** ildikov has joined #openstack-infra12:38
*** gordc has joined #openstack-infra12:40
*** rhallisey has joined #openstack-infra12:43
*** roxanaghe has joined #openstack-infra12:44
*** thorst has joined #openstack-infra12:46
*** shardy has quit IRC12:47
*** thorst_ has joined #openstack-infra12:47
*** roxanaghe has quit IRC12:48
*** fhubik is now known as fhubik_brb12:48
*** jaypipes has joined #openstack-infra12:48
openstackgerritBeth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard  https://review.openstack.org/28619412:49
*** fhubik_brb is now known as fhubik12:49
*** thorst has quit IRC12:51
*** amotoki has joined #openstack-infra12:52
*** baoli has joined #openstack-infra12:54
*** salv-orlando has joined #openstack-infra12:55
*** andymaier has quit IRC12:57
openstackgerritMarton Kiss proposed openstack-infra/groups: Add a standalone map page  https://review.openstack.org/28594312:59
*** dims has quit IRC13:02
*** fabio_ has quit IRC13:06
*** fabio_ has joined #openstack-infra13:06
*** abregman is now known as abregman|brb13:07
*** links has quit IRC13:07
*** dims has joined #openstack-infra13:08
*** rlandy has joined #openstack-infra13:10
*** julim has joined #openstack-infra13:10
*** yamamoto has quit IRC13:10
*** yamamoto has joined #openstack-infra13:11
*** yamamoto has quit IRC13:11
*** yamamoto has joined #openstack-infra13:12
*** cloudtrainme has joined #openstack-infra13:12
*** exploreshaifali has joined #openstack-infra13:12
odyssey4mehi everyone - it seems we have a mismatch between git.o.o and github - http://git.openstack.org/cgit/openstack/openstack-ansible-os_horizon is there, but not in github - how do we resolev this?13:14
*** pradk has joined #openstack-infra13:14
*** Aegil has quit IRC13:15
*** lucasagomes is now known as lucas-hungry13:15
*** sdake has joined #openstack-infra13:16
AJaegerodyssey4me: keep in mind that github is only a mirror, git.openstack.org is the master.13:16
AJaegerodyssey4me: you should be able to develop with this setup13:16
odyssey4meAJaeger sure, I'm just trying to understand how the mirror process missed a whole repo13:16
AJaegerodyssey4me: still, it's not what we intend currently, so needs manual fixing13:16
AJaegerodyssey4me: let'S ask yolanda whether she can manually fix this13:17
*** zeih has quit IRC13:17
AJaegerodyssey4me: github rate limits calls and that might screw up setting up a repo...13:17
*** zeih has joined #openstack-infra13:18
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Add a third scenario for packstack integration testing  https://review.openstack.org/28657913:19
AJaegerodyssey4me: An admin needs to fix this13:19
*** Daisy has joined #openstack-infra13:19
*** andymaier has joined #openstack-infra13:20
*** sdake has quit IRC13:21
*** sdake has joined #openstack-infra13:21
*** grue_pm has quit IRC13:22
*** kgiusti has joined #openstack-infra13:22
yolandahi, back from launch13:25
*** dprince has joined #openstack-infra13:25
yolandaso github not replicating again?13:25
AJaegeryolanda: permission problem again?13:25
yolandai guess, going to fix13:25
AJaegerhttps://github.com/openstack/openstack-ansible-os_horizon does not exist at all ;(13:26
*** esikachev has joined #openstack-infra13:26
yolandathat's even worse13:26
AJaegerodyssey4me: are all other repos there?13:27
openstackgerritMerged openstack-infra/release-tools: better handling of stable flag  https://review.openstack.org/28552013:27
odyssey4meAJaeger we're working through the various repositories to check which came through and which didn't and will report back.13:27
yolandagoing to check manage-projects output13:28
AJaegerthanks, yolanda and odyssey4me13:28
openstackgerritAkihiro Motoki proposed openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0  https://review.openstack.org/28658713:29
yolanda manage_projects - ERROR - Problems creating openstack/openstack-ansible-os_horizon, moving on.13:29
yolandaTraceback (most recent call last):13:29
yolandagoing to try that manually13:29
odyssey4methanks AJaeger & yolanda13:30
*** zeih has quit IRC13:30
*** zeih has joined #openstack-infra13:31
*** yamamoto has quit IRC13:32
*** slagle has joined #openstack-infra13:33
*** cloudtrainme has quit IRC13:34
yolandaodyssey4me, AJaeger, fixed13:37
yolandalet me know if there are more repos to be fixed13:37
odyssey4methanks yolanda !13:37
daemontool__AJaeger, at your convenience: https://review.openstack.org/#/c/271072/13:37
AJaegerdaemontool__: please explain what that is...13:38
*** yamamoto has joined #openstack-infra13:38
daemontool__AJaeger,  is about adding modules to openstack/requirements13:38
daemontool__and adding the freezer* to projects.txt on that same repo13:38
*** yamamoto has quit IRC13:39
AJaegerdaemontool__: I'm not a core for requirements, can't help with that.13:39
*** baoli has quit IRC13:40
daemontool__AJaeger,  ok ty13:40
*** baoli has joined #openstack-infra13:40
*** fhubik is now known as fhubik_brb13:42
*** fhubik_brb is now known as fhubik13:42
*** fhubik is now known as fhubik_brb13:42
*** baoli has quit IRC13:43
*** jtomasek_ has joined #openstack-infra13:43
*** baoli has joined #openstack-infra13:43
AJaegerfungi, infra-root: The openSSL DROWN has been published, see https://drownattack.com/13:44
AJaegerCodename "DROWN" for "Decrypting RSA using Obsolete and Weakened eNcryption"13:44
AJaeger" server is vulnerable to DROWN if: It allows SSLv2 connections OR Its private key is used on any other server that allows SSLv2 connections"13:45
AJaegerDo we use SSLv2 anywhere in the OpenStack infrastructure?13:46
mordredAJaeger: I believe we shut it off a while ago13:46
AJaegermordred: I grepped around and hope so as well ;)13:46
*** dizquierdo has quit IRC13:47
AJaegerBut I didn't check each and every place13:47
mordredAJaeger: yes13:47
mordredAJaeger: http://codesearch.openstack.org/?q=SSLProtocol&i=nope&files=&repos=13:48
*** claudiub|2 has joined #openstack-infra13:48
mordredAJaeger: not only is infra good, most of openstack deployment is good13:48
mordredwith one exceptoin - but that's because of something that's parameterized13:48
mordredand doesn't set it automatically13:49
*** dkranz has joined #openstack-infra13:49
*** shardy has joined #openstack-infra13:50
*** sdake_ has joined #openstack-infra13:50
AJaegermordred: check http://codesearch.openstack.org/?q=SSLEngine&i=nope&files=&repos= - seems that the newly imported charm is not secure13:50
AJaegerjamespage: http://git.openstack.org/cgit/openstack/charm-heat/tree/hooks/charmhelpers/contrib/openstack/templates/openstack_https_frontend13:50
*** claudiub has quit IRC13:51
mordredAJaeger: wow. am I about to write a patch to a juju charm?13:51
jamespageoh nice13:52
mordredjamespage: ooh, maybe you want to :)13:52
jamespagemordred, let me deal with that - we still have a sync-y thing to deal with13:52
mordredbtw - the openstack security team has a great doc:13:52
AJaegerand compass as well: http://git.openstack.org/cgit/openstack/compass-web/tree/v2/dboards/sample/apache_ldap.conf13:52
mordredwith some specific config lines13:52
*** yamamoto has joined #openstack-infra13:53
*** maishsk_ has joined #openstack-infra13:53
*** maishsk has quit IRC13:53
*** maishsk_ is now known as maishsk13:53
*** jtomasek_ has quit IRC13:54
*** sdake has quit IRC13:54
AJaegermordred: is this one fine? http://git.openstack.org/cgit/openstack-infra/puppet-httpd/tree/templates/vhost-proxy.conf.erb13:54
*** otsuka has quit IRC13:54
AJaegerIt does not set ciphers...13:55
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add auto-discovery test job to ironic-inspector  https://review.openstack.org/27784313:56
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Create new partitioning element.  https://review.openstack.org/25988114:00
*** bgaifullin has quit IRC14:01
mordredAJaeger: well, I don't think we use that template anywhere14:01
*** hichihara has joined #openstack-infra14:02
*** annegentle has joined #openstack-infra14:02
AJaegerAnd this one: http://git.openstack.org/cgit/openstack-infra/puppet-storyboard/tree/templates/storyboard_https.vhost.erb ?14:02
*** max_lobur has left #openstack-infra14:02
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list  https://review.openstack.org/28606214:02
openstackgerritMonty Taylor proposed openstack-infra/puppet-httpd: Add SSL Procotol and Cipher config to default vhost  https://review.openstack.org/28661014:04
*** derekh is now known as ndipanov_14:04
mordredAJaeger: yah - let's update storyboard14:04
*** ndipanov_ is now known as derekh14:04
AJaegerthanks, mordred14:04
openstackgerritMonty Taylor proposed openstack-infra/puppet-storyboard: Update SSLProtocol and SSLCipherSuite  https://review.openstack.org/28661214:06
yolandamordred, cool . Going to review14:06
*** amitgandhinz has joined #openstack-infra14:07
yolandamordred, i was expecting to see SSLProtocol ALL -SSLv2 -SSLv314:08
yolandais the same as your change or do we miss some protocol?14:08
*** baoli has quit IRC14:08
mordredyolanda: yah - I used the line from http://git.openstack.org/cgit/openstack/security-doc/tree/security-guide/source/secure-communication/tls-proxies-and-http-services.rst#n25414:08
*** abregman|brb is now known as abregman14:09
yolandadon't we need SSLHonorCipherOrder On ?14:09
*** lucas-hungry is now known as lucasagomes14:09
mordredyolanda: maybe? do we have that set other places?14:10
yolandai was reading about recommendations14:10
yolandabut haven't checked other places14:10
AJaegerIt's not set in security-doc either14:11
*** berendt has joined #openstack-infra14:11
AJaegerhttps://review.openstack.org/286616 fixes salt-formula-horizon14:11
mordredif it is needed, we should send a patch into the security guide too14:12
*** baoli has joined #openstack-infra14:12
yolandait may be a nice to have, but not really needed. Is to ensure that the first match of the cipher list is always used14:12
mordredof course, perhaps what we should really do is remove TLS 1.0 from the list14:12
mordredSSLHonorCipherOrder is only needed with SSLv3 and TLS 1.014:13
berendthi. can a core reviewer of system-config please have a look at https://review.openstack.org/#/c/284297/ and give a +2A. it is a request for a openstack-de mailinglist. I have a user group meetup tomorrow and want to be able to announce the availability of the list.14:13
yolandamordred and yes, tls v1.0 is not recommended14:14
*** thiagop has joined #openstack-infra14:14
mordreddisabling tls v1.0 apparently means we'd be dropping support for IE7-1014:14
*** vgridnev has quit IRC14:14
*** ociuhandu has quit IRC14:15
*** baoli has quit IRC14:15
*** jpr has quit IRC14:15
yolandawe should? :)14:15
mordredwell, let's wait for fungi on that one14:15
*** claudiub has joined #openstack-infra14:15
*** ociuhandu has joined #openstack-infra14:16
yolandamordred, can you take  a look at https://review.openstack.org/285433 ? the glean hostname fix, it was not passing tests14:17
*** Daisy has quit IRC14:17
*** claudiub|2 has quit IRC14:18
*** berendt has left #openstack-infra14:18
*** sdake_ is now known as sdake14:19
mordredyolanda: so - jeblair gave me a good argument as to why we should perhaps use instead of the actual ip of the server14:19
yolandamordred, what should be the reason?14:19
mordredyolanda: which is that if the ip changes, the etc hosts entry will not work anymore14:19
mordredyolanda: but with it will14:20
*** vgridnev has joined #openstack-infra14:20
yolandamordred, also in the other hand, setting to can give another problems, such as the one we had in rabbitmq and binding address14:20
mordredso if we have glean on a vm, and it runs once at vm boot, and sets up the network interfaces to use dhcp14:20
mordredand then the cloud changes the ip of the server14:21
mordredwe could have an issue14:21
mordredthe rabbit thing is a good point though14:21
yolandawe had controller pointing to and rabbitmq with a binding address to controller00.xx. As a consecuence, any of the computes could not reach rabbit because it was limited to and we had to open for all14:21
yolandaalso looking at documentation, if the server has a fixed ip, is the recommended way to go14:21
*** ociuhandu has quit IRC14:21
mordredah - SO14:22
mordredmaybe we do this14:22
yolandaif the ip changes, we can run glean again14:22
mordredbecause we have two sets of glean users14:22
*** baoli has joined #openstack-infra14:22
openstackgerritMerged openstack-infra/tripleo-ci: Convert the container job to a noop  https://review.openstack.org/28532514:22
*** jsavak has joined #openstack-infra14:22
mordredif we're setting the server up for static IPs (like with bifrost) - that means glean will know the IP and will be writing it to the network config14:22
mordredand thus we shoudl write an IP to the hosts file14:23
*** baoli has quit IRC14:23
*** tiswanso has joined #openstack-infra14:23
mordredbut, if glean does not write static network config and instead is doing dhcp - then I think we should do
yolandawell we rely on info present in metadata14:23
yolandaso if nothing is on network, we default to
mordredso that logic I just said is already there :)14:23
* mordred re-reads the patch14:24
yolandaah yes, i implemented that logic14:24
mordredduh. yes. I see that now14:24
mordredthanks yolanda14:24
*** woodster_ has joined #openstack-infra14:24
sdaguethis seems odd - http://logs.openstack.org/43/281143/16/check/gate-nova-tox-functional/70d60c9/console.html#_2016-02-29_17_35_21_26314:24
yolandathe thing that i don't trust to much, is picking the first interface found. for my tests it has been fine but i don't know all use cases14:24
*** vgridnev has quit IRC14:25
mordredyolanda: looks great14:25
mordredyolanda: I think first interface found for now is probably fine14:25
yolandaand there was that nice difference betwen python 2.7 and python 3 versions. I had to do a sorted, because python2.7 was not returning the same item than python 314:25
*** Daisy has joined #openstack-infra14:25
yolandaso the tests were crazy14:25
*** Daisy has quit IRC14:25
*** mrmartin has quit IRC14:25
mordredsdague: I agree14:25
*** baoli has joined #openstack-infra14:25
lucasagomesclarkb, sdague hi, if you guys have some time today, mind taking a look at https://review.openstack.org/#/c/284036/ ? This will allow we test ipmitool drivers in the ironic gate (which is our reference driver, but it wasn't tested before)14:26
mordredyolanda: :)14:26
*** Daisy has joined #openstack-infra14:26
lucasagomesso we would like to get it tested soon and make the jobs voting as soon as they establish14:26
lucasagomesthank you14:26
*** vgridnev has joined #openstack-infra14:26
*** aysyd has quit IRC14:27
*** aysyd has joined #openstack-infra14:28
*** _amrith_ is now known as amrith14:29
*** kzaitsev_mb has quit IRC14:29
*** jroll has quit IRC14:29
*** jroll has joined #openstack-infra14:30
*** jroll has quit IRC14:30
*** jroll has joined #openstack-infra14:30
pabelangerso, we did get a spammer overnight14:30
*** kzaitsev_mb has joined #openstack-infra14:30
*** sfinucan has joined #openstack-infra14:30
pabelangermaybe 4 of them14:30
mordredpabelanger: that's much better!14:30
pabelangerthey switched to uploading png files :)14:31
pabelangermordred: indeed14:31
*** andymaier has quit IRC14:31
*** edmondsw has joined #openstack-infra14:31
pabelangerShould spend some time looking at how https://www.mediawiki.org/wiki/Extension:TitleBlacklist works14:31
pabelangerso we can blacklist the phonenumbers they are using14:32
*** shardy has quit IRC14:33
*** eharney has joined #openstack-infra14:33
*** ociuhandu has joined #openstack-infra14:34
*** shardy has joined #openstack-infra14:35
*** sshnaidm has quit IRC14:35
*** weshay has joined #openstack-infra14:36
*** rbrndt has joined #openstack-infra14:36
*** C_W has joined #openstack-infra14:37
*** baoli has quit IRC14:39
*** ociuhandu has quit IRC14:39
*** d0ugal has quit IRC14:39
*** d0ugal has joined #openstack-infra14:40
*** d0ugal has quit IRC14:40
*** _ody has quit IRC14:40
yolandamordred, thx for review, let's see if we can pick another +2 today14:40
yolandathat could be causing problems to glean users14:40
*** d0ugal has joined #openstack-infra14:41
*** baoli has joined #openstack-infra14:41
*** Daisy has quit IRC14:44
fungimordred: yeah, i'm reviewing the advisory now14:45
mordredfungi: cool. when you get done, there are two outstanding questions14:46
mordredfungi: a) should we go ahead and disable tls v1.0 (if not, should we set SSLHonorCipherOrder)14:46
fungiAJaeger: thanks for the heads up14:46
mordredfungi: b) should we update our SSLProtocol and SSLCipherSuite settings in our files to match the recommendations made in the security guide14:47
*** ociuhandu has joined #openstack-infra14:47
mtreinishmordred:, fungi: if you get a sec can you restart the subunit gearman worker the necessary patches landed14:47
mtreinishhopefully it'll stay up for more than 1 test result now14:47
*** xyang1 has joined #openstack-infra14:48
*** sdake has quit IRC14:49
fungimordred: where did the suggestion to disable tls 1.0 come from?14:50
fungiand yeah, i vaguely recall standardizing all our configs in infra to drop pre-tls protocol versions already but the double-check was great14:50
fungihaving a look at the security guide recommendations now14:51
mordredfungi: well - tls 1.0 came from looking at the SSLHonorCipherOrder setting - which is only needed for tls 1.0 and ssl v314:52
*** alivigni has joined #openstack-infra14:52
mordredfungi: but then that led me to check in to tls 1.0 - and it's being dropped from PCI compliance as of June of this year - so it's on its way out of general use14:52
mordredfungi: I don't think that one is urgent, but thought I'd mention it14:52
mordred(not that we need to be PCI compliant, mind you, but if _they14:53
mordred_they're_ dropping it ...)14:53
fungiyeah, i'm not opposed, though it would be interesting to find out what platforms/browsers we're dropping compatibility with when doing that14:53
*** daemontool_ has joined #openstack-infra14:53
bkeroI'm sure there are some selenium services that can tell you14:54
fungiyeah, there's plenty of documentation out there too14:55
*** doug-fish has joined #openstack-infra14:55
fungiwe can just do it and then point ssllabs.com at the site we're testing, for example14:55
*** zeih has quit IRC14:55
*** _ody has joined #openstack-infra14:56
*** sdake has joined #openstack-infra14:56
*** daemontool__ has quit IRC14:56
*** e0ne has quit IRC14:56
*** edmondsw has quit IRC14:57
fungimordred: so 286610 was the only necessary change you spotted?14:57
mordredfungi: yah14:58
mordredfungi: we'd be dropping support for IE 7-1014:58
*** sorantis has joined #openstack-infra14:59
*** C_W has quit IRC14:59
*** shardy has quit IRC15:00
bkeroThere are these handy guidelines too: https://wiki.mozilla.org/Security/Guidelines/Web_Security15:01
fungii scanned security.o.o for example and this report lists the browsers which are relying on tls 1.0 to use it https://www.ssllabs.com/ssltest/analyze.html?d=security.openstack.org&s=2001%3a4800%3a7813%3a516%3a3bc3%3ad7f6%3aff05%3a488215:01
bkero(also with config generator)++15:01
*** claudiub|2 has joined #openstack-infra15:01
fungisafari on osx 10.8 for example15:02
*** fawadkhaliq has joined #openstack-infra15:02
openstackgerritBeth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard  https://review.openstack.org/28619415:02
*** claudiub has quit IRC15:04
AJaegersdague: that looks indeed strange ;(15:04
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quota support  https://review.openstack.org/28511015:04
*** Daisy has joined #openstack-infra15:04
fungilooks like SSLHonorCipherOrder is an i-know-better-than-you option, where you can make the server force its preferences on the client rather than letting the client choose its preferred order15:05
*** dizquierdo has joined #openstack-infra15:06
fungii'm not especially opposed, but it's trading default secure configuration assumptions for one party over the other15:06
*** exploreshaifali has quit IRC15:07
*** bgaifullin has joined #openstack-infra15:07
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Make ceph jobs non-voting until bug 1551305 is fixed  https://review.openstack.org/28664215:07
openstackbug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed] https://launchpad.net/bugs/155130515:07
*** yamahata has joined #openstack-infra15:07
mriedemsdague: dansmith: jamespage: ^15:07
fungii'm more in favor of just dropping tls v1 as long as we don't expect to cater to the older android, safari, internet explorer, et cetera versions which lack tls v1.1 support15:08
*** cloudtrainme has joined #openstack-infra15:08
sdaguemriedem: +215:08
sdaguefungi / AJaeger / mordred can we get another review on - https://review.openstack.org/#/c/28664215:08
openstackgerritDimitri Mazmanov proposed openstack-infra/project-config: Add check-requirements job to Kingbird  https://review.openstack.org/28664615:08
bkerofungi: would a potential openstack client on older android versions use TLS 1.0?15:09
*** Daisy has quit IRC15:09
*** jpr has joined #openstack-infra15:09
fungibkero: if it uses openssl 0.9.8y then perhaps15:09
bkerooh jeez15:10
bkeroThey're going to run out of letters soon15:10
fungiyes, after one more letter15:10
bkeroNot saying it isn't worth deprecating. And even at that point if you make a webapp with cordova it'll bundle a new browser which likely does its own SSL/TLS.15:11
*** jordanP has quit IRC15:11
*** korzen has quit IRC15:12
*** zz_dimtruck is now known as dimtruck15:12
fungiwe basically already engaged in this exercise once in the not too distant past, when we dropped ssl v3 support15:12
AJaegersdague: let me check...15:12
*** vgridnev has quit IRC15:13
AJaegerfungi, you're too fast for me ;)15:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Display a modal when a card is clicked on  https://review.openstack.org/28428015:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal to confirm archiving cards  https://review.openstack.org/28541715:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Allow a custom format string to be passed to time-moment  https://review.openstack.org/28427515:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a calendar directive  https://review.openstack.org/27850815:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a $resource wrapper for Due Dates  https://review.openstack.org/28427815:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add Due Dates to boards  https://review.openstack.org/28427915:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add onBlur and onFocus callbacks to user-typeahead  https://review.openstack.org/28427615:14
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Improve the board CSS a little  https://review.openstack.org/28427715:14
clarkbcatching up ssl thing dosnt affect us because we are tls only?15:14
AJaegerproject-config cores, could you review https://review.openstack.org/285949 https://review.openstack.org/#/c/285487/ and https://review.openstack.org/#/c/285148/ , please? Those are project-config changes that remove jobs...15:14
*** vgridnev has joined #openstack-infra15:15
*** gnuoy_ has joined #openstack-infra15:16
*** vgridnev has quit IRC15:16
*** gnuoy_ has quit IRC15:16
fungiclarkb: yep15:16
ShrewsGheRivero: you're going to want to add a reno release note for the new quota apis15:17
*** ajmiller_ has joined #openstack-infra15:17
*** ajmiller has joined #openstack-infra15:17
fungiclarkb: we've pretty much arrived now at there's one change mordred wrote to improve an apache vhost template we're not actually using but someone else might, and discussing additional tls hardening opportunities unrelated to today's threat advisory15:18
*** bgaifullin has quit IRC15:18
*** ajmiller has quit IRC15:19
*** sdake_ has joined #openstack-infra15:20
*** sigmavirus24_awa is now known as sigmavirus2415:20
*** vgridnev has joined #openstack-infra15:21
*** pradk has quit IRC15:22
*** pradk has joined #openstack-infra15:22
rcarrillocruzShrews: shouldn't the get_quota thing something like get_<resource>_quota15:22
*** sdake has quit IRC15:23
*** daemontool__ has joined #openstack-infra15:24
Shrewsrcarrillocruz: maybe? i haven't reviewed the code thoroughly yet15:25
Shrewsrcarrillocruz: best to ask GheRivero15:25
GheRiveroI don't know, get_nova_ram_quota looks too much/specific, but get_nova_quota could be an option. depending on the server version, you can have different set of quotas15:27
*** mtanino has joined #openstack-infra15:27
*** daemontool_ has quit IRC15:28
GheRiveroand it can be wrapped in a more generic get_quotas, so you can get all the quotas without specifying which service to get quotas from15:29
*** rossella_s has quit IRC15:32
*** rossella_s has joined #openstack-infra15:32
*** sshnaidm has joined #openstack-infra15:34
anteayaAJaeger: is disabling translations for server projects part of feature freeze? https://review.openstack.org/#/c/285949/15:35
odyssey4meFYI it would seem that the dsvm trusty image for OVH has an out of date/corrupt apt cache - we don't have to update the cache on any other providers as far as we've seen, but on OVH it appears to be mandatory15:36
*** asettle has quit IRC15:36
pabelangerodyssey4me: I've found cache on ubuntu in the gate to be flaky at best. For me, I've simply added the step in to ansible to ensure it has been updated before gating tests.15:37
*** pblaho has quit IRC15:37
anteayathe jenkinsii appear to be jenkinsing15:38
odyssey4mepabelanger yep, we're doing the same - I figured that it should be noted though in case it's not widely known15:38
*** sbelous_ has quit IRC15:38
anteayaovh has an increased error node launch attempts: http://grafana.openstack.org/dashboard/db/nodepool-ovh15:39
AJaegeranteaya: it's part of translation process - they start now with mitaka15:39
AJaegerand thus liberty gets retired15:39
anteayadoes ovh have floating ips, or perhaps we have maxed out on ovh quota?15:39
anteayaAJaeger: ah thank you15:39
*** andymaier has joined #openstack-infra15:40
*** annegentle has quit IRC15:40
*** keedya has joined #openstack-infra15:40
clarkbpabelanger: odyssey4me you should always update apt cache before doing anything else15:41
AJaegeranteaya: I discussed on i18n mailing list15:41
AJaegeranteaya: thanks for reviewing!15:41
fungianteaya: yeah, now that the rax-iad quota issue has been identified, ovh is next in line for figuring out. i'll take a look shortly15:41
openstackgerritDiana Whitten proposed openstack/requirements: Bump django-compressor to 2.0  https://review.openstack.org/28666315:41
*** jpr has quit IRC15:42
*** zeih has joined #openstack-infra15:42
openstackgerritMerged openstack-infra/project-config: Remove gate-barbican-tox-bandit  https://review.openstack.org/28548715:42
anteayaAJaeger: okay thanks, I don't follow the i18n mailing list, thanks for helping me to understand15:42
anteayaAJaeger: and you're welcome15:42
anteayafungi: thanks so much15:42
*** ryanpetrello has quit IRC15:43
*** mrmartin has joined #openstack-infra15:43
*** jordanP has joined #openstack-infra15:43
*** esker has quit IRC15:43
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043115:43
*** yamamoto has quit IRC15:43
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job  https://review.openstack.org/27342415:43
*** fhubik_brb is now known as fhubik15:44
*** icey has quit IRC15:44
*** ryanpetrello has joined #openstack-infra15:44
*** esker has joined #openstack-infra15:44
openstackgerritMerged openstack-infra/project-config: Remove oslo.vmware bandit job  https://review.openstack.org/28514815:44
openstackgerritMerged openstack-infra/project-config: Make ceph jobs non-voting until bug 1551305 is fixed  https://review.openstack.org/28664215:44
openstackbug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed] https://launchpad.net/bugs/155130515:44
*** icey has joined #openstack-infra15:44
clarkbodyssey4me: pabelanger our indexes are valid for a couple hours after new packages arrive iirc but that is still shorter than image refresh interval15:45
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Enable network isolation on all the jobs.  https://review.openstack.org/28567415:45
*** amotoki has quit IRC15:46
smcginnismriedem: Thanks for working on that ceph issue.15:46
*** zeih has quit IRC15:46
openstackgerritMerged openstack-infra/project-config: Disable Liberty translations for Server projects  https://review.openstack.org/28594915:48
pabelangerclarkb: yup, that's what I do now15:49
*** jaypipes has quit IRC15:50
*** jlanoux has quit IRC15:50
*** shardy has joined #openstack-infra15:51
*** yamahata has quit IRC15:51
fungiyeah, so no clue why but nova boot in obh-gra1 is resulting in a lot of instances ending up in ERROR state15:52
fungier, ovh-gra115:52
clarkbfungi: if you nova show their uuids sometimes you get more info15:52
*** yamamoto has joined #openstack-infra15:52
fungiyep, am about there15:52
clarkbanyone want to review (and hopfeully +A) https://review.openstack.org/#/c/285473/ ? I can start to work on booting a mirror in osic with that in15:53
fungialso bluebox is all "No more floating ips in pool external. (HTTP 404)" again. will clean up momentarily15:54
*** kzaitsev_mb has quit IRC15:54
*** gokrokve has joined #openstack-infra15:54
anteayaclarkb: I'm only +1 on system-config but you have mine15:55
openstackgerritMerged openstack-infra/shade: Allow testing against Ansible dev branch  https://review.openstack.org/28545015:55
fungiby the time i can dig the failure and uuid out of the debug log, the instance is no longer there according to nova15:57
*** maishsk has quit IRC15:58
pabelangerfungi: AJaeger: Missed some backscroll over the last 24 hours, did we loop back to -bindep rollout discussion?15:59
*** jsavak has quit IRC15:59
clarkbya nodepool can delete them pretty quickly, if you do a tail -f | grep ERROR and immediately run show on the output of that you might catch one15:59
fungipabelanger: not yet. still trying to quell wildfires with a thimble brigade15:59
anteayapabelanger: you know about eavesdrop.o.o yes? http://eavesdrop.openstack.org/irclogs/%23openstack-infra/15:59
*** jsavak has joined #openstack-infra16:00
pabelangeranteaya: indeed. I should start there first, before asking people to recap for me16:00
pabelangerfungi: sure, let me know if I can help16:00
anteayapabelanger: no a recap is fine, sometimes you don't have time for the backscroll16:00
anteayapabelanger: but you said you missed some, so wanted to make sure you had access to logs if you wanted16:00
fungithough at the moment the bluebox errors are so much more numerous than the ovh errors, i'm going to clean up the floating ip leak first16:01
*** ilyashakhat has joined #openstack-infra16:01
openstackgerritDiana Whitten proposed openstack/requirements: Bump django-compressor to 2.0  https://review.openstack.org/28666316:02
anteayaI'm hardly a role model at backscroll right now16:02
fungigrr, just realized i'm missing a conference call too16:03
*** Guest67668 has joined #openstack-infra16:04
*** kevinbenton has quit IRC16:04
*** jaosorior is now known as jaosorior_away16:06
*** ajmiller_ has quit IRC16:06
*** ajmiller_ has joined #openstack-infra16:06
*** chlong_ has quit IRC16:10
*** jswarren__ is now known as jswarren16:11
pleia2clarkb: +Aed 28547316:12
clarkbpleia2: woot thanks16:12
*** abregman has quit IRC16:12
AndyU@clarkb @nibalizer @jeblair @fungi etc :  we had a sort conversation here on 1/29. I was asking about donating infrastructure. Have made some progress and now I have some more questions. Here's the potential scenario. The company I work for hosts a cloud for openstack which we would isolate from our network and expose to the internet. Potentially several hundred physical blade servers. These are machines that we'd be repurposi16:13
AndyUIf one goes down we might not fix it and would just trash it.16:13
*** e0ne has joined #openstack-infra16:13
*** Guest67668 is now known as annegentl_16:13
AndyUQuestions: (1) How much support do you typically need from the cloud host? Can you give me an idea of the kinds of support and the frequency with which you tend to need it.16:13
*** jaypipes has joined #openstack-infra16:13
AndyU(2) Any concerns about hardware dying and not getting replaced?16:13
*** jcooley_ has quit IRC16:14
*** arxcruz has quit IRC16:14
AndyU(3) Does each node really need a public IP address??  That could be a real problem. Public IP's are getting to be in very short supply on my end I'm told. Any other options that you could foresee?16:14
*** dizquierdo has quit IRC16:14
AndyUNeed info for a meeting in 2 hours time  :-)16:14
clarkbAndyU: for your first message you cut off after "These are machines that we'd be repurposi"16:14
*** dizquierdo has joined #openstack-infra16:15
pleia2AndyU: for 3, yes, it really needs a public ip16:15
AndyUoh sorry. I'll reformat and send again in smaller pieces. stand by ;-)16:15
mordredAndyU: we do support IPv6 - so public IPv6 works if there is not enough public IPv416:15
pleia2AndyU: re: #2 - are you saying that over time the donation of nodes will decrease because you don't plan on replacing the hardware?16:16
AndyUcompleting the first post:  These are machines that we'd be repurposing and are 2-3 years old.16:16
pleia2oh, that's not so old16:17
AndyUok, I'll investigate IP options on my end.16:17
mordredAndyU: for 1 - the main thing that we usually need is for things we physically can't do - we manage everything from baremetal provisioning on up16:17
mordredAndyU: so it's usually things related to switch config16:17
*** Qiming has quit IRC16:17
mordredthat we are usually not allowed to touch where we need support16:17
*** bpokorny has joined #openstack-infra16:17
mordredbut those usually only need to be set up once to start16:17
mordred(we also would probably nee to be able to report "dude, the entire network went away" to someone)16:17
AndyUOnce switches, routers are configured do you find that they need to be changed?16:18
clarkbmordred: well wait16:18
mordredAndyU: not typically, no16:18
clarkbmordred: it depends on the type of donation, we can consume preexisting/managed cloud16:18
mordredclarkb: yes indeed16:18
clarkbmordred: or we can consume hardware more directly and provision the cloud ourselves16:18
mordredbut if this is hardware donation for an infra-cloud region16:18
clarkbright so I think question one is what type of donation is this16:18
clarkband that determines the type of support16:19
AndyUDo you prefer that we host a cloud and just provision you n number of nodes/tenants  to do with as you please?16:19
mordredI got the idea from "If one goes down we might not fix it and would just trash it." that it was hardware - but that's an excellent question16:19
fungiright, we're still working through our very first hardware hosting donation for infra-cloud. we're pretty good at consuming donated cloud resources however16:19
*** jlanoux has joined #openstack-infra16:20
anteayaAndyU: I'm confident you were linked to http://docs.openstack.org/infra/system-config/contribute-cloud.html during an earlier conversation, just adding it here for completeness16:20
mordredAndyU: yah - that's easier - although the need for public IPs still exists16:20
fungiso preference at this point is still to have free accounts/quotas in a reachable openstack-based cloud16:20
jeblairi'm here, and supportive of this conversation, but am going to let other people talk16:20
AndyUWe could potentially have a large numberof bare metal blade servers which we'd look to provide in whatever means is most advantagious to both sides. Want to eliminate the need for any kind of site access. Minimize support.16:20
mordredAndyU: yah. we also do not want to need site access or to bother you with a lot of support :)16:21
AndyUyes, I saw the link. Looking to dive deeper now16:21
*** yamamoto has quit IRC16:21
anteayaAndyU: great, thank you16:22
*** yamamoto has joined #openstack-infra16:23
mtreinishinfra-root: we still need to restart the subunit gearman worker on subunit-worker01.o.o The fixes landed yesterday so hopefully it will stay up now16:24
AndyUForgive my ignorance because I'm coming from a more management and less technical perspective on this. I just moved into this area a couple months ago. Knowing the resources potentially at our disposal, how would you optimally want them leveraged?16:24
pleia2mtreinish: I don't know how to do that, but if one of my co-conspirators is willing to show me, we can take care of it16:24
fungiAndyU: ideally you would install and run a reachable openstack cloud on the hardware, give us accounts/quotas for the services we connect to it, and then be reachable somehow in case we run into issues with it16:25
fungiAndyU: where by "you" i of course mean someone in your organization16:25
*** sorantis has quit IRC16:26
*** yamahata has joined #openstack-infra16:26
AndyUok. Does that openstack cloud need to be upgraded periodically, regularly by us?16:26
openstackgerritmariam john proposed openstack/requirements: Add couchdb to global requirements  https://review.openstack.org/28519116:26
mtreinishpleia2: it should just be logging into the server and doing something like service subunit-gearman-worker-A start (or something like that)16:26
fungiprobably. assuming you already run one or more openstack clouds anyway, you would probably just follow the same process you're doing elsewhere in your organization. i don't really know enough about how you're using openstack to guess16:27
AndyUok, great16:27
pleia2aha, someone as root has run /etc/init.d/jenkins-subunit-worker-A start16:28
fungiso far most of our donations have come from public cloud service providers, so the details around our needs are mostly from that perspective16:28
dhellmannwhat does it mean when the zuul status page reports a "queue length" of 2890 events?16:28
pleia2but that's jenkins16:28
anteayadhellmann: usually the gate is resetting16:28
AndyUSo you don't really care how we necessarily host the cloud on our end, you just want a bunch of tenants and the keys to use them as you wish. Does that sound right?16:28
*** _nadya_ has quit IRC16:28
fungiAndyU: and reachable ipv4 or ipv6 addresses to be able to assign to the virtual instances we boot there, yes16:28
dhellmannanteaya : ok, thanks. that seemed like a large number, compared to what I'm used to seeing there16:29
anteayadhellmann: 13 patches merging woot16:29
*** _nadya_ has joined #openstack-infra16:29
mtreinishpleia2: does that mean it's running? because we don't have anything being added to the db? maybe s/start/restart ?16:29
anteayadhellmann: yes it spikes when zuul is recalculating16:29
pleia2mtreinish: doesn't mean anything, I was just looking through the command history :)16:29
mtreinishah, ok16:29
*** ddecapit has joined #openstack-infra16:29
fungipleia2: i typically do `sudo service subunit-gearman-worker-A restart` so that probaly wasn't me16:30
AndyUok, got it. And I hear you saying that support requests would likely be very infrequent and just related to hosting issues??16:30
jeblairdhellmann: when zuul reloads its configuration, it drops its cache of gerrit changes, which slows it down a bit.16:30
*** ddecapit is now known as DuaneDeC716:30
*** kevinbenton has joined #openstack-infra16:30
fungiAndyU: yep, if we notice performance issues, or the environment goes offline, or something like that16:30
*** armax has joined #openstack-infra16:31
fungiAndyU: we're generally a pretty competent user, so tend not to reach out for assistance unless something is actually broken16:31
pleia2mtreinish: ok, started -A16:31
mtreinishpleia2: cool, thanks16:31
AndyUcool. And on the question of hardare failures that might take down some nodes. I presume you could live with that? re-juggle on your end?  It's unlikely but with that many servers it's bould to happen sometimes.16:32
dhellmannjeblair : somehow I seem to end up surprised by these sorts of cases and the change in behavior from what usually seems like "smooth" processing. I'll be happy when more of the release stuff is automated and I don't need to watch for jobs to finish. :-)16:32
mtreinishpleia2: I'll give it a min and see if anything is added, but we might need to do some debugging16:32
pleia2mtreinish: sure, I'm around16:33
*** vincentll has quit IRC16:33
*** _nadya_ has quit IRC16:33
*** fawadkhaliq has quit IRC16:33
jeblairdhellmann: yeah; the cache is becoming enough of a problem that i think we're going to seriously rework it in zuulv316:33
*** ajmiller__ has joined #openstack-infra16:33
jeblairdhellmann: but just to be sure, i logged in and checked the log, and it is indeed busy querying gerrit as fast as it can16:34
dhellmannjeblair : oh, I wasn't complaining about zuul, just commenting on my own pattern of behavior16:34
*** tiswanso has quit IRC16:34
*** yamamoto has quit IRC16:34
*** yamamoto has joined #openstack-infra16:34
jeblairdhellmann: no worries.  i'm just over-sharing.16:34
*** yamamoto has quit IRC16:34
dhellmannjeblair : np, I like learning16:35
*** tiswanso has joined #openstack-infra16:35
AndyULet me explain further - we could have cases where say you were getting 200 nodes, something breaks and now we can only give you 196. Presumable you can cope with that?16:35
*** ajmiller_ has quit IRC16:35
anteayawow that really cleared out the gate, yay merging patches16:35
anteayaAndyU: we can reset quota easily16:36
anteayawe just change a yaml file16:36
*** yamahata has quit IRC16:36
AndyUOk. I expected that but I just wanted it said here.  That's all I need for now! <fingers crossed>  Thanks all  ;-)16:37
clarkbAndyU: AndyU yup that is something we can handle fairly easily16:37
anteayaclarity is great, thank you16:37
mtreinishpleia2: ok is there anything in the log? I'd expect it to have added something by now (assuming there was a backlog and the previous fail state didn't drop everything)16:38
*** cloudtrainme has quit IRC16:38
*** e0ne has quit IRC16:38
AndyUThe bigger hrudle on our end is probably less getting things set up and made available and more concerns over the ongoing support.  The info you gave me is very helfull. Thanks again.16:38
*** eharney has quit IRC16:39
*** apoorvad has joined #openstack-infra16:39
*** esikachev has quit IRC16:39
*** vgridnev has quit IRC16:40
*** e0ne has joined #openstack-infra16:40
*** eharney has joined #openstack-infra16:41
openstackgerritYih Leong Sun proposed openstack-infra/infra-manual: Suggest to include a link to setting up gerrit on windows env.  https://review.openstack.org/28670316:41
*** madorn has quit IRC16:41
pleia2mtreinish: checking16:41
*** vgridnev has joined #openstack-infra16:41
openstackgerritMerged openstack/requirements: Add networking-nec to projects.txt  https://review.openstack.org/28013716:41
openstackgerritMerged openstack-infra/system-config: Add OSIC clouds.yaml details  https://review.openstack.org/28547316:41
pleia2mtreinish: heh, not running anymore, digging a bit more16:41
mtreinishwell, that would explain it :)16:42
*** DuaneDeC7 has quit IRC16:42
*** DuaneDeC7 has joined #openstack-infra16:42
pleia2mtreinish: had to remove a stray pid file, seems better now16:42
anteayaAndyU: understood, do share if there are further quesions16:43
anteayaAndyU: and thanks for thinking of us!16:44
mtreinishpleia2: ok, it added 1 run to the db, but I don't see any others being added16:45
mtreinishis there anything in the log, or is it just being sluggish16:46
pleia2mtreinish: seeing a lot of http://paste.openstack.org/show/488789/16:46
*** mrmartin has quit IRC16:46
*** fhubik has quit IRC16:47
*** exploreshaifali has joined #openstack-infra16:47
*** thorst_ is now known as thorst_afk16:47
mtreinishpleia2: that's expected I think, although I have no idea why the gearman client is adding events with urls like that16:48
*** pfallenop has quit IRC16:48
mtreinishoh, actually it's probably because: https://review.openstack.org/#/c/281383/ hasn't landed16:48
*** sridhar_ram1 has joined #openstack-infra16:48
*** pfallenop has joined #openstack-infra16:48
*** scheuran has quit IRC16:49
mtreinishpleia2: but is there anything besides 404s being logged?16:49
clarkbI am debugging my mount_volume script additions and after attaching /dev/vdc via cinder then mkpart lvming on /dev/vdc no /dev/vdc1 device is mknod'd in /dev16:49
clarkbI shouldn't need to explicitly mknod right?16:49
pleia2mtreinish: ok, also this http://paste.openstack.org/show/488792/16:50
*** jpr has joined #openstack-infra16:50
AndyU@pleia2 Reading back I don't think I clearly addressed your question "[10:16] <pleia2> AndyU: re: #2 - are you saying that over time the donation of nodes will decrease because you don't plan on replacing the hardware?" - Yes, we might not replace them or it might be a long time before more servers are added to fill the gap.16:51
mtreinishpleia2: ah, that is more likely the real problem16:51
pleia2mtreinish: but I guess I should only be looking for jobs with grenade-dsvm in them :)16:51
pleia2AndyU: I think it came out in the end, thank you16:51
mtreinishpleia2: can you stop the worker so we don't exhaust the backlog and I'll push a fix for that16:51
*** gokrokve has quit IRC16:52
pleia2mtreinish: ok, done16:52
AndyUAgree - again... just want tobe clear  ;-)16:52
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job  https://review.openstack.org/27921816:52
mtreinishpleia2: although I'm not sure what the fix is :)16:52
*** vgridnev has quit IRC16:52
*** ilyashakhat has quit IRC16:52
*** sridhar_ram1 has quit IRC16:53
*** vgridnev has joined #openstack-infra16:53
pleia2mtreinish: I can email you the debug log if you want to dig through for other things16:53
*** sridhar_ram1 has joined #openstack-infra16:53
*** toabctl has quit IRC16:53
mtreinishpleia2: sure, that might help16:54
*** sridhar_ram1 has quit IRC16:54
*** sridhar_ram1 has joined #openstack-infra16:54
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job  https://review.openstack.org/27374316:54
pleia2mtreinish: sent16:54
mtreinishbut I'm not sure what would be closing the IO object before we pass it into subunit2sql (which is what the exception is indicating is happening)16:54
mtreinishpleia2: thanks16:54
*** rcernin has quit IRC16:55
*** jpr has quit IRC16:55
anteayaclarkb: I don't know16:56
*** kzaitsev_mb has joined #openstack-infra16:56
*** fawadkhaliq has joined #openstack-infra16:56
*** mikelk has quit IRC16:56
*** toabctl has joined #openstack-infra16:57
anteayawooot I got an out of office reply in Welsh16:57
pleia2re: http://lists.openstack.org/pipermail/openstack-infra/2016-March/003941.html do we just drop the log file on the filesystem? (no bot magic will delete it?)16:58
pleia2I'll fix up the filename too16:58
anteayathats what you get for emailing someone from Cardiff University16:58
anteayaI think that is the process, yolanda has added logs successfully in the past, I believe16:59
*** DuaneDeC7 has quit IRC17:00
fungianteaya: they're quite proud of their language, even if only something like 25% are fluent in it and 4% speak it as their first language (last time i looked at the statistics on it anyway)17:00
anteayait is an awesome language17:00
yolandayes, i did it my just creating the file in the right place, and ensuring perms17:00
fungii spent a few years trying to learn welsh, but it's extremely opaque to anyone who doesn't already have some familiarity with celtic languages (which i most certainly do not)17:01
anteayathey have every reason to be proud17:01
*** annegentl_ has quit IRC17:01
clarkbpartprobe does seem to add a /dev/vdc1 but pvcreate is still unhappy about it and blkid exits 2 on /dev/vdc117:01
clarkbso weird17:01
anteayaI met mhickey at neutron mid-cycle17:01
anteayahe is looking into information for me so that I might learn irish17:01
anteayasounds like a lot of fun17:01
anteayaI'm looking forward to finding out more17:02
anteayanot sure I'm ready for welsh yet17:02
fungithough that's a northern celtic language while welsh is a southern celtic language, so they're pretty different linguistically17:02
anteayaso much I have to learn17:02
*** matrohon has quit IRC17:02
anteayaclarkb: :(17:02
*** gyee has joined #openstack-infra17:03
fungiother northern celtic languages include scottish gaelic and manx17:03
anteayaI don't know manx at all17:03
*** jsavak has quit IRC17:03
fungisouthern celtic languages besides welsh are basically all dead (gaulish) or an academic curiosity (cornish, brethonic)17:03
*** annegentl_ has joined #openstack-infra17:04
anteayabring back the languages!17:04
nibalizerwooo more cloud donations, thanks AndyU17:04
fungier, breton i meant17:04
*** vgridnev has quit IRC17:04
*** jsavak has joined #openstack-infra17:05
*** ifarkas has quit IRC17:05
*** jistr has quit IRC17:05
*** yamamoto has joined #openstack-infra17:05
anteayafungi: you could just be making up words, I wouldn't know the difference17:05
fungioh, and cumbric... i'd almost forgotten about that one17:05
anteayahow can you forget about cumbric17:05
clarkbfungi: do you know if http://docs.openstack.org/infra/system-config/sysadmin.html#cinder-volume-management is relying on an implicit reboot between volume attach and pvcreate?17:06
fungiclarkb: you should not need a reboot between those steps, no17:06
fungiclarkb: kernel hotplugging should cause the disk to just appear magically17:06
*** maishsk has joined #openstack-infra17:06
clarkbthats a good point the disk does and parted seems to wrok fine17:07
clarkbpvcreate however is one unhappy camper17:07
*** pcaruana has quit IRC17:07
fungiclarkb: usually i double-check dmesg and then make sure the kernel added the device to the /dev tree17:07
iremizovHi guys. Could you please review this patch set https://review.openstack.org/#/c/284680/17:07
clarkbfungi: yup I have it reliably added to /dev with partprobe17:08
clarkbfungi: http://paste.openstack.org/show/488795/17:08
fungithanks, was just about to ask what error it was throwing17:09
*** Swami has joined #openstack-infra17:10
*** yamamoto has quit IRC17:10
*** achanda has joined #openstack-infra17:10
*** hashar has quit IRC17:10
*** jpr has joined #openstack-infra17:11
fungihuh, i wonder if something is setting weirdness in the mbr?17:11
fungihow does parted describe /dev/vdc?17:11
fungimsdos not gpt hopefully17:12
fungisince you explicitly told it to create an msdos partition table17:12
clarkbhttp://paste.openstack.org/show/488797/ from parted --list17:13
clarkbyup msdos17:13
fungitry -vvvv with pvcreate to get more details?17:13
fungiweird device type maybe?17:14
mtreinishpleia2: so I'm still at a loss for what is closing the IO object before we parse it. My only thought is we're reuising the old object (because we close that after we're done)17:14
mtreinishbut the python docs for Queue say get() will remove and return a queued item17:14
clarkbhttp://paste.openstack.org/show/488798/ that isn't much more info17:15
*** ashleighfarnham has joined #openstack-infra17:15
clarkb Iam about to attach a second device and rerun through steps manually to see if something in my script is wrong17:16
fungi/dev/vdc1: Skipping (regex)17:16
*** maishsk_ has joined #openstack-infra17:16
clarkbis that sayin a regex somewhere says don't lvm this?17:17
fungilook in /etc/lvm.conf?17:17
*** dims_ has joined #openstack-infra17:17
*** zeih_ has joined #openstack-infra17:17
*** maishsk has quit IRC17:17
*** maishsk_ is now known as maishsk17:17
fungimaybe we have something weird in there?17:17
*** dims has quit IRC17:19
*** salv-orl_ has joined #openstack-infra17:19
clarkbthe filter should be fine but maybe global_filter is breaking us17:20
*** bpokorny has quit IRC17:20
*** ashleighfarnham has quit IRC17:20
*** esikachev has joined #openstack-infra17:22
*** salv-orlando has quit IRC17:22
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Update Entity Events processing  https://review.openstack.org/28672317:23
fungii mean, the block devices on rax are /dev/vdXY so maybe check the lvm.conf on static.o.o?17:24
openstackgerritMerged openstack-infra/openstackid-resources: Update Entity Events processing  https://review.openstack.org/28672317:24
mtreinishpleia2: yeah I just checked the Queue code and confirmed what the docs say. It does a deque popleft() when you call get()17:24
mtreinishso back to the drawing board17:24
fungiclarkb: that might be the only other prover where we're using a xen domu17:24
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for volume-backed live migration abort bug 1524898  https://review.openstack.org/28672517:25
openstackbug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,Confirmed] https://launchpad.net/bugs/152489817:25
clarkbfungi: they are xvd* on rax (I am on osic with vd*) but there is no global filter on the mirror host, also appears to be not the same verison of ubuntu17:25
clarkber s/mirror/static/17:26
clarkblet me check mirror.ord17:26
clarkbno global on mirror.ord17:26
*** esikachev has quit IRC17:26
clarkb# from devstack makes me really skeptical17:27
clarkband I did run devstack here so going to start from cleaner state17:27
*** erlon has joined #openstack-infra17:27
clarkbthat was it17:27
fungioh, right17:27
fungiwhat was the reason for running devstack on it?17:28
*** notmorgan is now known as morgan17:28
clarkbit was my osic test box17:28
clarkbso tested devstack run time and now testing volume attach17:28
fungiaha, so not the mirror host, just testing out the steps before making the mirror host17:28
clarkbthat'll learn me17:29
*** dtantsur is now known as dtantsur|afk17:29
clarkbI am going to leave the partprobe there even though I don't think it is strictly necessary17:29
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes  https://review.openstack.org/25204117:30
*** tiswanso has quit IRC17:30
*** degorenko is now known as _degorenko|afk17:31
*** f1ller is now known as filler17:32
*** tiswanso has joined #openstack-infra17:32
*** bpokorny has joined #openstack-infra17:32
*** cloudtrainme has joined #openstack-infra17:32
openstackgerritFrancesco Longo proposed openstack-infra/project-config: Added IoTronic project.  https://review.openstack.org/28611317:32
fungiyeah, i don't think i've ever needed to partprobe on a modern kernel17:33
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Test Puppet Parser Future - Do not merge  https://review.openstack.org/28673217:33
*** cloudtrainme has quit IRC17:33
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Add more debug logging for closed file issues  https://review.openstack.org/28673317:33
mtreinishpleia2: ^^^17:33
*** kushal has quit IRC17:33
mtreinishthat should help narrow it down at least17:33
*** sfinucan has quit IRC17:34
*** mrmartin has joined #openstack-infra17:34
*** thorst_afk has quit IRC17:35
openstackgerritClark Boylan proposed openstack-infra/system-config: Add support to shade-launch-node for cinder attach  https://review.openstack.org/28547717:35
*** thorst_afk has joined #openstack-infra17:36
*** thorst_afk is now known as thorst_17:36
mtreinishpleia2: once that lands and puppet applies it we can restart the worker and collect more data17:37
pleia2sounds good17:37
mtreinishthat should hopefully let us figure out why it's trying to read closed files17:37
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Added expand=location to events endpoints  https://review.openstack.org/28673517:38
mtreinishI still think reusing the old one is the most likely case, because it worked the first time but failed all the others after17:38
*** annegentl_ has quit IRC17:38
mtreinishI just have no idea why that would be happening17:38
pleia2would be good to get this one in too so it's not so noisy https://review.openstack.org/#/c/281383/17:38
mtreinishpleia2: yep, that's a good call17:39
dougwiglooks like gerritbot has gone to lunch in #openstack-lbaas.  known issue?17:39
SpamapStrash: ACK, I will take a look later today. THanks for the reminder.17:39
openstackgerritMerged openstack-infra/openstackid-resources: Added expand=location to events endpoints  https://review.openstack.org/28673517:39
fungiit still seems to be working in here17:39
fungidougwig: can you elaborate?17:40
openstackgerritBogdan Dobrelya proposed openstack-infra/project-config: Adjust acls for fuel-noop-fixtures  https://review.openstack.org/28610917:40
dougwigfungi: the last "proposed" message was at 4:40am, and i just submitted a bunch. nothing.17:40
fungidougwig: is openstackgerrit in the channel?17:40
dougwigfungi: yes.17:41
*** jlanoux has quit IRC17:41
fungii'll check recent config changes for it17:41
dougwigfungi: ok, i'll peek there too.17:41
dougwiggerritbot/channels.yaml looks fine.17:42
*** ihrachys has quit IRC17:42
*** Jeffrey4l has quit IRC17:43
nibalizerclarkb: pleia2 want to look at https://review.openstack.org/#/c/285740/ ? I think that will make pretty timing graphs for our puppet-ansible runs17:43
*** ihrachys has joined #openstack-infra17:43
*** derekh has quit IRC17:43
fungidougwig: so give me an example change number you submitted which never got echoed in channel17:44
fungii'll have a look in the gerritbot debug logs17:44
dougwigfungi: 28638017:44
*** harlowja_at_home has joined #openstack-infra17:45
dougwigfungi: PS1 at 9pm did echo.  PS2 at 9:36am did not.17:45
*** tphummel has joined #openstack-infra17:45
clarkbnew fail mode, neutronclient --insecure does not seem to work17:47
anteayamorning zaro17:47
openstackgerritBeth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard  https://review.openstack.org/28619417:47
*** jsavak has quit IRC17:47
*** HeOS has quit IRC17:47
*** ihrachys has quit IRC17:47
beisnerfungi, hi from the new openstack/charm-* projects (ci).  we've got our bot reviewing --verified N on ci-sandbox OK.  but that verified data isn't hitting the stream when we do the same against our projects.  i must be missing something.  thoughts on what to check?17:48
fungidougwig: 2016-03-01 17:36:48,780 INFO gerritbot: Sending "Doug Wiegley proposed openstack/neutron-lbaas: WIP - delete lbaasv2 agent driver  https://review.openstack.org/286380" to #openstack-lbaas17:48
*** jsavak has joined #openstack-infra17:48
*** claudiub|2 has quit IRC17:49
openstackgerritBeth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard  https://review.openstack.org/28619417:49
fungibeisner: by default acls don't allow arbitrary accounts to leave a verified label vote17:49
dougwigfungi: peek here, i said "lunch" right about when that should've hit: http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2016-03-01.log.html17:49
fungibeisner: so the acl would need to be adjusted to add that permission to a group we can put that account into17:50
fungibeisner: look at, say, the openstack/cinder acl config for an example17:50
beisnerfungi, ack, thank you17:50
*** sdake_ has quit IRC17:51
*** dizquierdo has quit IRC17:52
*** piet has joined #openstack-infra17:54
fungidougwig: yeah, and i don't see it restarting around that time, or disappearing in a netsplit or anything17:54
fungicould it have possibly been devoiced on that channel?17:55
*** sbelous_ has joined #openstack-infra17:55
*** kushal has joined #openstack-infra17:55
anteayaI think I will have some lunch before the meetings begin17:55
dougwigfungi: not that was reflected in channel.  i also don't see a netsplit.17:56
fungiyeah, trying to scarf down some food myself so i can spend a few minutes prepping to chair the meeting17:56
fungidougwig: yeah, i'm not finding obvious errors in the log, but still looking17:56
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add release announcement job to django_openstack_auth  https://review.openstack.org/28674717:56
dougwigfungi: i think the only people besides infra that have admin is the original channel owner (me), but unless someone has hacked my freenode account, i haven't recovered admin there in a **long** time.17:57
*** vgridnev has joined #openstack-infra17:58
*** baoli has quit IRC17:58
*** jamesmcarthur has joined #openstack-infra17:59
fungidougwig: fwiw, that patch is the _only_ comment i see gerritbot logging it sent after the one which is reflected in the channel log at 12:40 utc17:59
fungiso it may not be so much "gerritbot has gone silent" as "one message from gerritbot never made it to the channel"17:59
fungiyou suggested there were others?17:59
clarkbhttps://bugs.launchpad.net/python-neutronclient/+bug/1538959 now affects infra /me checks the this bug affects me flag17:59
openstackLaunchpad bug 1538959 in python-neutronclient "--insecure option did not take effect" [Undecided,In progress] - Assigned to Zhongcheng Lao (zlao)17:59
dougwigfungi: indeed, another that i just sent did show up.17:59
dougwigfungi: i have to disappear into a meatspace meeting for a bit.18:00
*** krtaylor has quit IRC18:00
*** sc68cal has joined #openstack-infra18:00
fungithe gerritbot debug log makes no mention of #openstack-lbaas between 12:40:51 and 17:36:48 utc18:01
*** sbelous_ has quit IRC18:01
fungiand the 17:36 logs were only for your 286380 patch upload18:01
*** jsavak has quit IRC18:02
clarkbfor those following along if you downgrade neutronclient then everything works18:02
dougwigfungi: i'll keep an eye on it. i was getting complaints last week of some missing announcements, but i didn't start watching until today.18:02
fungiclarkb: backward-incompatible change in neutronclient?18:02
fungidougwig: thanks, more examples will hopefully help narrow this down18:02
*** jsavak has joined #openstack-infra18:03
clarkbfungi: yes, --insecure is now a noop18:03
clarkbso if you hvae to not verify ssl as in case with osic then you can't use latest neutronclient18:03
*** zeih_ has quit IRC18:03
fungiclarkb: so we probably need to do something similar to what we've done for infra-cloud?18:03
fungiand if it's a self-signed cert, they may need to regenerate it with basic constranits set to let it act as a ca18:04
*** BobBall is now known as BobBall_AWOL18:04
*** ajmiller__ is now known as ajmiller18:04
fungiotherwise we just need to find out what ca it's signed by and add a trust for that18:05
*** esikachev has joined #openstack-infra18:05
clarkbfungi: we can't18:05
clarkb the issue is there is no DNS so the CN in the cert doesn't match18:05
*** mriedem has quit IRC18:05
clarkbwe could hack /etc/hosts but meh18:06
fungiwhat's the cn on it?18:06
*** sarob has joined #openstack-infra18:06
clarkbcloud1.osic.rackspace.com it is self signed18:06
*** mriedem has joined #openstack-infra18:06
clarkb(I tested the infra-cloud hack and ran into ip vs cn mismatches and lack of dns records)18:06
*** baoli has joined #openstack-infra18:07
*** yamamoto has joined #openstack-infra18:07
clarkbok security groups in both osic accounts should be working properly now18:07
anteayaalso is now a good time to have the neutron default security groups chat?18:08
clarkbnext step is boot a mirror but I need to run errands (that unfortunately overlap with our meeting today)18:08
fungiclarkb: yeah, so presumably we need to get them to add a dns record for that name18:08
clarkbfungi: yup if they did that we could use infra-cloud hack18:08
anteayaclarkb: oh so I guess not right now18:08
clarkbanteaya: no, not good for me right now18:08
fungiotherwise that's probably stretching the bounds of just-a-little-too-broken18:08
*** e0ne has quit IRC18:08
clarkbfungi: I mean the only things that can get compromised are the resources in the broken service :)18:09
anteayaclarkb: nod18:09
clarkbeven the afs stuff is all read only which anyone on the planet has access to?18:09
fungibut also the infra-cloud trust hack still depends on the self-signed cert also having the right (non-default for openssl) basic constraints configuration18:09
clarkbI was trying to think of a situation where something else could be in trouble if we got man in the middled18:10
*** kzaitsev_mb has quit IRC18:10
Shrewsso, https://review.openstack.org/285455 is not moving through the gate, and i'm a bit stumped why. who wants to be the first to point out where i'm being stupid??  :)18:10
*** vgridnev has quit IRC18:10
Shrewsthere will be prizes18:10
clarkbShrews: if that was approved before the shade dependency was merged it will need to be approved again18:10
cody-somervillecrinkle: Hey. Thanks for getting those requirements to Allison. Do you know if we have confirmation that the machines were shipped out in time to avoid being affected by the March 1st freeze?18:10
clarkbShrews: zuul won't auto queue deps that don't share a gate pipeline18:10
Shrewsclarkb: yeah, that's the case. a stuffed unicorn will be sent your way. can you re-approve for me?18:11
clarkbfungi: afs is read only, ansible ssh only exposes the public key, etc18:11
clarkbShrews: yes I can18:11
fungiclarkb: worst case for the comprimised trust is probably that nodepool trusts it to sign certs for other clouds. so _if_ someone got their hands on it _and_ also could redirect nodepool's network traffic/dns lookups, then it could maybe be used to malicious ends for anything we run on nodepool-managed workers18:11
openstackgerritgreghaynes proposed openstack/diskimage-builder: Make debootstrap cache opt-in  https://review.openstack.org/28588618:11
fungibut if they can get at the private key for that cert, then they can probably already compromise our nodepool-managed workloads in their environment anyway18:12
*** yamamoto has quit IRC18:12
fungihard to defend against an inside actor, the ssl cert is not the weakest link there18:13
clarkbexcept in the case of not verifying ssl they don't need the private key18:13
*** jed56 has quit IRC18:13
fungioh, i get you. i was talking about the risks of nodepool trusting the self-signed cert they're using18:13
clarkbfungi: that isn't an option so  Iam not really worrying about it right now18:13
fungior trusting their cloud-local certificate authority even if the api cert is not self-signed18:13
*** baoli has quit IRC18:14
fungiclarkb: right, sorry, i was thinking ahead to once they add a dns recored18:14
clarkbwell it is if we want to edit /etc/hosts and yolo dns18:14
clarkbbut I don't relaly want to do that either18:14
openstackgerritMerged openstack-infra/project-config: Added new repository for fuel-plugin-murano  https://review.openstack.org/26956718:14
clarkbwe could just offer to expense the real cert and dns record for them18:14
clarkbI can pay that out of my weekly beer budget18:14
fungithat might not be a terrible short-term workaround, but it's definitely not a long-term solution and i'd rather just press them to fix their dns18:15
*** dizquierdo has joined #openstack-infra18:15
fungithe /etc/hosts workaround i mean18:15
*** cznewt has quit IRC18:15
*** tongli has joined #openstack-infra18:15
AJaegerpabelanger: EmilienM 's change for puppet-lint to use ubuntu-trusty merged and I haven't seen him complaining yet, so assume it's fine. EmilienM can you confirm?18:16
AJaegerpabelanger: but that's all progress on the bindep front...18:16
EmilienMI always complain, is that what you say? :-P18:17
EmilienMall is fine for us AFIK :-) let me double check18:17
openstackgerritMerged openstack-infra/storyboard: Change MySQL search mode to 'boolean'  https://review.openstack.org/28189018:17
crinklecody-somerville: i have not seen confirmation yet18:17
EmilienMAJaeger, pabelanger: all looks fine: http://logs.openstack.org/82/282182/3/gate/gate-puppet-glance-puppet-lint/e343f6a/console.html18:17
fungiclarkb: apparently there are ns records for osic.rackspace.com to ns.rackspace.com and ns2.rackspace.com so in theory they were planning to have resource records under that subdomain18:17
*** openstackgerrit has quit IRC18:18
EmilienMAJaeger, pabelanger: wait, the job run on bare-trusty, is it expected?18:18
*** openstackgerrit has joined #openstack-infra18:18
pabelangerAJaeger: neat, let me look and see what is going on with it.18:18
pabelangerEmilienM: Ya, it should be using ubuntu-trusty18:18
pabelangerwhen did it merge?18:18
fungiEmilienM: did the job-template use node: {node} instead of node: ubuntu-trusty? if so, it was probably taking your default node: bare-trusty from the instantiating project entry18:18
EmilienMpabelanger: last night18:19
fungiEmilienM: looks right to me. i'll check the jenkins master where that job ran for indication it actually got the updated config18:20
pabelangerEmilienM: AJaeger: https://jenkins04.openstack.org/job/gate-puppet-heat-puppet-lint/79/consoleText is using bindep18:20
AJaegerthis one on ubuntu-trusty: http://logs.openstack.org/76/281376/6/check/gate-puppet-pacemaker-puppet-lint/b2e28e1/console.html18:20
EmilienMpabelanger: when did tat run?18:21
EmilienMok so we're good18:21
pabelangerI don't expect much issues with puppet-lint honestly, since they installed gem files themself18:21
pabelangerEmilienM: now, in the gate18:21
EmilienMfungi: sorry for the wrong link though the job ran this morning18:21
AJaegerpabelanger: my change failed wtih " Gems in the group system_tests were not installed." -is that ok?18:21
EmilienMpabelanger: nice catch! thanks again18:21
*** kzaitsev_mb has joined #openstack-infra18:21
pabelangerAJaeger: where did you see that?18:22
openstackgerritMerged openstack-infra/project-config: Add non-voting shade job to test upstream Ansible  https://review.openstack.org/28545518:22
AJaegerpabelanger: http://logs.openstack.org/76/281376/6/check/gate-puppet-pacemaker-puppet-lint/b2e28e1/console.html - but error is something else18:22
*** lucasagomes is now known as lucas-dinner18:22
AJaegerpabelanger: I see it also on http://logs.openstack.org/82/282182/3/check/gate-puppet-glance-puppet-lint/cce794c/console.html - which succeeds18:23
pabelangerAJaeger: Ya, that is a pep8 failure for puppet18:23
pabelangerformatting issue on the patchset18:23
AJaegerSo, we're fine on the bindep front, great!18:23
pabelangerI'm going to add an experimental job for ansible roles now18:24
*** flepied has quit IRC18:24
fungiEmilienM: on jenkins02 where your example ran, the gate-puppet-glance-puppet-lint job is not updated to use bindep and ubuntu-trusty nodes18:24
*** sputnik13 has joined #openstack-infra18:24
EmilienMfungi: do I need to patch something else?18:25
AJaegerfungi: manual jjb run needed?18:25
fungiyeah, i think jjb updates are probably broken on at least one master again. i'll try to get them synced up while i prep for the meeting18:25
AJaegerthanks, fungi18:25
*** fabio_ has quit IRC18:25
AJaegerfungi, still ok to add an agenda item?18:26
*** serverascode has quit IRC18:26
fungiEmilienM: so basically, the update has propagated far enough that it looks like the new job works fine, we're just still running the old version of the job on some changes until i fix jjb updates18:26
fungiAJaeger: sure18:26
* AJaeger adds at the end of a long list18:27
fungiAJaeger: just be mindful that the agenda may be getting full so we could run out of time18:27
fungiyeah, that18:27
*** zhiyan has quit IRC18:27
*** cznewt has joined #openstack-infra18:27
EmilienMfungi: last question: puppet CI has some jobs runnong on devstack-trusty - should I switch them too?18:27
*** weshay has quit IRC18:27
*** blogan_ is now known as blogan18:27
* AJaeger managed the captcha18:28
*** serverascode has joined #openstack-infra18:28
*** zhiyan has joined #openstack-infra18:28
clarkbfungi Ya I will send them email asking if they plan to get that done soon18:28
clarkbfungi any thoughts on whether or not the mirror host and maybe a max-servers: 1 nodepool config shluld wait on that?18:29
fungiEmilienM: should be fine to, yes. the ubuntu-trusty nodes are supposed to be basically identical to devstack-trusty, except that we can't easily update ubuntu-trusty in rackspace while we still build devstack-trusty there via an older snapshot-based method18:30
EmilienMfungi: ack18:30
*** kushal has quit IRC18:30
*** jaosorior_away is now known as jaosorior18:31
*** ashtokolov_ has joined #openstack-infra18:31
*** evgenyl_ has joined #openstack-infra18:31
anteayaAJaeger: yay18:31
*** roaet- has joined #openstack-infra18:31
*** jordanP has quit IRC18:33
*** sambetts is now known as sambetts|afk18:33
fungiEmilienM: so the general risk with ubuntu-trusty nodes at the moment is that in rackspace they may lag behind updates in other providers due to our glance issues there18:33
fungithough maybe that's solved now?18:33
fungiclarkb: do you recall the most recent status on that front?18:33
*** kzaitsev_mb has quit IRC18:34
EmilienMfungi: ok good to know. I'll rune xperimental jobs first, like you did.18:34
AJaegeranteaya: could you put 286497 and 286527 on your review queue, please? One is a skip-rule addition for manila, the other resorts all of them.18:34
anteayaI will look after soup18:35
clarkbfungi still broken I pasted the error last night18:35
clarkbfungi no tenant specifird on swift token reup18:35
clarkbif you look at the builder debug log you can see traceback or findy paste link18:36
*** roaet_ has quit IRC18:37
*** ashtokolov has quit IRC18:37
*** evgenyl has quit IRC18:37
AJaegeris there a specific reason that django_openstack_auth does not use publish-to-pypi template? See https://review.openstack.org/#/c/286747/1/zuul/layout.yaml18:37
*** ashtokolov_ is now known as ashtokolov18:37
*** evgenyl_ is now known as evgenyl18:37
AJaegerthanks, anteaya . Enjoy your soup!18:39
*** krtaylor has joined #openstack-infra18:42
*** BobBall_1WOL has joined #openstack-infra18:44
*** dims has joined #openstack-infra18:45
*** dmellado has quit IRC18:48
*** dkehn has quit IRC18:50
*** BobBall_AWOL has quit IRC18:50
*** itsuugo has quit IRC18:50
*** dkehn has joined #openstack-infra18:50
*** wolsen has quit IRC18:50
*** mdenny has quit IRC18:52
*** dims_ has quit IRC18:52
*** rbrndt has quit IRC18:52
*** pahuang has quit IRC18:52
*** rbrndt has joined #openstack-infra18:52
*** pahuang has joined #openstack-infra18:52
*** ociuhandu has quit IRC18:52
*** acabot has quit IRC18:52
*** afazekas has quit IRC18:52
*** _ody has quit IRC18:52
*** xiangxinyong has quit IRC18:54
*** aeng has quit IRC18:54
*** bpokorny has quit IRC18:54
*** SpamapS has quit IRC18:54
*** openstack has joined #openstack-infra19:15
*** SpamapS has joined #openstack-infra19:15
anteayameetbot is still missing in #openstack-meeting-3 if someone has a moment19:15
*** openstackstatus has joined #openstack-infra19:17
*** ChanServ sets mode: +v openstackstatus19:17
anteayameetbot is back in meeting-3 now, thank you19:18
*** stevelle has left #openstack-infra19:18
*** sripriya_ has joined #openstack-infra19:18
*** esikachev has joined #openstack-infra19:19
*** openstackgerrit has joined #openstack-infra19:19
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on OR filtering  https://review.openstack.org/28678619:22
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix create_server() with a named network  https://review.openstack.org/28678719:23
*** maximov_ has quit IRC19:25
*** bryan_att has quit IRC19:25
*** mgkwill has quit IRC19:25
*** markmcclain has quit IRC19:25
*** gnuoy has quit IRC19:25
*** clif_h has quit IRC19:25
*** odyssey4me has quit IRC19:25
*** mwhahaha has quit IRC19:25
*** msuriar has quit IRC19:25
*** briancurtin has quit IRC19:25
*** ikalnitsky has quit IRC19:25
*** Adri2000 has quit IRC19:25
*** sulo has quit IRC19:25
*** whoops has quit IRC19:25
*** StevenK has quit IRC19:25
*** lane_kong has quit IRC19:25
*** mhayden has quit IRC19:25
*** niska has quit IRC19:25
*** nibalizer has quit IRC19:25
*** alaski has quit IRC19:25
*** bauzas has quit IRC19:25
*** tdasilva has quit IRC19:25
*** johnthetubaguy has quit IRC19:25
*** maishsk has quit IRC19:26
*** sc68cal has quit IRC19:27
openstackgerritMerged openstack-infra/openstackid-resources: Fix on OR filtering  https://review.openstack.org/28678619:28
*** ikalnitsky has joined #openstack-infra19:31
*** maximov_ has joined #openstack-infra19:31
*** bryan_att has joined #openstack-infra19:31
*** mgkwill has joined #openstack-infra19:31
*** markmcclain has joined #openstack-infra19:31
*** gnuoy has joined #openstack-infra19:31
*** clif_h has joined #openstack-infra19:31
*** odyssey4me has joined #openstack-infra19:31
*** mwhahaha has joined #openstack-infra19:31
*** bauzas has joined #openstack-infra19:31
*** msuriar has joined #openstack-infra19:31
*** briancurtin has joined #openstack-infra19:31
*** Adri2000 has joined #openstack-infra19:31
*** sulo has joined #openstack-infra19:31
*** whoops has joined #openstack-infra19:31
*** StevenK has joined #openstack-infra19:31
*** lane_kong has joined #openstack-infra19:31
*** mhayden has joined #openstack-infra19:31
*** niska has joined #openstack-infra19:31
*** nibalizer has joined #openstack-infra19:31
*** alaski has joined #openstack-infra19:31
*** tdasilva has joined #openstack-infra19:31
*** johnthetubaguy has joined #openstack-infra19:31
*** mhayden has quit IRC19:31
*** abregman has joined #openstack-infra19:32
openstackgerritMerged openstack-infra/storyboard: Updated documentation for installing Storyboard  https://review.openstack.org/28619419:32
*** mhayden has joined #openstack-infra19:33
*** geekinutah has joined #openstack-infra19:36
*** taron1 has quit IRC19:37
*** sigmavirus24 is now known as sigmavirus24_awa19:37
geekinutahfolks, reading http://docs.openstack.org/infra/system-config/contribute-cloud.html19:37
geekinutahspecifically in the requirements, "A public IP address"19:37
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI  https://review.openstack.org/28679319:38
nibalizergeekinutah: hi, we're actually in a meeting in a different channel right now19:38
nibalizerso we'll be more 'here' in about 20 minutes19:38
geekinutahdoes it matter where this IP address is, like preference for fixed IP to be public or floating or both19:38
geekinutahnibalizer: np, I'll lurk while you meet19:38
*** ajmiller_ has joined #openstack-infra19:39
mordredgeekinutah: we _prefer_ fixed public, but can handle floating if that's what you can provide19:39
nibalizer#openstack-meeting is the channel if you want to lurk that19:39
mordredgeekinutah: (managing floating ips requires more api calls and is more prone to failures than clouds with public fixed ips via dhcp - but we have a floating ip cloud in our set currently, and we also ran on hp for a few years which was floating)19:39
pabelangerSo user Martcheap on the wiki emailed me asking why his account was blocked.  He used in email system on wiki.o.o.19:40
geekinutahmordred: makes sense, thx19:40
pabelangerlooking at his contrib log, he is clearly spamming19:40
pabelangerI told him to connect here and talk about it19:40
mordredpabelanger: hah19:40
mordredpabelanger: you're so nice19:40
mordredpabelanger: I would have said "you are blocked because you are spamming"19:41
docaedopabelanger: wow - that's some nerve!19:41
pabelangerwell, mostly curious if they do join! And see what they say19:41
*** vgridnev has joined #openstack-infra19:41
*** ajmiller has quit IRC19:41
docaedoyeah great response, I think I'd have done the same, will be fun to see if they join19:42
*** vgridnev has quit IRC19:43
*** Sukhdev has joined #openstack-infra19:43
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add test for os_server Ansible module  https://review.openstack.org/28542419:44
*** esikachev has quit IRC19:46
nibalizerpabelanger: bizzare19:47
*** esikachev has joined #openstack-infra19:47
pabelangerYa, I guess we should check if they really are an openstack contributor or not19:48
*** gyee has quit IRC19:48
bkero"What? This spam-bot signed a CLA too?"19:48
*** hashar has joined #openstack-infra19:49
*** sigmavirus24_awa is now known as sigmavirus2419:52
*** ajmiller_ is now known as ajmiller19:52
openstackgerritFrancesco Longo proposed openstack-infra/project-config: Added IoTronic project.  https://review.openstack.org/28611319:53
*** erikwilson has joined #openstack-infra19:53
*** sc68cal has joined #openstack-infra19:53
*** terryw is now known as otherwiseguy19:54
mtreinishnibalizer, mordred, clarkb, jeblair: if you get a sec can you look at: https://review.openstack.org/286733 and https://review.openstack.org/28138319:54
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add test for os_server Ansible module  https://review.openstack.org/28542419:54
*** erikwilson has quit IRC19:54
mtreinishwe need the first 1 as a first step in debugging why the subunit worker is passing closed files into subunit2sql19:54
*** rockyg has joined #openstack-infra19:56
*** esikachev has quit IRC19:57
anteayapabelanger: I want you to get from them who is telling them to do this19:58
anteayapabelanger: obviously us blocking them is blocking their paycheque19:58
*** maishsk has joined #openstack-infra19:58
*** david-lyle has quit IRC19:59
*** kgiusti has left #openstack-infra19:59
annegentl_hi fungi, had another question about cutoff date for patches for the Austin summit, do you know?20:00
AJaegerWe run out of time in the infra-meeting, so here's what I wanted to share. armax, mestery,dougwig this is for you as20:01
annegentl_I'm the one people ask, I must be super accessible :)20:01
AJaegerConstraints are enabled for nova, glance, cinder. Neutron still uses -constraints jobs, patch up at https://review.openstack.org/286777 and https://review.openstack.org/286778 to move them over.20:01
AJaegerNow waiting for post jobs to get constraint enabled - jesusaurus, have you made any progress on that one?20:01
fungiannegentl_: saw in scrollback but then you disappeared20:01
AJaegerWhat's our timeline? What will we get done for Mitaka?20:01
AJaegerI'd like to ask lifeless, to write an email telling projects what do to do if they want to use constraints...20:01
*** vgridnev has joined #openstack-infra20:01
fungiannegentl_: i'm planning to send the last batch on thursday20:01
annegentl_fungi: yeah sorry :) online and offline lately20:01
annegentl_fungi: ok, sounds good, so land by Thursday?20:01
armaxAJaeger: ack20:02
mtreinishfungi, I knew I was forgetting someone in my ping about the puppet-subunit2sql patches... :)20:02
fungiannegentl_: yes, it could be as early as thursday depending on how my week shapes up20:02
armaxihrachys and I talked about this and we were thinking of taking care of this after M320:02
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs  https://review.openstack.org/27342420:02
armaxAJaeger: is that ok?20:02
fungiannegentl_: so merging by thursday is guaranteed safe20:02
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Use isinstance() for result type checking  https://review.openstack.org/28681120:02
dougwigAJaeger: "if they want to use" <-- is it really helpful if it's voluntary?20:02
*** dizquierdo has quit IRC20:02
annegentl_fungi: cool thanks20:03
mrmartinpabelanger: what is the situation with the wiki?20:03
AJaegerarmax: That's fine with me. I can make 777 as WIP until then. M3 is this week, correct? So, this is next week?20:03
armaxAJaeger: aye20:03
jesusaurAJaeger: sorry, I've been firefighting internal issues for a few weeks, I haven't been able to debug the issues with my zuul-cloner change20:03
AJaegerdougwig: let's see how lifeless writes it up;) But it's for projects to enable it.20:03
AJaegerjesusaur: Ah, thanks for the update20:04
*** vgridnev has quit IRC20:04
*** taron1 has joined #openstack-infra20:04
*** amitgandhinz has quit IRC20:04
armaxAJaeger: thanks for beating us to it20:04
greghaynesianw: Would you say that simple-init has been most of your new distro debugging?20:04
*** sdake has quit IRC20:05
ianwgreghaynes: that was a big part of it20:05
greghaynesianw: thats kind of the assumption I have been operating on since thats what its looked like to me20:05
ianwmostly it has been fallout from the switch to "-minimal" build20:05
*** daemontool__ has joined #openstack-infra20:05
ianwwhich is closely tied to simple-init, but also more20:05
*** amitgandhinz has joined #openstack-infra20:05
AJaegerarmax: ;)20:06
fungimrmartin: apparently the update is that at least one spammer has reached out to us asking why we blocked them20:06
ianwgreghaynes: i want to add back that docker image build job as a separate job from the build functional tests, sound ok?20:06
mrmartinand why?20:06
fungimrmartin: my guess is this means spammers may have compromised legitimate lp accounts20:06
greghaynesianw: why separate?20:06
greghaynesianw: I had no idea it went away20:06
mrmartinanyway, the LP openid implementation is broken20:06
ianwgreghaynes: well, as you found it wasn't really running due to no docker20:06
mrmartinthey are not handling properly a handler expiration, so it never expires20:07
*** julim has quit IRC20:07
fungioh, neat20:07
ianwgreghaynes: i mean in upstream.  i just feel like the functional tests are doing more than enough, and this is really a separate thing20:07
*** jsavak has quit IRC20:07
mrmartinI realized that during the askbot openstackid.org integration, openstackid implementation is much better from this aspect. (openid assoc handler)20:07
*** jsavak has joined #openstack-infra20:08
*** julim has joined #openstack-infra20:08
*** jcoufal_ has quit IRC20:08
greghaynesianw: oh, making it separate wont actually help failure rate (itll just fail in the other test at the same rate), it would only help run time at the expense of using a node, and the idea is that dib tests are really fast to add on since cache warmup is the slow part20:08
fungiyeah, it has crossed my mind that moving the wiki to openstackid sooner than lpanned might help matters, but i also don't want us to rush that migration since it's possible to end up in a worse state long-term if we don't figure out a way to try to associate accounts20:09
*** daemontool_ has quit IRC20:09
greghaynesianw: https://review.openstack.org/#/c/177002/12 if you havent seen it20:09
*** esikachev has joined #openstack-infra20:09
*** david-lyle has joined #openstack-infra20:09
*** maishsk has quit IRC20:09
greghaynesianw: conflicts after your test runner reworking, but that is what is needed for docker to work20:09
mrmartinyeah, account migration is a problem there.20:09
ianwgreghaynes: yep, that's what i'm referring to20:09
*** julim has quit IRC20:10
ianwit really looks different to the other func tests to me20:10
greghaynesianw: thats fine, theres nothing wrong with running a series of tests in serial20:10
mrmartinI can check how mediawiki works with openstackid.org, in an ideal world, it must work properly.20:10
greghaynesianw: whether or not to do that isnt a matter of organization, its a matter of performance20:10
*** jamesmcarthur has quit IRC20:10
hasharmrmartin: I am not sure how well maintained is the mw OpenId extension though20:11
ianwgreghaynes: alright, well either way i was going to rebase those changes on the newer test runner -- want me to do that?20:11
greghaynesianw: if you could thatd be awesoome20:11
*** gokrokve has joined #openstack-infra20:12
*** austin81 has quit IRC20:12
ianwgreghaynes: ok, that's pretty much bubbled to the top of my todo list (which is nice, because it means fires i know about are out, for now :)20:12
*** jaosorior has quit IRC20:12
ianwmy main bubble is getting centos/f23-minimal functional test, but that's a sub-bubble :)20:13
greghaynesianw: :)20:13
*** abregman has quit IRC20:13
*** piet has quit IRC20:13
*** abregman has joined #openstack-infra20:13
*** flepied1 has joined #openstack-infra20:14
AJaegerwe have removed pandoc and dvipng from bindep (project-config) but not yet from system-config setup, could I get some review for https://review.openstack.org/#/c/286242/ and https://review.openstack.org/#/c/284371/ , please?20:14
*** sdake has joined #openstack-infra20:16
*** flepied has quit IRC20:16
*** ihrachys has quit IRC20:18
*** maishsk has joined #openstack-infra20:19
openstackgerritStephen Gordon proposed openstack-infra/project-config: Skip magnum functional test jobs on docs changes  https://review.openstack.org/27789220:19
mrmartinanyway, what's the next step with wiki?20:19
*** |-paul-| has joined #openstack-infra20:20
*** yamahata has quit IRC20:21
openstackgerritThomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs  https://review.openstack.org/28613620:21
*** xyang1 has quit IRC20:21
pabelangermrmartin: only 1 spammer today, I haven't done anything today to stop them20:26
pabelangerso, the changes we made yesterday have stopped them for the moment20:26
mrmartinsometimes they are going away, but it doesn't mean we solved the issue20:26
fungithough clearly leaving new account creation disabled is not a long-term fix20:26
mrmartinwe should add a real captcha for the new account creation20:26
pabelangergetting things undercontrol was step 120:26
fungiand yeah, there is no long-term fix to spam really, just ever escalating barriers to usability until you find the sweet-spot between what spammers are willing to endure to use your system and what legitimate users are willing to endure. luckily the former is usually lower than the latter20:27
openstackgerritTim Buckley proposed openstack-infra/subunit2sql: Add API methods for getting tests by prefix  https://review.openstack.org/28333420:27
clarkbya wasn't in meeting channel (haven't rejoined since my client died in ft collins) but had to see tax preparer20:28
clarkbwill join now so that I am there though20:28
clarkbthat is all done now so yay20:28
mrmartinI can check this captcha at new LP account issue in a dev environment.20:28
fungicaptcha on new account creation probably solves this _if_ the spammers aren't using a captcha solver service to blow through them20:28
clarkbfungi: did you have an opinion on whether or not we should be booting the osic mirror and configuring a max server of one in nodepool before the ssl situation is better?20:28
mrmartinwho provides a well-working captcha service?20:28
pabelangerI still don't think we have the properly tooling inplace to stop spammers. Things like spamblacklist title and others appear to be the current methods.  Once setup and installed, then wiki admins should be able to deal with most of it out side of -infra20:29
pabelangersomething we can work on moving forward20:29
fungiclarkb: i'm inclined to wait until they can add a dns entry--that seems like it should be quick and cheap20:29
nibalizerpabelanger: should we look at upgrading the blacklist module?20:30
*** sdake has quit IRC20:31
*** _nadya_ has quit IRC20:31
funginibalizer: i think upgrading things is going to be hit-or-miss until we can move wiki.o.o to a newer distro release so that we can install a newer mediawiki to support newer plugins20:31
pabelangernibalizer: not sure, I need to read up on it more. I think the current one only deals with http links. Where most of our current spam is not using http links, just phone numbers and such20:31
fungialso, spam mitigation is one of those areas where you often end up in a continual upgrade cycle to get the new filtering/blocking features necessary to thwart spammers who have figured out ways around your previous solutions20:32
clarkbok I can send mail to them20:32
mrmartinmod-security with a phone matching rule?20:32
clarkbsee whether or not that is a thing20:32
fungithanks clarkb!20:33
*** hashar has quit IRC20:33
EmilienMpabelanger: if ubuntu-trusty is the new node to use for ubuntu, what is the centos one?20:33
fungimrmartin: well, general pattern blocks like "don't allow anything that looks like a phone number" would, for example, prevent us from updating https://wiki.openstack.org/wiki/Infrastructure/Conferencing because we document a legitimate phone service we use/run20:34
*** austin81 has joined #openstack-infra20:34
fungiEmilienM: devstack-centos7 (we haven't renamed it)20:34
EmilienMfungi: ok. So I just need to use ubuntu-trusty instead of devstack-trusty, right?20:34
*** tongli has quit IRC20:35
fungiEmilienM: really either would probably work the same. basically we never had a bare-centos7 so there was no need to migrate jobs to a new name for those. we git rid of bare-centos6 when we dropped python 2.6 testing so never ended up using the centos-6 images i was working on20:35
pabelangerEmilienM: ubuntu-trusty and devstack-centos7 are the current dibs20:35
*** jsavak has quit IRC20:35
pabelangerinfact, devstack-centos7 is not the final boss.  I believe ianw is working on centos-7 dibs20:36
fungier, got rid (finger memory always makes me want to type "git")20:36
pabelangerfrom centos-minimal20:36
EmilienMfungi: ok so I don't need to patch our CI to use ubuntu-trusty I guess20:37
funginot sure what you mean there20:37
EmilienMour puppet-beaker & puppet-integration jobs use devstack-trusty nodes20:37
pabelangerubuntu-trusty is what we are using for bindep20:37
fungianyway, there will come a point when we move devstack-trusty jobs to ubuntu-trusty and devstack-centos7 jobs to centos-720:37
fungiEmilienM: yeah, no need to switch those now unless you simply want to20:38
*** e0ne has joined #openstack-infra20:38
EmilienMfungi: yeah that's why I'm asking so I can help you with the puppet jobs20:38
*** Guest95751 is now known as Vivek20:38
EmilienMbut if no help is needed on that, I'll let the transition happen20:38
*** Vivek has quit IRC20:38
*** Vivek has joined #openstack-infra20:38
*** kushal has quit IRC20:38
pabelangerfungi: 286785 adds bindep for ansible jobs20:39
fungithe move off devstack-.* nodes will be much simpler to orchestrate en mass because the underlying configuration is basically the same. it's the move off bare-.* nodes which is taking a lot of extra care to get right without being too disruptive20:39
pabelangerexperimental functional testing for ubuntu-trusty20:39
*** yamahata has joined #openstack-infra20:39
EmilienMfungi: once thing I noticed is that it takes a lot of time (lately at least) to get a devstack-trusty node comparing to a ubuntu-trusty node20:39
fungiEmilienM: that may simply be a demand issue, because not a lot of jobs are using ubuntu-trusty yet. i wouldn't count on it remaining that way for long20:40
*** abregman is now known as abregman|nb20:40
EmilienMfungi: ok good to know, so I won't patch that thing. i'll let you manage that en mass - thanks for this work btw20:41
*** exploreshaifali has quit IRC20:42
*** maishsk_ has joined #openstack-infra20:42
*** maishsk has quit IRC20:43
*** sdake has joined #openstack-infra20:43
*** maishsk_ is now known as maishsk20:43
*** pcaruana has joined #openstack-infra20:44
annegentl_fungi: hey one area I wanted to understand, is infra able to consume an OSIC pop-up cloud? As in, would that work as an extra resource around release times?20:45
trashSpamapS: thanks20:45
*** jamielennox|away is now known as jamielennox20:45
clarkbannegentl_: I am currently trying to add in osic proper20:46
clarkbannegentl_: so I think the answer is yes, but I odn't know what an osic pop-up cloud is20:46
*** e0ne has quit IRC20:47
annegentl_clarkb: oh just my term for "hey have some OSIC"20:47
nibalizergeekinutah: did ou get your questions answered?20:47
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Skip dsvm jobs on release note only tempest changes  https://review.openstack.org/28683120:47
*** yamahata has quit IRC20:47
clarkbannegentl_: in theory all you would need to do is bump our quota and reflect that in nodepool20:48
*** jsavak has joined #openstack-infra20:48
annegentl_clarkb: pop-up, meaning, doesn't have to be there except at peak times20:48
annegentl_clarkb: okay, good to know.20:48
clarkb(once I get it running, currently trying to sort out ssl)20:48
*** bpokorny has joined #openstack-infra20:48
*** hashar has joined #openstack-infra20:49
*** gyee has joined #openstack-infra20:50
*** jpr has quit IRC20:51
openstackgerritMerged openstack-infra/shade: Fix heat create_stack and delete_stack  https://review.openstack.org/27604520:54
*** gildub has joined #openstack-infra20:54
*** piet has joined #openstack-infra20:55
*** e0ne has joined #openstack-infra20:55
*** julim has joined #openstack-infra20:55
*** esikachev has quit IRC20:56
*** sdake has quit IRC20:56
*** amrith is now known as _amrith_20:56
*** hichihara has quit IRC20:57
*** rguillebert has quit IRC20:57
clarkbfungi: I did just confirm that /etc/hosts using the cert they present to veirfy the self signed cert works20:58
clarkbfungi: so I don't think they made a bad self signed cert at the very least20:58
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI  https://review.openstack.org/28679320:58
*** maishsk has quit IRC20:58
*** sigmavirus24 is now known as sigmavirus24_awa20:58
*** sigmavirus24_awa is now known as sigmavirus2420:58
*** e0ne has quit IRC20:59
geekinutahnibalizer: yeah, I'm sure I'll have more later :-)20:59
*** sdake has joined #openstack-infra20:59
geekinutahI'm crawling through nodepool and friends to try and answer some preemptively20:59
anteayaannegentl_: well given the amount of time it takes to figure out a cloud, it is optimal if we have the cloud purring along prior to the rush21:00
anteayaelse we spend time debugging that cloud under load21:00
anteayawhich does happen21:00
nibalizerwhat organizaion are you with?21:01
fungiannegentl_: yeah, basically osic is shaping up nicely from what clarkb has tested, minus a dns update request and the fact that they freaked out when we wanted more than 100 virtual machines and ip addresses for them21:01
annegentl_IP addresses, the new gold rush21:01
annegentl_anteaya: good point21:02
*** maishsk has joined #openstack-infra21:02
clarkbannegentl_: thankfully we have a new metal to make ip addresses out of that is dirt cheap21:02
fungialso we've repeatedly said we gladly use ipv6 instead of ipv4 for this, to which they've been completely silent (i'm willing to bet they don't want to figure out ipv6 in neutron)21:02
anteayaannegentl_: now we won't say no to new resources at any time21:02
fungihah, yes, what clarkb just said21:02
anteayabut the notion that we have a bunch we bring online week prior to milestone is just going to mean a different kind of broken21:03
*** mrmartin has quit IRC21:03
annegentl_anteaya: fungi: yeah I think it's interesting to not only have flexible cloud resources but entirely flexible whole clouds, and to do that, the flexible whole cloud parceling, they had to do network engineering.21:03
clarkbfungi: ok, email sent to osic about the dns thing I cc'd you21:03
fungiright, average time to production for consuming donated cloud environments has been improving, but is still probably a 1-month minimum21:03
annegentl_parcelling? I dunno21:03
anteayaannegentl_: yup21:04
anteayait is a utopia21:04
anteayaand I like it21:04
fungithanks clarkb!21:04
anteayabut we have a whole bunch of work to get there21:04
*** jsavak has quit IRC21:04
*** weshay has joined #openstack-infra21:04
fungii feel a lot better if they solve that than keeping a workaround in /etc/hosts on nodepool.o.o forever21:04
clarkbfungi: indeed21:04
* clarkb is currently etc hosting on laptop and using "verified" connections21:05
anteayafeel comfy and safe?21:05
clarkbI mean21:06
*** david-lyle has quit IRC21:08
geekinutahnibalizer: I work for Mirantis, but this research is unrelated to them21:08
nibalizergeekinutah: ok21:09
nibalizerlet us know what information you need21:09
nibalizerwe'll be happy to work with you21:09
anteayageekinutah: and whenever you can reveal a group name we can reference among ourselves that would help us21:11
geekinutahthanks, I will for sure21:11
anteayasince right now it is, the thing geekinutah is asking about21:11
anteayawhich is a bit long for a title21:11
anteayageekinutah: thanks for helping :)21:11
fungii don't mind not knowing who is working where, but being able to tell who is working together is useful21:11
geekinutahwell, include AndyU in that long title21:11
anteayatrue that21:11
anteayageekinutah: ah wondered if this was the same topic21:12
geekinutahwe will try and be public soon, just working through internal approvals yada yada21:12
fungiyep, knowing this is related to something maybe already talked about helps avoid a lot of repetition when we don't know how briefed someone is on something21:13
geekinutahcompletely understood, will fix soon21:13
*** korzen has joined #openstack-infra21:13
clarkbfungi: see dns response? if I still worked at intel I coukd fix this21:14
*** ldnunes has quit IRC21:14
fungiall of the jenkins masters except jenkins01 are caught up for configs now. 01 is hitting some proxy errors and likely could benefit from a shutdown/restart so i'll do that21:14
anteayafungi: ack, thanks21:14
anteayageekinutah: thank you21:15
clarkbI had root on this ns21:15
fungiclarkb: um, that's not the domain in question though?21:15
clarkbfungi: no sounds like they arent doing anything for current cn21:15
clarkband are going to use that other domain21:15
clarkbI suppose we can ask for current cn21:16
fungiclarkb: so... they deployed with a cert for which they never intended to have working dns, and want to replace the cert and use a different dns name in a domain they don't control?21:16
fungithat just seems odd21:16
fungihogepodge: ^ your osic friends and their choices21:17
*** tphummel has quit IRC21:17
lifelessAJaeger: I need to catch up on the status of the various discussions21:17
*** david-lyle has joined #openstack-infra21:17
lifelessAJaeger: but basically, just change tox, right?21:17
clarkbfungi: I will respond asking for the rax controlled domain to be used in interim if possible and see if I can't get the intel side expedited21:18
clarkbfungi: cloud1.osic.org does resolve but I guess they haven't been able to get a new cert for it yet21:18
fungiclarkb: cool. i guess there's some possible benefit in the near term to adding it to nodepool to take it for a spin even if we're going to need to change the endpoint name before it's in full swing21:20
*** kzaitsev_mb has joined #openstack-infra21:20
clarkbfungi: yup, the biggest risk is whether or not we feel to exposed without verifying any part of the connections21:20
fungithough if they're planning to not use a self-signed cert, maybe we don't need the extra hassle of adding the temporary trust for it21:21
fungiwould have just been courteous of them to let us know up front that this wasn't a completely baked deployment21:21
fungiset better expectations21:21
clarkbI think they did mention it was a work in progress but ya I didn't realiez that they didn't even have dns working21:22
*** abregman_ has joined #openstack-infra21:22
*** tphummel has joined #openstack-infra21:22
fungiand didn't deploy it with the cert they were planning to use longer-term21:22
fungijenkins01 does indeed appear to be suffering from the typical thread leak issue21:23
*** abregman|nb has quit IRC21:24
*** annegentl_ has quit IRC21:25
*** annegentl_ has joined #openstack-infra21:28
clarkbit seems to be a function of load/jobs run and not time21:28
clarkbI haven't heard from upstream since my last round of updating the bug with info on how we run into it21:28
fungiclarkb: i have a feeling our older jenkins masters (particularly 01 and 02 but also to a lesser extent 03 and 04) perform worse than more recent ones and may be hitting this with a proportionally greater frequency as a result21:29
clarkbya 01 and 02 do seem to have the most trouble with large jjb updates21:30
fungijjb run-times across each of them are dramatically graduated21:30
*** sripriya__ has joined #openstack-infra21:31
*** dims_ has joined #openstack-infra21:31
*** sridhar_ram has joined #openstack-infra21:32
*** korzen has quit IRC21:32
*** yamahata has joined #openstack-infra21:32
clarkbfungi: so here is what  Ithink I will do any you can comment if it is terrible idea or not. Update puppetmaster's /etc/hosts to have entry for current osic cert CN. Launch mirror with the self signed cert verifying itself. This should get us "verified" connections when bringing up a mirror host and make sure all the infrastructure pieces are there, but hold off on adding to nodepool until we can do21:32
clarkbit properly21:33
*** dims has quit IRC21:33
*** sridhar_ram1 has quit IRC21:33
fungino objection from me. we've seen no indication that they plan to tear down and rebuild the environment so should be fine to employ minimal workarounds like that to parallelize the setup work on our part and on their part21:33
ianwclarkb / jhesketh : so i guess images didn't get up to rax -- https://jenkins06.openstack.org/job/gate-tempest-dsvm-platform-fedora23-nv/18/console (doesn't have the timestamp at the top http://nodepool.openstack.org/dib.fedora-23.log <- build was ok)21:34
*** sripriya_ has quit IRC21:34
clarkbianw: yes we are still failing to upload reliably there21:34
fungias you pointed out earlier, the afs client mirrors are not holding any secret/privileged data we're transferring over that connection anyway21:34
clarkbianw: we now attempt to reup the token but swiftclient complains about missing tenant id21:35
*** [1]Thelo has joined #openstack-infra21:37
*** jpr has joined #openstack-infra21:39
*** Thelo has quit IRC21:40
*** [1]Thelo is now known as Thelo21:40
anteayaclarkb: how is now for the neutron default security groups discussion?21:40
clarkbthe error you get when doing a non insecure conection against unverifiable cloud is fun. Could not determine suitable url for plugin21:41
clarkbanteaya: works great21:41
anteayaI'm asking you first then will see how the timing is for neutron folks21:41
anteayaawesome let's see who I can round up from neutron21:41
*** derekh has joined #openstack-infra21:41
anteayasc68cal: awesome21:42
anteayathank you21:42
ianwclarkb: in shade?21:42
anteayalet's just see if kevinbenton and dougwig are also available21:43
kevinbentoni am!21:43
sc68calsounds good21:43
anteayaso to set the stage21:43
anteayainfra created an infra cloud21:43
*** Sukhdev has quit IRC21:43
anteayaand has some user stories from that experience21:43
anteayaone of the involves neutron default security groups21:43
anteayaand clarkb has more details21:43
*** baoli has joined #openstack-infra21:43
*** rguillebert has joined #openstack-infra21:44
clarkbianw: it is mostly os-client-config and swiftclient I Think21:44
clarkbright so security groups21:44
anteayaI don't believe rcarrillocruz is around at this time unfortunately, though he also has some thoughts21:44
clarkbthere are two general issues. The first is that the defaults don't let you do anything, and the other is that the defaults include broken rules by default (inter group rules)21:44
*** baoli_ has joined #openstack-infra21:44
clarkbso for every project we create I have to delete 2 rules and add 2 new ones. Always21:45
kevinbentonclarkb: what do you mean they are broken rules?21:45
openstackgerritRyan Beisner proposed openstack-infra/project-config: Enable verified label for charms  https://review.openstack.org/28685321:45
sc68calthe default rules IIRC are just to allow outbound21:45
ianwclarkb: this is what you mostly referring to? https://review.openstack.org/#/c/255623/21:46
clarkbkevinbenton: intergroup rules eg default group members can talk to default are broken21:46
dougwigclarkb: fwiw, i agree and  hate the defaults.21:46
clarkbkevinbenton: they put strain on the db or something and results in clarkb getting 2am phone calls from clouds21:46
anteayayay dougwig is here too21:46
clarkbkevinbenton: so now as a rule I always delete them21:46
dougwigclarkb: #1 cause of user complaints.21:46
clarkbbecause I hate 2am phone calls21:47
kevinbentonclarkb: i don't understand. is it that there is a performance issue or is that they don't actually do what they are supposed to?21:47
*** maishsk has quit IRC21:47
mtreinishinfra-root: I'm gonna keep bugging about: https://review.openstack.org/286733 and https://review.openstack.org/281383 so we can get to the bottom of why the subunit worker is getting stuck21:47
mtreinishthe result collection has been down for ~2 weeks and I'd really like to sort this soon21:47
nibalizermtreinish: ok21:47
mtreinishnibalizer: thanks21:48
*** watersoul has quit IRC21:48
clarkbkevinbenton: they do what they are supposed to, but you can apparently kill the clouds using them21:48
clarkbkevinbenton: which is why we now have quotas around rules?21:48
*** baoli has quit IRC21:48
clarkbkevinbenton: so we have attempted to work around the issue but haven't actually fixed it so 2am phone calls still a potential problem21:48
*** watersoul has joined #openstack-infra21:49
kevinbentonclarkb: hmmm. you may have stumbled onto a bug. the default rules should not result in killing the cloud21:49
kevinbentonclarkb: the quota is just because tons of rules will eventually choke iptables21:49
*** |-paul-| has quit IRC21:49
kevinbentonclarkb: but a basic query for members should not wipe out the server21:50
*** sdague has quit IRC21:50
kevinbentonclarkb: how many members are there of the group when it falls over?21:50
clarkbkevinbenton: it had to do wit hthe number of members in the group21:50
kevinbentonclarkb: :)21:50
clarkbkevinbenton: when I got the 2am call we had ~600 members21:50
kevinbentonclarkb: ok. i will see if i can repro21:50
fungikevinbenton: it's possible this has been fixed in neutron. there was a time when having several hundred instances resulted in a combinatorial matrix of rules for every instance pairing in iptables21:50
kevinbentonfungi: yes, that should be long gone21:50
clarkbour/my response at the time was to remove the useless rules completely21:50
fungikevinbenton: and then adding and deleting instances from the group rapidly would cause enough churn to bring the accounting to its knees21:50
kevinbentonfungi: ipset fixed that on the iptables side21:50
kevinbentonfungi: and sane queries fixed the server side (I thought)21:51
kevinbentonwhat version of neutron is this?21:51
clarkbit was whatever hpcloud was running21:51
fungikevinbenton: kilo21:51
fungioh, hpcloud21:51
funginot what we deployed21:51
clarkbright 2am phone call was hpcloud21:51
*** abregman has joined #openstack-infra21:52
fungii mean, from our perspective those are irrelevant default rules anyway, because we want default allow everything to and from everywhere, so additional rules allowing more specific sources/destinations are not doing anything21:52
*** jpr has quit IRC21:52
kevinbentonfungi: ok21:52
sc68calThe defaults already allow half that - the outbound piece21:52
sc68calso really it should just be two rules you add, to allow inbound on v4 and v6 ?21:53
kevinbentonfungi: so unfortunately the thread in the past about operator-configurable defaults did not end well21:53
kevinbentonfungi: because it results in different experiences per cloud21:53
fungiwe assume we are representative of at least a significant slice of the userbase who prefer to do traffic filtering on individual instances interfaces and drive that via configuration management, rather than centrally in a fake network firewall in the cloud provider's network21:53
kevinbentonfungi: fake firewall?21:54
*** fawadkhaliq has quit IRC21:54
sc68calfungi: I can think of a significant openstack deployment where that is not allowed, by policy, for compliance reasons21:54
kevinbentonfungi: what would you do it with on the instance?21:54
fungibut i get that there are likely others who would prefer a default block everything ruleset in the security rules in neutron/nova21:54
kevinbenton(nevermind, not fighting about whether or not iptables is fake) :)21:54
kevinbentonmay i suggest just setting port_security_enabled to False on the network these VMs are attaching to?21:54
clarkbright so the other problem is literally on every cloud I have to update the rules21:55
clarkbthis tells me that our defaults are broken21:55
fungikevinbenton: fake network firewall (e.g. not dedicated hardware) vs host firewall on the instance21:55
fungifake was probably not the word i wanted there21:55
kevinbentonit sounds like you don't want any port security21:55
fungivirtual network firewall perhaps21:55
*** rhallisey has quit IRC21:55
*** austin81 has left #openstack-infra21:55
clarkbideally ( and this may not be possible ) our defaults should capture a reasonable subset of what users need so that they don't have to customize everything21:55
fungikevinbenton: we use iptables on the hosts for port security21:55
fungion the instances i mean21:55
*** abregman_ has quit IRC21:55
fungiwithin each instance's operating system21:56
kevinbentonack, so are the hosts 100% trusted in these cases?21:56
sc68calclarkb: if the security group API extension is not useful, it is just an extension and you can disable it in your neutron deployment. I don't know anything about infra cloud so I don't know if you can make that decision21:56
clarkbsc68cal: I cannot disbale it in most of the clouds I use21:56
clarkbsc68cal: I am not talking infra cloud21:56
*** annegentl_ has quit IRC21:56
clarkbsc68cal: I am talking generally this is a problem on every single cloud I have used21:56
clarkb(except for rackspace21:56
sc68calok, then what about kevinbenton 's suggestion about disabling port security21:57
clarkband it occurs to me that if users have to "fix" this every time they use a new cloud we probably aren't doing what the users need21:57
kevinbentonclarkb: we match the defaults of AWS IIRC21:57
*** annegentl_ has joined #openstack-infra21:57
anteayaoh sorry perhaps I set the inital story incorrectly21:58
kevinbentonclarkb: (i think that's the history of where they came from)21:58
anteayaI apologize if I created confustion21:58
anteayaor confusion21:58
*** rhallisey has joined #openstack-infra21:59
dougwigkevinbenton: so we value uniform awfulness over operator configurable maybe not awfulness?21:59
* dougwig hides.21:59
anteayaokay so perhaps doing what AWS has passed the best before date?21:59
*** abregman is now known as abregman|nb21:59
dougwignova-net by default is open, right?21:59
sc68calno it is not21:59
clarkbI also field a lot of questions on the Internet about "I am using neutron and cannot ping or ssh to my instance"21:59
kevinbentondougwig: so we can change it, but I'm surprised to hear this coming from you since it creates insecure VMs by default :)21:59
clarkbmost of the time the problem there is "edit your security groups"22:00
kevinbentondougwig: i recall a signnificant bikeshed around DNSMASQ leaking queries...22:00
kevinbentonso how was this dealt with with nova-net?22:00
clarkbkevinbenton: I am not sure that more open security groups implies insecure VMs22:00
dougwigkevinbenton: i am never opposed to handing operators a gun and a lot of ammo. it becomes their choice which foot they shoot.22:00
sc68calclarkb: these issues are basically the difference between people who used only bare metal and expect certain behaviors versus cloud22:00
*** pcaruana has quit IRC22:01
sc68calI've had this conversation multiple times - the issue is the security group default behavior is usually part of a security policy dictated by other parts of the organiation, and also matches a "secure by default" behavior22:01
kevinbentondougwig: it's not the operators, it's the users22:01
kevinbentondougwig: i use one cloud, i get protected VMs by default22:01
kevinbentondougwig: i use another, i get exposed VMs by default22:01
sc68cal^ this too22:01
clarkbkevinbenton: uh more like "possibly protected but hard to confirm at all times"22:01
clarkbwhich is another separate issue that seems to get fixed every 6-12 months22:02
kevinbentonclarkb: i'm not following22:02
clarkbkevinbenton: there have been several vulnerabilities where security groups don't actually apply22:02
clarkbkevinbenton: which is a big reason for using instance local iptables22:02
*** dims_ has quit IRC22:02
sc68calthat's FUD, and trolling22:02
clarkbit is not22:02
dougwigkevinbenton: so, at AWS it's explicit, and at DO it's open by default.  and never have I, as a user, been confused.22:02
anteayalet's stick with details22:03
clarkbas a cloud user you have little insight into whether or not a seucrity group rule is actually working22:03
clarkbyou can test a point in time22:03
clarkbhowever as a cloud user running your own local firewall you can inspect state22:03
sc68calclarkb: if a security group rule is applied to a group, and a VM is part of that group, it will be applied, Hence we return the apropriate HTTP code to the API request22:03
openstackgerritDoug Hellmann proposed openstack-infra/project-config: update django_openstack_auth to use publish-to-pypi jobs  https://review.openstack.org/28674722:04
*** fawadkhaliq has joined #openstack-infra22:04
*** Sukhdev has joined #openstack-infra22:04
kevinbentonok. as i understand it, we don't trust neutron to do what the API says22:04
kevinbentonso shut off port security and call it a day22:04
kevinbentonproblem solved22:04
mtreinishugh, connreset on centos apply job: http://logs.openstack.org/33/286733/1/gate/gate-openstackci-beaker-centos7-dsvm/0e5f1e0/console.html22:04
clarkbkevinbenton: how do you turn it off as a user?22:05
mtreinishthat just added like another 2 hrs until that patch can merge now.. :(22:05
kevinbentonneutron net-create mynet --port-security-enabled=False22:05
clarkbkevinbenton: and if I cannot create my own networks?22:05
sc68calor do a port-update on a port to set port security to false22:05
kevinbentonclarkb: well if you don't own the networks then you can't do this22:05
sc68calor create, iirc - kevinbenton can correct me if wrong22:05
kevinbentonthis won't work for networks you don't own22:06
clarkbright in several of our clouds we get the the networks we get22:06
kevinbentonbecause it turns off all kinds of filtering22:06
dougwigthat nonsense being the burden of end users is... nonsense.  bleh.22:06
openstackgerritMerged openstack-infra/system-config: Add a job filter for old side subunit files  https://review.openstack.org/28138322:06
anteayadougwig: can you expand?22:06
kevinbentonclarkb: ok. so if you don't control the cloud, how do you propose the default rules be changed?22:06
kevinbentonclarkb: are you suggesting for Neutron to change it's default rules for everyone?22:07
clarkbkevinbenton: yes22:07
clarkbat the very least I think it is reasonable to pass through protocols like ssh22:07
fungipart of the challenge, i think, is that because default security groups are configurable by the provider, every provider decides to configure them differently, so step 1 in hooking up to a new cloud provider is to figure out what random things they've decided to block on your instances22:07
clarkbbecause without that you aren't doing much with your instances22:07
fungifor example, no gre in ovj22:07
kevinbentonfungi: they can't with neutron though22:07
clarkbfungi: with neutron you can't change the defaults that new projects get22:08
*** dprince has quit IRC22:08
clarkbfungi: I think the gre thing is a weird outlier22:08
clarkb(which probably should get solved, it just isn't directly related)22:08
sc68calallowing inbound SSH to every instance is .... very aggressive22:08
fungikevinbenton: that seems counter to our experience. i think internap is using neutron and they just recently adjusted their global default security groups at our recommendation22:08
sc68calI think someone from a security would flip out22:08
fungibecause they were allowing all tcp and udp, as if it were the internet22:08
kevinbentonfungi: they changed code then22:08
*** thorst_ has quit IRC22:09
fungikevinbenton: i wouldn't be surprised, though mgagne probably has more details on that22:09
clarkbsc68cal: there is a balance between conservative rules to protect instances and useable so that users don't have to fight their clouds22:09
dougwigclarkb, fungi: how is the neutron experience different (and worse) than the nova-net default closed rules?22:09
clarkbdougwig: nova-net allowed providers to change the defaults and every nova net cloud I used allowed ssh22:10
clarkbdougwig: neutron does not allow this and I have to edit the rules in every new project22:10
kevinbentonfungi: https://github.com/openstack/neutron/blob/master/neutron/db/securitygroups_db.py#L169-L18522:10
*** dims has joined #openstack-infra22:10
sc68calis that good though? different clouds having different defaults?22:10
kevinbentonclarkb: but i thought you didn't control the clouds? In this case would you ask the operator to change their defaults?22:10
clarkbsc68cal: not necessarily, I think the goodness is in having a default that is sane for users22:10
fungiright, i'm not convinced configurable defaults is necessarily good since it creates interoperability rifts22:10
sc68calclarkb: ok - and I guess I think allowing inbound everything by default is not sane.22:11
clarkbkevinbenton: no I think my goal would be to make a default ruleset that is good for users22:11
clarkbsc68cal: I am not suggesting that22:11
*** jamesmcarthur has joined #openstack-infra22:11
clarkb(I mean that would be even better for us specifically but I can see the arguments against it too)22:11
anteayalet's stay focused on finding some common ground here22:11
anteayaI think it is possible22:11
fungiright, where the previous discussions ended up at was that nobody could agree what ports were reasonable for initial ingress by default, so the decision was to just allow no ingress at all and make everyone change it before they could reach anything22:12
kevinbentonfungi: +1. this is where it ultimately lands whenever we repaint this shed22:12
dougwigwhat do the operators want and/or do?  do they just edit neutron to specify defaults anyway, regardless of our ideology on this?22:12
*** tiswanso has quit IRC22:12
fungibasically there's no one config that's good for everyone, so install a config that's consistently bad for everyone22:12
anteayaI don't know as we have asked the operators community22:12
sc68calIs there a reason those rules need to be in the default security group? Why not create a security group with the rules that you need, and create them - heat template/provisioning system/ etc22:13
sc68calwhatever you use22:13
anteayaI just thought some benefit might come out of infra sharing their use cases22:13
*** abregman|nb has quit IRC22:13
clarkbsc68cal: because I think we should provide useable defaults to the majority of our uesrs without fiddling extra knobs22:13
kevinbentonThe issue I see now is that if we change this it's a big behavior change on upgrade22:13
*** dkranz has joined #openstack-infra22:13
sc68calclarkb: believe me, I am a fan of that idea, but the usable default we've got right now is the best compromise I think we have, allowing outbound and blocking inbound22:13
clarkbcurrent situation is as fungi says consistently bad for everyone so none of your users win22:13
sc68caland educating users to bring a security group and rules along with images and whatever other stuff they need for the cloud, holistic solution22:14
fungibecause making something easier for a majority of users is unfair if you can't solve it for everyone, so punish them all22:14
sc68calit's not just "upload your image and off we go"22:14
*** shardy has quit IRC22:15
sc68calfungi: I also do not believe we are "punishing all" users22:15
anteayaokay so we might not come up with a solution today22:15
kevinbentonWe're sort of backed into a corner. I don't see how we can swap the default because it's a major change in the end-user expectations of filtering22:15
fungihyperbole, yes. i apologize22:15
anteayabut I think we are gaining something by sharing our thoughts with each other22:15
dougwigAWS puts this closure in your face initially, forcing its discovery.  other big public clouds (DO, rax) are open by default.  the magic inability to ping without knowledge is indeed kinda ... not good.22:15
kevinbentondougwig: so you are proposing a change to horizon :)22:16
dougwigit's not that i want it open by default, per se.22:16
clarkbdougwig: right icmp and a small set of services that are consistently necessary to use the cloud seem like a good compromise22:16
anteayaI think we have consencous that the current situation isn't ideal22:16
*** sridhar_ram has quit IRC22:16
fungii also disagree and think icmp echo should be allowed to any host which is participating on the global internet, but i get that there are people who still fear ping-of-death will rear its ugly head 20 years later, or think that security by obscurity is good for breakfast22:16
anteayaand that as of yet we don't have agreement on a clear way forward22:16
anteayabut it sounds like we would like one22:16
sc68calI think icmp echo inbound default is something I'd be willing to give ground on22:17
kevinbentoni think the next step is for clarkb or fungi to file an RFE on neutron so these complaints are visible to the rest of the neutron cores22:17
kevinbentonand so they can see it's coming from infra22:17
anteayause cases maybe instead of complaints?22:17
fungiwell, all icmp really. any operating system which has insecure handling of any icmp echo type is downright silly in this day and age22:17
fungiand blocking arbitrary control and messaging types leads to all sorts of terrible black-hole effects22:18
kevinbentonI think allowing ICMP and blocking others may even be more misleading if users don't understand that there is filtering between them and their VM22:18
*** sarob has quit IRC22:18
clarkbit is also particularly detrimental for fragmentation when dealing with smaller mtus which is a side effect of using neutron in the first place22:18
kevinbentonat least now nothing inbound works22:18
fungigrr, why did i write icmp echo type? icmp type22:18
*** Sukhdev has quit IRC22:19
*** sridhar_ram has joined #openstack-infra22:19
*** dimtruck is now known as zz_dimtruck22:20
clarkbin other news I have a clouds.yaml that works on one host but not another22:20
fungianyway, yes, all ic protocols and ports for egress, all ip protocol 1 (icmp) types ingress, tcp ports 22 (ssh) and 3389 (rdp), and make it known that base images with password auth are discouraged22:20
fungigah, ip protocols22:20
fungii should give up on keyboards today22:21
sc68calwhoa whoa no 22 or 338922:21
fungisc68cal: how do you reach your instances?22:21
clarkbthen the other semi related item we have run into is that other tcp types seem ot be forgotten and are not reliable (eg GRE)22:21
*** keedya has quit IRC22:21
sc68calfungi: I don't allow 22 or 3389 from the whole internet.22:21
fungiout of band console? vpn tunnel?22:22
sc68calfungi: I provide a cidr of known good addresses.22:22
sc68caland only that cidr22:22
fungisc68cal: the good news is, you can fix that in your security groups before you nova boot!22:22
*** alivigni has quit IRC22:22
fungii hear neutron makes that pretty easy22:22
sc68calI am absolutley not a fan of allowing 22 or 3389 by default - that is way out of bounds22:23
fungithose seem like the bare minimum to me22:23
clarkbmost of your users are going to need that by default22:23
*** achanda has quit IRC22:23
hasharfungi: I dont buy the ICMP flood argument either. IIRC lot of ISP on internet handle them on a best effort basis so a huge flood of ICMP should see a nice packet loss on the way to the target22:23
sc68calHaving those by default was a great way to get your instances deleted for compliance reasons where I used to work22:23
fungihashar: yep, icmp response throttling is pretty commonplace for decades now22:24
fungisc68cal: this is probably a clash between people who want to participate in the internet (and need to be this tall to ride) vs people who want to run private enterprise networks and get to them through something internet-connected22:25
*** sarob has joined #openstack-infra22:25
sc68calThis was not private enterprise networks.22:25
sc68calyou got a v6 address, by default. You were on the internet.22:25
sc68calso the whole enterprise vs "public cloud" is a red herring22:26
fungiequating to traditional non-virtualized environments, when i paid for a colo i got an internet drop from a stateless, non-filtering routed gateway. all decisions from the end of that cambe down were mine and the responsibility for securing the environment from outside threats was mine alone22:27
*** sarob has quit IRC22:27
fungii also ran colocation facilities for many years, and that's precisely how we operated22:27
sc68calcloud != colo22:27
*** krtaylor has quit IRC22:27
fungicloud: somewhere to run my virtual servers; colo: somewhere to run my physical servers22:27
*** sripriya has joined #openstack-infra22:27
anteayaokay I'm starting to feel like this discussion is less productive than it might be22:28
anteayawe seem to have some emotional attachment to things that I wasn't aware of, sorry about that22:28
anteayathanks for taking some time to chat about this, I appreciate it22:28
anteayaperhaps we should maybe come back to this again some time?22:29
anteayathanks for the additional details on both sides22:29
fungianteaya: it likely boils down to there being extremes of opinion between network engineers, some who feel the internet should be open by default, others who feel the internet should be closed by default22:29
anteayaI certainly learned some things I didn't know before22:29
anteayafungi: yup, we seem to have some differences here22:29
*** david-lyle has quit IRC22:29
anteayaand I apologize for not knowing in advance how large the gap was22:30
anteayasorry about that folks22:30
anteayaI value you all22:30
anteayayou are amazing people22:30
*** yamahata has quit IRC22:30
anteayathank you for your time today22:30
sc68calfungi: that viewpoint is not uniform across network engineers.22:30
*** sripriya__ has quit IRC22:31
fungii'm sure there is a gradient of opinions, and possibly middle ground somewhere22:31
fungii was more defining the spectrum22:32
anteayaso let's come back to this another day22:32
anteayaand see if we can make more progress finding that middle ground22:32
*** annegentl_ has quit IRC22:32
sc68calagreed - middle ground would be good22:33
fungii mean, i'm thrilled that neutron doesn't also block all egress by default. but since i always have to change the security groups anyway, i guess it wuold be adjustable at the same time as opening up ingress22:33
ianwclarkb / greghaynes : i'm trying to get some insight into the rax upload issue at a low level, without nodepool on top.  i'm doing something like http://paste.openstack.org/show/488860/ to simulate uploading several images.  am i going to see the issue, or is it only when timeouts occur?22:33
*** verdurin has quit IRC22:33
ianwor do i need to do the uploads in parallel threads?22:34
clarkbianw: you should see it break by doing that if you have a long period between uplodas (longer than the token expiry)22:34
clarkbianw: basically we need swiftclient to get a new token after the old one expires and that isn't reliable yet22:34
ianwclarkb: how long is token expiry?22:34
*** verdurin has joined #openstack-infra22:34
clarkbianw: it varies between clouds I think rax is 24 hours?22:34
clarkbdon't quote me on that, it is long enough that first upload after a service restart works but second upload a day later may not22:35
clarkbianw: you might be able to game it by updating time.time to be a day later22:35
clarkbnot sure how much of the verification is on the server vs client side for expiration checking22:35
fungiokay, second pass of jjb update with jenkins01 in prepare for shutdown managed to complete without further error22:36
jrollpretty sure rackspace is 24h22:36
*** achanda has joined #openstack-infra22:36
*** angdraug has quit IRC22:36
fungiand one job left to finish before i can clean up and restart jenkins01 now22:36
clarkbsc68cal: kevinbenton: right so understanding that there is a spectrum of opinions here I think the thing to think about is whether or not there is a position that can address the default needs of your "default" users and maybe make people that need fancy to do the fancy themselves. This is in contrast to current setup where everyone must do the fancy regardless22:37
*** hashar has quit IRC22:38
sc68calclarkb: I agree with the sentiment of everyone having to do all the fancy themselves, and yes I want to reduce the tedium of getting a new tenant up and running22:38
*** aysyd has quit IRC22:38
sc68calI just think ssh and rdp inbound by default is a non-starter22:39
fungithe other feature request we might want to write up (and i don't know how deeply ingrained in neutron's design this is to even be feasible) is an option to disable filtering _and_ state tracking for a network22:39
*** dizquierdo has joined #openstack-infra22:40
fungibecause really, while we can say neutron i want to allow "connections" to and from a range of ports and all protocols, whatever, if the hosts aren't configured to properly handle state for obscure protocols they'll still end up breaking22:40
clarkbright thats the GRE problem22:41
fungie.g. loading appropriate conntrack modules on the hosts22:41
fungi_if_ the kernel weren't explicitly told to cram this through stateful iptables rules, that would cease to be an issue22:41
*** jtomasek has quit IRC22:41
sc68calI know a couple instances where state tracking kills a usecase or project that was going to use neutron22:42
sc68calthe issue will be, to get rid of conntrack, you have to disable port security22:42
fungiwhich can be done on a per-network basis?22:42
*** piet has quit IRC22:43
fungii'm saying disable all filtering and also disable state tracking, so that sounds possible i guess22:43
*** rhallisey has quit IRC22:43
clarkbthe altnerative to that would be to explicitly grok more than icmp, tcp, and udp right?22:43
sc68calfungi: yeah I think disabling state tracking is a side-effect of disabling filtering at this point in time22:43
fungii get that it's pretty hard to deal with stateless filtering rulesets (i did it for more years than i care to think, but do not wish to relive that)22:43
*** annegentl_ has joined #openstack-infra22:44
sc68calfungi: yeah I don't think stateless filtering would be fun to go back to :)22:45
*** zz_dimtruck is now known as dimtruck22:45
fungiyay! jenkins01 idle. cleaning up and restarting now22:45
fungisc68cal: so is "disabling port security" (e.g. running a neutron network with no packet filtering and no state tracking) already a feature we can make use of in neutron, or is that a feature request/spec?22:46
kevinbentonfungi: you can use it if you own the network22:47
kevinbentonfungi: but not if you don't22:47
fungiokay, that's a start22:47
kevinbentonfungi: because it shuts off everything including anti-spoofing features22:47
sc68calfungi: I think we'd just need to verify that no state is being put into conntrack on the hypervisor host, my suspicion is that it it already does.22:47
fungiand by "own the network" you mean neutron network create blah22:47
funginot completely control the neutron deployment in the openstack cloud22:47
*** amitgandhinz has quit IRC22:48
fungithat seems like something we may want to give a shot22:48
sc68calI forget what release we added the port security api ext22:48
clarkbgah new problem in osic22:49
sc68calbut it's optional... sooooo we're still kind of at square one22:49
fungithough provider networks are probably going to fall in the category of "not controlled by us"22:49
kevinbentonkilo or juno22:49
clarkbthe catalog uses the IP address22:49
anteayaclarkb: :(22:49
clarkbwhich means even /etc/hosts update is almost useless22:49
clarkbthere goes that idea22:49
fungiclarkb: oh wow22:49
clarkbfungi: I think our option is to not verify or to wait22:49
fungithat's a complete non-starter22:49
kevinbentonfungi: are you seeing the rules interfere with GRE even if you add GRE rules bi-directionally?22:49
clarkbkevinbenton: we add "allow all ip rules"22:49
clarkbkevinbenton: and that isn't sufficient22:49
fungikevinbenton: does neutron have a list of other ip protocols that have to be worked around in such a fashion?22:50
fungior the kernel's iptables/conntrack documentation covers this maybe?22:50
sc68calGRE is udp - what exactly is your "allow all IP rule"22:51
fungigre is not udp22:51
fungigre is ip protocol 4722:51
sc68calsorry, brainfart22:51
fungiudp is ip protocol 1722:51
fungithey do share a digit122:52
sc68calSo, what protocol is your security group rule22:52
sc68calfor allow all ip22:52
*** mriedem has quit IRC22:52
clarkbsc68cal: whatever the rule create command does for not specigying a type22:52
kevinbentonfungi: so there was a change at some point to make sure allow rules were hit before the ALLOW ESTABLISHED rule that punted to conntrack22:53
*** rbrndt has quit IRC22:53
*** dingyichen has joined #openstack-infra22:53
fungikevinbenton: oh, so this may simply be that provider not running a new enough neutron to have that?22:53
sc68calclarkb: ok, I don't remember that off the top of my head - but if you can give us the actual security group rule, it'll list the protocol22:54
*** baoli_ has quit IRC22:54
ianwclarkb: fyi, resetting the system clock into the future does not seem to expire the token22:54
*** baoli has joined #openstack-infra22:54
kevinbentonfungi: i'm not 100% sure of the behavior of iptables when it's missing a module to process a particular protocol22:54
kevinbentonfungi: but it may fix it22:54
kevinbentonfungi: let me find the commit so i can get a release it's present in22:54
*** jamesmcarthur has quit IRC22:55
*** andymaier has quit IRC22:55
fungithis was observed in ovh's environment, which drove us to switch from gre to vxlan for the overlay in devstack multi-node scenarios22:55
*** dims has quit IRC22:55
kevinbentonfungi: i mispoke, it changed it so dropping INVALID packets would happen after user-defined rules22:56
kevinbentonfungi: but it may still apply22:56
fungiwhich also has potential issues since broadcast over vxlan ends up being multicast ip which also won't really work correctly22:56
kevinbentonfungi: this change is in 7.0.3 7.0.2 7.0.1 7.0.022:56
kevinbentonfungi: https://github.com/openstack/neutron/commit/0a258afc7ee3c03974dffa2c0dd0b7b367034cc7#diff-abf220de4c2165d9e5bfd6dde12b3f4f22:56
fungikevinbenton: cool, so liberty or later?22:57
kevinbentonfungi: looks like it22:57
fungithanks--that's useful information22:57
kevinbentonfungi: do you have control over the agents running vxlan?22:57
fungikevinbenton: yeah, they're in devstack-running nova instances22:57
kevinbentonfungi: i take it you are running linux bridge?22:58
*** baoli has quit IRC22:58
fungithey form the fake lan between devstack hosts since we don't have control of the actual lan in our providers22:58
*** baoli has joined #openstack-infra22:59
kevinbentonfungi: well i ask because that multicast behavior can be stopped23:00
*** sc68cal has quit IRC23:00
kevinbentonfungi: and flood to unicast tunnels instead23:00
*** harlowja_at_home has quit IRC23:00
*** baoli has quit IRC23:00
fungikevinbenton: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt23:00
*** david-lyle has joined #openstack-infra23:00
*** sridhar_ram has quit IRC23:00
*** dims has joined #openstack-infra23:00
anteayakevinbenton: thanks for the ml post23:01
fungiclarkb: do we have a pending change to s/gre/vxlan/ on that?23:01
*** baoli has joined #openstack-infra23:01
kevinbentonanteaya: no prob23:01
*** sridhar_ram has joined #openstack-infra23:02
*** asalkeld has left #openstack-infra23:02
clarkbfungi: we do not, but we should !23:02
clarkbI am going to take the password file lock now23:02
fungikevinbenton: so anyway, we don't use the "linuxbridge" driver for neutron to set that up (or neutron at all to set that up), but we do use the bridge driver in the linux kernel23:02
fungiclarkb: i can update that, just need to know if anything more substantial than encapsulation changed when we switched the setup to vxlan23:03
*** dingyichen has quit IRC23:03
clarkbfungi: no that was it23:05
fungicool, patch on the way in moments before i forget23:05
fungikevinbenton: so sounds like there is an option we can tweak in the vxlan kernel config to switch from multicast to unicast flood?23:06
fungiif i understand what you were suggesting23:06
*** tpsilva has quit IRC23:07
*** ianw has quit IRC23:07
kevinbentonfungi: sorry, got pulled away for a sec23:08
kevinbentonfungi: yeah, if you are using linux bridge agent, clear the setting for vxlan_group23:09
*** doug-fis_ has joined #openstack-infra23:09
*** dingyichen has joined #openstack-infra23:09
*** doug-fis_ has quit IRC23:09
kevinbentonfungi: that will ensure the 'group' parameter is not passed to the vxlan links created23:09
*** doug-fis_ has joined #openstack-infra23:09
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix on Blowfish Password test  https://review.openstack.org/28687723:09
kevinbentonfungi: then the traffic should be flooded to all tunnels when it's broadcast/multicast23:10
*** rfolco_ has quit IRC23:11
*** baoli has quit IRC23:11
*** zhurong has joined #openstack-infra23:11
*** smarcet has joined #openstack-infra23:11
clarkbok releasing lock on passwords file23:11
clarkbtaking hiera lock now23:12
openstackgerritJeremy Stanley proposed openstack-infra/devstack-gate: Update multinode setup doc to VXLAN  https://review.openstack.org/28688023:12
*** doug-fish has quit IRC23:13
*** chlong_ has joined #openstack-infra23:13
*** zhurong has quit IRC23:14
*** doug-fis_ has quit IRC23:14
openstackgerritClark Boylan proposed openstack-infra/system-config: Use project_name not _id with OSIC  https://review.openstack.org/28688123:14
fungikevinbenton: oh! openvswitch actually, looking at the code. i misremembered23:14
kevinbentonfungi: oh, well that should be easy then. openvswitch doesn't support targeting multicast for its vxlan tunnels :)23:15
clarkbfungi: ^ 286881 fixes an osic clouds.yaml thing23:15
fungikevinbenton: great! theoretical problem averted. thanks! ;)23:15
clarkbfungi: it was linuxbridge but then when we got DVR stuff running there were people that felt strongly it should be ovs...23:15
kevinbentonfungi: np23:15
clarkbfungi: reality is it probably didn't matter all that much when using GRE but with vxlan maybe things like this are different23:16
fungisounds like yes, they are23:16
*** krtaylor has joined #openstack-infra23:16
fungiat least its default behavior avoided us spewing multicast into our providers' networks they were never going to forward23:16
*** Sukhdev has joined #openstack-infra23:18
kevinbentonthis is one of those cases where ovs's shortcomings are a feature!23:18
*** pcrews_ has joined #openstack-infra23:18
kevinbentonOVS overlays at scale are a nightmare because it lacks translation to multicast for encapped broadcast/multicast23:18
*** dingyichen has quit IRC23:19
*** pcrews__ has quit IRC23:19
*** salv-orl_ has joined #openstack-infra23:19
*** Jeffrey4l has joined #openstack-infra23:21
fungiyeah, i can see that being undesirable in a large production environment23:22
fungithinking back to the number of times i used multicast-enabled protocols23:22
*** salv-orlando has quit IRC23:22
*** dingyichen has joined #openstack-infra23:22
*** kzaitsev_mb has quit IRC23:23
*** annegentl_ has quit IRC23:24
*** annegentl_ has joined #openstack-infra23:25
kevinbentonyeah, if you don't block it, a tenant could easily saturate a network by dumping a nice 100mbps multicast stream to a network with lots of instances23:26
*** kzaitsev_mb has joined #openstack-infra23:26
*** jpr has joined #openstack-infra23:26
*** erikwilson has joined #openstack-infra23:27
fungisounds absolutely crippling23:27
*** annegentl_ has quit IRC23:30
*** erikwilson has quit IRC23:30
*** Jeffrey4l has quit IRC23:31
* clarkb does a few more hiera edits23:31
openstackgerritMerged openstack-infra/openstackid: Fix on Blowfish Password test  https://review.openstack.org/28687723:33
*** darrenc is now known as darrenc_afk23:33
*** gildub has quit IRC23:34
*** ashleighfarnham has joined #openstack-infra23:34
clarkband done releasing hiera lock23:34
openstackgerritClark Boylan proposed openstack-infra/system-config: Add vexxhost cloud credentials  https://review.openstack.org/28689523:37
clarkbfungi: ^23:37
clarkbmnaser: ^ you too may be interested in that. It is our first step in bringing in a cloud23:38
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenstackId relase 1.0.13  https://review.openstack.org/28689623:38
clarkbmnaser: once that is in we can work on editing security groups (if they are in place), getting quotas to be what you expect, build a cloud local mirror host, and run some test runs on the initial hosts23:39
fungiclarkb: mnaser: thanks!!!23:39
anteayais the ca in vexxhost canada or california?23:40
clarkbI think canada23:40
fungismarcet: 1.0.13 is working well on openstackid-dev i take it?23:40
*** david-lyle has quit IRC23:41
clarkbanteaya: your country is well represented :)23:41
*** Qiming has joined #openstack-infra23:41
smarcetfungi: i test it locally23:42
anteayawere doing something23:42
smarcetwe found a security hole23:42
clarkbanteaya: ovh has a region in your CA too23:42
smarcetand we need to release this fix asap23:42
*** mriedem has joined #openstack-infra23:42
clarkbanteaya: with this up and running something like 2/8 regions will be in canada23:42
anteayanear montreal23:42
anteayawe're doing something useful23:42
*** dimtruck is now known as zz_dimtruck23:44
*** annegentl_ has joined #openstack-infra23:46
*** doug-fish has joined #openstack-infra23:46
*** sdake has quit IRC23:47
*** doug-fish has quit IRC23:47
mnaseranteaya it is in montreal :)23:47
clarkbmnaser: I am going to go ahead and do a quick tempest baseline on the 8GB 8vcpu flavor23:47
mnaserclarkb: +1 on that review, and sure let me know if you run into anything23:48
clarkbmnaser: will do, thanks a gain23:49
*** dkranz has quit IRC23:49
fungismarcet: given the urgency of that security update, i kicked 286896 straight into the gate pipeline so it doesn't have to wait for available check resources23:49
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenstackId relase 1.0.13  https://review.openstack.org/28689623:49
fungioh, you just updated it anyway23:49
smarcetfungi: cool tnx :)23:49
mnaserno problem, i'll be up a bit late today as we have some maint stuff to do (but nothing service impacting)23:50
*** salv-orlando has joined #openstack-infra23:50
*** salv-orlando has quit IRC23:50
fungismarcet: thanks for adding more detail in the commit message!23:50
*** salv-orl_ has quit IRC23:50
*** darrenc_afk is now known as darrenc23:50
smarcetfungi: yes i updated commit message :)23:50
*** salv-orlando has joined #openstack-infra23:50
anteayamnaser: awesome23:51
anteayamnaser: may I come for a tour sometime?23:52
*** dingyichen has quit IRC23:52
mnasersure, we have 2 facilities right now (two az).  quite busy in the current few days as we're deploying a bunch of private clouds but drop me an email and we can organize something23:52
mnaseremail = my nick @ company name dot com :)23:53
*** gordc has quit IRC23:53
*** dingyichen has joined #openstack-infra23:54
fungiso... many... clouds.yamls23:54
funginext oscc feature request: composite configuration files23:55
fungiclarkb: i know you didn't start the trend, but why do we hide our project names away in hiera?23:57
anteayamnaser: nice, thank you23:57
clarkbfungi: usernames too, I honestly don't know23:57
clarkbfungi: maybe keystone treats them as privileged?23:57
fungiright, i can almost see not disclosing the username (though really, no, i'd be fine disclosing that) but even the project name? seems extreme to keep it secret23:58
clarkbfungi: we should ask mordred23:58
*** dizquierdo has quit IRC23:58
anteayamnaser: sent23:59
*** kzaitsev_mb has quit IRC23:59
openstackgerritMerged openstack-infra/puppet-subunit2sql: Add more debug logging for closed file issues  https://review.openstack.org/28673323:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!