Thursday, 2017-10-12

« Wednesday, 2017-10-11 Index Friday, 2017-10-13 »
*** Apoorva_ has joined #openstack-operators00:36
*** Apoorva has quit IRC00:39
*** Apoorva_ has quit IRC00:40
*** umbSublime has quit IRC00:45
*** umbSublime has joined #openstack-operators00:45
*** umbSublime has quit IRC00:45
*** umbSublime has joined #openstack-operators00:45
*** AlexeyAbashkin has joined #openstack-operators00:49
*** AlexeyAbashkin has quit IRC00:53
*** rebase has quit IRC00:57
*** rebase has joined #openstack-operators00:58
*** CrazyFish has joined #openstack-operators01:17
CrazyFishI'm hoping someone can help me undo a huge mistake01:21
CrazyFishAround 11 hours ago, my users started experiencing network connectivity issues01:22
CrazyFishOn investigation, I'd noticed that it had been a while since neutron-ovs-cleanup had run, and I ran it, not realizing what I was doing, and I destroyed the routers and dhcp agents for the rest of the users01:24
CrazyFishI tried to get back on track by restarting neutron-l3 and neutron-dhcp agents, but after 3 hours of namespaces being built, we were seeing some flickering connectivity. I did some digging and found that ovs-vsctl was returning a timeout. The ovs-vswitchd logs revealed a "Too many open files" error, so I tried raising the ulimit from 1024 to 655356. In order to get the change to stick, I tried restarting openvswitch-non01:30
CrazyFishbut ovs-vswitchd did not come back up01:31
CrazyFishnow neutron-l3 and dhcp agents are in a failed state, and trying to restart them just hangs with no output or logging01:31
CrazyFishDoes anyone have an idea for a next step?01:32
*** rebase has quit IRC01:36
*** sdague has quit IRC01:44
*** AlexeyAbashkin has joined #openstack-operators01:46
*** AlexeyAbashkin has quit IRC01:50
*** nikhil has quit IRC01:54
*** kukacz has quit IRC02:00
*** kukacz has joined #openstack-operators02:01
*** markvoelker has quit IRC02:39
*** junbo has quit IRC02:52
*** VW has joined #openstack-operators02:57
*** trungnv has quit IRC03:11
*** trungnv has joined #openstack-operators03:19
*** VW has quit IRC03:39
*** arnewiebalck_ has joined #openstack-operators04:08
*** gkadam has joined #openstack-operators04:09
*** markvoelker has joined #openstack-operators04:39
*** trungnv has quit IRC04:45
*** fandrieu has quit IRC04:58
*** fandrieu has joined #openstack-operators05:05
*** dmibrid_ has quit IRC05:13
*** markvoelker has quit IRC05:14
*** arnewiebalck_ has quit IRC05:14
*** gkadam_ has joined #openstack-operators05:38
*** gkadam has quit IRC05:40
*** gkadam__ has joined #openstack-operators05:40
*** gkadam__ has quit IRC05:42
*** gkadam_ has quit IRC05:43
*** gkadam has joined #openstack-operators05:43
*** threestrands has quit IRC05:48
*** Oku_OS-away is now known as Oku_OS06:08
*** markvoelker has joined #openstack-operators06:11
*** yprokule has joined #openstack-operators06:21
*** rcernin has joined #openstack-operators06:35
*** gkadam has quit IRC06:40
*** gkadam has joined #openstack-operators06:42
*** markvoelker has quit IRC06:44
*** Dinesh_Bhor has quit IRC07:01
*** Dinesh_Bhor has joined #openstack-operators07:07
*** simon-AS559 has joined #openstack-operators07:09
*** simon-AS559 has quit IRC07:14
*** simon-AS559 has joined #openstack-operators07:14
*** pcaruana has joined #openstack-operators07:17
*** gkadam is now known as gkadam-afk07:24
*** fandrieu has quit IRC07:30
*** AlexeyAbashkin has joined #openstack-operators07:32
*** tesseract has joined #openstack-operators07:32
*** simon-AS5591 has joined #openstack-operators07:33
*** simon-AS559 has quit IRC07:34
*** markvoelker has joined #openstack-operators07:41
*** fandrieu has joined #openstack-operators07:51
*** fandrieu has quit IRC07:58
*** d0ugal has joined #openstack-operators07:59
*** fandrieu has joined #openstack-operators08:00
*** fandrieu has quit IRC08:04
*** fandrieu_ has joined #openstack-operators08:04
*** dbecker has joined #openstack-operators08:05
*** fandrieu has joined #openstack-operators08:07
*** fandrieu_ has quit IRC08:09
*** markvoelker has quit IRC08:14
*** simon-AS5591 has quit IRC08:22
*** simon-AS559 has joined #openstack-operators08:22
*** gkadam-afk has quit IRC08:28
*** gkadam has joined #openstack-operators08:31
*** rmart04 has joined #openstack-operators08:32
*** derekh has joined #openstack-operators08:38
*** spectr has quit IRC08:49
admin0CrazyFish, try a restart with strace .. maybe it will tell you what might be the issue .. if too many open files, you can also try to close other applications so that file handlers are free and then try to restart it again09:07
admin0morning fello operators :)09:08
*** spectr has joined #openstack-operators09:19
*** fandrieu has quit IRC09:21
*** logan- has quit IRC09:27
*** gkadam_ has joined #openstack-operators09:36
*** fragatina has joined #openstack-operators09:37
*** gkadam has quit IRC09:38
CrazyFishI will give that a go :)09:48
*** electrofelix has joined #openstack-operators10:00
*** fandrieu has joined #openstack-operators10:07
*** spectr has quit IRC10:12
*** markvoelker has joined #openstack-operators10:12
*** spectr has joined #openstack-operators10:25
*** thingee has quit IRC10:28
*** thingee has joined #openstack-operators10:28
*** shewless_ has joined #openstack-operators10:32
*** treiz has quit IRC10:40
*** markvoelker has quit IRC10:45
*** treiz has joined #openstack-operators10:48
*** clayton has quit IRC10:49
*** clayton has joined #openstack-operators10:51
*** logan- has joined #openstack-operators10:57
*** AlexeyAbashkin has quit IRC11:01
*** AlexeyAbashkin has joined #openstack-operators11:05
*** sdague has joined #openstack-operators11:11
zioprotoCrazyFish: please, you need to tell use version of Openstack, version of the operating system, and if you are using systemd or not11:26
zioprotoCrazyFish: systemd also puts some limits11:26
arnewiebalckzioproto: Concerning the Cinder quota, where did you get the bad feedback?11:32
zioprotoarnewiebalck: see the ticket comments11:32
zioprotoarnewiebalck: Horizon does not accept RFE, anyone interested should write a BP11:33
zioprotoso, it is not like to report a bug, and there is a dev team picking up the task11:33
arnewiebalckzioproto: We filed a bp, no?11:33
* arnewiebalck checking11:33
zioprotoarnewiebalck: if we filled it, it should be linked to that bug. But it feels like it is not linked11:34
arnewiebalckzioproto: https://blueprints.launchpad.net/horizon/+spec/quotas-per-cinder-volume-type11:34
zioprotoarnewiebalck: great, I linked the bug to it now !11:35
arnewiebalckzioproto: we should try to follow up on this upstream, we’d like to get rid of our local patch :)11:35
zioprotoarnewiebalck: yes, did you already push your local patch to gerrit ?11:36
arnewiebalckzioproto: No.11:36
zioprotoarnewiebalck: that is the first step forward !11:36
arnewiebalckzioproto: I ddon’t think there is much point if the bp is not accepted.11:37
* arnewiebalck checking if this was discussed during last Horizon meeting11:37
zioprotowell, if you push the patch, at least other people can carry it locally11:38
zioprotoand if there is a critical mass around the patch11:38
arnewiebalckzioproto: true :)11:38
zioprotoat one point it will get accepted11:38
arnewiebalckzioproto: let me talk to Mateusz as the author of our local patch what the status is11:38
*** bauruine has quit IRC11:39
*** bauruine has joined #openstack-operators11:39
*** markvoelker has joined #openstack-operators11:43
zioprotook !11:48
CrazyFishZioproto - We're running Mitaka on Ubuntu 16.04.2 (with systemd). We have dedicated compute nodes, but neutron, glance, heat and nova-scheduler are all on the failing node.11:58
makowalszioproto arnewiebalck: I have just pinged horizon channel about the blueprint, there was quite a lot of things yesterday during the meeting12:01
makowalsRegarding pushing the patch, I'd prefer to get some feedback about the bp itself before. If anyone wants to carry a local patch, our commit is public anyway so this is not an issue12:02
arnewiebalckmakowals: thx!12:08
zioprotocan we have the link to the commit to cherry pick ?12:12
*** markvoelker has quit IRC12:16
makowalshttps://gitlab.cern.ch/cloud-infrastructure/horizon/commit/37a38b65530c388bc43755a7cf8d28e5b302fbd1 and https://gitlab.cern.ch/cloud-infrastructure/horizon/commit/340082c5ccbb1161641a7a349a2209c02092c6a412:16
makowalsI'm doing now "clean" upstream commit so you should have it by the end of the day12:16
*** liverpooler has joined #openstack-operators12:19
zioprotothanks12:20
*** dminer has joined #openstack-operators12:25
*** markvoelker has joined #openstack-operators12:33
*** CrazyFish has quit IRC12:38
*** gkadam_ has quit IRC12:46
*** mriedem has joined #openstack-operators12:47
*** VW has joined #openstack-operators12:55
makowalszioproto: Feel free to check https://review.openstack.org/#/c/511472/, it should work without any problems on top of master12:58
*** CrazyFish has joined #openstack-operators13:00
*** catintheroof has joined #openstack-operators13:06
*** catintheroof has quit IRC13:06
*** catintheroof has joined #openstack-operators13:06
*** catintheroof has quit IRC13:13
zioprotomakowals: thanks. Do you think I can cherry pick it in Newton ?13:14
*** catintheroof has joined #openstack-operators13:14
makowalszioproto: Hmm, hard to say as I cherry-picked it from our local Ocata and modified a bit. For working stuff for newton we have this set of 3 patches, combined they were working for quite some time13:16
makowals1) https://gitlab.cern.ch/cloud-infrastructure/horizon/commit/8e12fa1fe02f0334312ac0b0c6b07cd17b25dca313:16
makowals2) https://gitlab.cern.ch/cloud-infrastructure/horizon/commit/883c8345e2be915eb1b452017b84c8a72a65fe4b13:16
makowals3) https://gitlab.cern.ch/cloud-infrastructure/horizon/commit/8e6af4cc7ec20811aea792187f175144ed0d58c413:16
*** CrazyFish has quit IRC13:16
makowalsFeel free to ping me on PM if you have some detailed questions about it13:16
makowalsSo we don't pollute the channel with technical details13:17
*** Caterpillar2 has joined #openstack-operators13:19
zioprotothanks ! I have all the pointers to do some work now :)13:21
*** Caterpillar has quit IRC13:23
*** chlong has joined #openstack-operators13:25
*** dbecker has quit IRC13:27
*** dbecker has joined #openstack-operators13:27
*** CrazyFish has joined #openstack-operators13:41
CrazyFishQuick update - I think we found a solution. In /etc/systemd/system/openvswitch-nonetwork.service.d/override.conf, we added a LimitNOFILE=xxxx, and that appears to have gotten our services back on track13:45
CrazyFish:)13:45
*** CrazyFish has quit IRC13:45
*** VW has quit IRC13:48
*** rmart04 has quit IRC13:50
*** nikhil_ has joined #openstack-operators13:52
*** nikhil_ is now known as Guest4851613:52
*** Guest48516 is now known as nikhil_k13:53
*** shintaro has joined #openstack-operators14:00
*** VW has joined #openstack-operators14:05
*** chlong has quit IRC14:13
*** ChiTo has joined #openstack-operators14:15
*** chlong has joined #openstack-operators14:19
*** rcernin has quit IRC14:55
*** Oku_OS is now known as Oku_OS-away15:01
*** chlong has quit IRC15:02
*** shintaro has quit IRC15:05
*** AlexeyAbashkin has quit IRC15:07
*** chlong has joined #openstack-operators15:16
*** electrofelix has quit IRC15:30
*** yprokule has quit IRC15:41
*** simon-AS559 has quit IRC15:49
*** penick has joined #openstack-operators15:55
*** chlong has quit IRC16:03
*** masber has quit IRC16:09
*** fragatina has quit IRC16:28
*** penick_ has joined #openstack-operators16:34
*** Apoorva has joined #openstack-operators16:35
*** penick has quit IRC16:35
*** chyka has joined #openstack-operators16:39
*** gyee has joined #openstack-operators16:41
*** gyee has quit IRC16:44
*** derekh has quit IRC16:58
*** jbadiapa has quit IRC17:07
*** penick_ has quit IRC17:09
*** tesseract has quit IRC17:11
*** simon-AS559 has joined #openstack-operators17:16
*** simon-AS5591 has joined #openstack-operators17:17
*** simon-AS559 has quit IRC17:20
*** slaweq_ has joined #openstack-operators17:23
*** slaweq_ has quit IRC17:24
*** simon-AS5591 has quit IRC17:28
*** slaweq_ has joined #openstack-operators17:55
*** fandrieu has quit IRC17:55
*** B_Smith has quit IRC18:02
*** penick has joined #openstack-operators18:07
*** penick_ has joined #openstack-operators18:16
*** penick has quit IRC18:18
*** B_Smith has joined #openstack-operators18:29
*** ChiTo has quit IRC18:48
mriedemmgagne: thanks for commenting in https://review.openstack.org/#/c/501017/ - i've replied18:48
mriedemmgagne: at this point you probably have to talk to sdague about the way forward18:49
sdaguemgagne: how long are your flavor id fields?18:49
*** slaweq_ has quit IRC18:59
mgagnesdague: the length has no importance at this point19:02
mgagnesdague: because someone else could have longer ids than me19:02
*** harlowja has quit IRC19:03
*** slaweq_ has joined #openstack-operators19:03
sdaguemgagne: sure that's fine, which is why we were trying to curb expectations by just changing the write path validation19:03
sdagueotherwise it's just 3 random 255 character fields that you can update one of19:03
mgagneI see: flavorid => unique identifier, name => label you could show in a dropdown, description => (hopefully) longer text you can show in the UI once selected19:03
mgagnesdague: did you know id and name are unique?19:04
sdagueI didn't realize name was also uniq19:05
mgagneit is19:05
sdagueyeh, I see that now19:05
mgagneand mentioned it in the review, that's why description would serve a completely different purpose to me19:05
*** masber has joined #openstack-operators19:05
sdagueok19:06
*** slaweq_ has quit IRC19:06
*** slaweq_ has joined #openstack-operators19:06
mgagneI just don't feel the need to artificially restrict existing fields and possibly break existing usage patterns in the wild for the sake of it19:07
mriedemi didn't think about flavorid and name being unique19:07
mriedemok i'll go back to what this was before19:08
sdaguemriedem: yeh, the uniqueness constraint on name does change that. I didn't realize that part was also unique19:08
mriedemmgagne: thanks again for the feedback - really good to get this *before* we make these changes19:08
sdaguemgagne: ++ thanks19:09
mgagnemriedem, sdague: thanks, I do appreciate the feedback loop19:09
* mgagne is going back to upgrades19:09
*** aojea has joined #openstack-operators19:10
sdaguemriedem: honestly, is there any reason to not make description just a Text() field. If people really want to have more details annotated in there, it's not going to be a thing we index or search on, and then we don't ever have to be concerned about people wanting to increase it later.19:10
*** masber has quit IRC19:10
mriedemsdague: i was following what server description was19:10
sdaguesure, but server description is "note to self"19:11
mriedemtrue19:11
sdagueflavor description is info to users19:11
mriedemsomeone at some point had mentioned just making this a text field19:11
mgagnemriedem: any impact on general performance? I decide to put 4MB of "description" in there19:12
mriedemlooks like a TEXT column in mysql is 65536 bytes19:13
mriedemMEDIUMTEXT is 16MiB19:14
mgagneok, much more reasonable, but still potentially 64k more of data per flavor19:14
mriedemwhich is what we allow for several other things, including user_data19:14
mgagneright, just looking into potential performance impacts if any since flavor is a lot more "global" than one specific instance user_data19:16
sdaguemgagne: right, though it only impacts things if you use it all19:16
mriedemwell, we store a serialized version of the flavor in the instance_extras table,19:16
mriedemwe'd have to be smart about *not* including the flavor description in the embedded instance.flavor19:17
mgagneyea, that's more or less what I was thinking about19:17
sdaguemriedem: you said we were never going to display it for the server19:17
sdaguewhich means we shouldn't store it19:17
mriedemyes, i hadn't thought about not including it in the embedded one that we store with the instance19:17
mriedemwell, or just figuring it wasn't going to be big enough to care19:18
mriedemso are we still thinking TEXT at 65536 bytes is ok?19:22
sdagueI am19:26
sdagueas long as it's not stored in the instance19:26
mriedemyeah, turns out the embedded serialized flavor column in instance_extras is already a TEXT column,19:26
mriedemso we'd blow it up if the flavor description was in there too19:26
mgagneis 1024 enough then?19:27
mriedemwell, if we're going to make flavors.description a TEXT column, we could do 64k in the API if we wanted, we'd just have to not include the flavor description in the embedded instance.flavor19:29
*** dminer has quit IRC19:29
mgagneinteresting, there is no way to list instance types through EC2 api, you need to go on their website19:31
sdaguemgagne: iirc once you go past 255, the handling effectively doesn't hurt just having it go to the next perf boundary19:33
mriedemok so i'm hearing, flavors.description is a TEXT column, we allow up to 65535 in the API, and we don't ever store the flavor.description with the embedded instance.flavor19:37
mriedemand everyone can shove their raid config in the flavor description and be happy campers19:37
mgagnemy bad faith detector just exploded :P ;)19:38
mriedemif that is an ok plan then i'll update the spec19:38
sdaguemriedem: ++19:38
mgagneworks for me19:38
mriedemthe raid config was semi serious,19:38
mriedempart of why huawei wants this is to put detailed information in baremetal flavors19:39
mriedemlike hardware specs19:39
mgagneyea, we would have a similar usage case but so far, there is no intention to serialize something in there, only better description19:39
*** harlowja has joined #openstack-operators19:57
*** fandrieu has joined #openstack-operators19:58
*** pcaruana has quit IRC19:58
*** AlexeyAbashkin has joined #openstack-operators20:05
*** liverpooler has quit IRC20:07
*** dmibrid_ has joined #openstack-operators20:09
*** AlexeyAbashkin has quit IRC20:09
*** liverpooler has joined #openstack-operators20:10
*** liverpooler has quit IRC20:16
*** fandrieu_ has joined #openstack-operators20:19
*** fandrieu has quit IRC20:20
*** fandrieu_ is now known as fandrieu20:20
*** aojea has quit IRC20:29
*** aojea has joined #openstack-operators20:31
*** Apoorva_ has joined #openstack-operators20:41
*** Apoorva has quit IRC20:43
*** AlexeyAbashkin has joined #openstack-operators20:44
*** AlexeyAbashkin has quit IRC20:48
*** B_Smith has quit IRC20:57
*** B_Smith has joined #openstack-operators21:02
*** aojea has quit IRC21:32
*** AlexeyAbashkin has joined #openstack-operators21:43
*** bsilverman_ has left #openstack-operators21:46
*** AlexeyAbashkin has quit IRC21:47
*** slaweq_ has quit IRC21:50
*** catintheroof has quit IRC21:53
*** catintheroof has joined #openstack-operators21:53
*** threestrands has joined #openstack-operators21:54
*** catintheroof has quit IRC21:55
*** Apoorva_ has quit IRC21:55
*** penick_ has quit IRC22:11
*** penick has joined #openstack-operators22:11
*** mriedem1 has joined #openstack-operators22:14
*** mriedem has quit IRC22:15
*** threestrands has quit IRC22:28
*** threestrands has joined #openstack-operators22:37
*** mriedem1 has quit IRC22:43
*** AlexeyAbashkin has joined #openstack-operators22:43
*** AlexeyAbashkin has quit IRC22:47
*** Apoorva has joined #openstack-operators23:23
*** aojea has joined #openstack-operators23:39
*** AlexeyAbashkin has joined #openstack-operators23:42
*** aojea has quit IRC23:44
*** markvoelker has quit IRC23:46
*** AlexeyAbashkin has quit IRC23:46
*** masber has joined #openstack-operators23:47
« Wednesday, 2017-10-11 Index Friday, 2017-10-13 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!