Wednesday, 2016-04-13

« Tuesday, 2016-04-12 Index Thursday, 2016-04-14 »
*** ChrissKO has joined #openstack-chef00:12
*** ChrissKO has quit IRC00:16
*** zhiwei has joined #openstack-chef00:24
openstackgerritMerged openstack/cookbook-openstack-common: Bump RDO to Mitaka GA  https://review.openstack.org/30468201:11
*** ChrissKO has joined #openstack-chef02:00
*** ChrissKO has quit IRC02:04
*** ChrissKO has joined #openstack-chef03:48
*** ChrissKO has quit IRC03:52
*** ChrissKO has joined #openstack-chef05:36
*** ChrissKO has quit IRC05:40
*** infra-red has joined #openstack-chef05:44
*** nkrinner has joined #openstack-chef05:54
*** ChrissKO has joined #openstack-chef05:58
*** ChrissKO has quit IRC05:58
*** infra-re_ has joined #openstack-chef06:32
*** infra-red has quit IRC06:35
*** infra-re_ has quit IRC06:44
*** infra-red has joined #openstack-chef07:22
*** ChrissKO_ has joined #openstack-chef07:26
*** ChrissKO_ has quit IRC07:31
*** ChrissKO_ has joined #openstack-chef09:15
*** ChrissKO_ has quit IRC09:19
*** Ryjedo_ has joined #openstack-chef09:39
*** Ryjedo has quit IRC09:47
*** chlong has quit IRC09:47
*** jklare_ has quit IRC09:47
*** higgins has quit IRC09:47
*** mancdaz has quit IRC09:47
*** Ryjedo_ is now known as Ryjedo09:47
*** higgins has joined #openstack-chef09:48
*** jklare has joined #openstack-chef09:48
*** zhiwei has quit IRC09:54
*** chlong has joined #openstack-chef09:55
*** zhiwei has joined #openstack-chef10:12
*** zhiwei has quit IRC10:16
*** mancdaz has joined #openstack-chef10:16
*** zhiwei has joined #openstack-chef10:23
*** zhiwei has quit IRC10:27
*** zhiwei has joined #openstack-chef10:29
*** openstackstatus has quit IRC10:32
*** openstack has joined #openstack-chef11:32
*** ChanServ sets mode: +o openstack11:32
*** openstackgerrit has quit IRC11:35
*** zhiwei has joined #openstack-chef11:35
*** Daviey has quit IRC11:36
*** Daviey has joined #openstack-chef11:37
*** zhiwei has quit IRC11:39
*** openstackgerrit has joined #openstack-chef11:51
*** zhiwei has joined #openstack-chef11:57
*** zhiwei has quit IRC12:01
*** zhiwei has joined #openstack-chef12:14
*** zhiwei has quit IRC12:19
*** zhiwei has joined #openstack-chef12:31
*** zhiwei has quit IRC12:36
*** ChrissKO has joined #openstack-chef12:51
*** aspiers has quit IRC12:54
*** zhiwei has joined #openstack-chef12:55
*** ChrissKO has quit IRC12:56
openstackgerritJan Klare proposed openstack/cookbook-openstack-network: fixed plugin_template notifications for neutron-server service resource  https://review.openstack.org/30525812:57
*** zhiwei has quit IRC12:59
*** zhiwei has joined #openstack-chef13:17
*** infra-red has quit IRC13:19
*** zhiwei has quit IRC13:21
*** infra-red has joined #openstack-chef13:23
*** infra-re_ has joined #openstack-chef13:24
*** infra-red has quit IRC13:27
*** infra-re_ has quit IRC13:33
*** zhiwei has joined #openstack-chef13:33
*** boyvinal- has joined #openstack-chef13:35
*** bhunter_ has joined #openstack-chef13:35
*** infra-red has joined #openstack-chef13:36
*** zhiwei has quit IRC13:38
*** infra-red has quit IRC13:39
*** infra-red has joined #openstack-chef13:39
*** jklare_ has joined #openstack-chef13:43
*** jklare has quit IRC13:43
*** higgins has quit IRC13:43
*** [d__d] has quit IRC13:43
*** bhunter has quit IRC13:43
*** boyvinall has quit IRC13:43
*** bhunter_ is now known as bhunter13:43
*** higgins has joined #openstack-chef13:44
*** [d__d] has joined #openstack-chef13:45
*** zhiwei has joined #openstack-chef13:55
*** zhiwei has quit IRC14:00
*** markvan has quit IRC14:04
*** aspiers has joined #openstack-chef14:04
*** markvan has joined #openstack-chef14:08
*** zhiwei has joined #openstack-chef14:18
*** zhiwei has quit IRC14:22
*** laurent\ has quit IRC14:24
*** infra-re_ has joined #openstack-chef14:35
*** infra-red has quit IRC14:37
*** ChrissKO has joined #openstack-chef14:39
*** laurent\ has joined #openstack-chef14:39
*** zhiwei has joined #openstack-chef14:40
sc`looks like i can push the title myself, but was advised that the wordy, catchy title might attract conference attendees that don't know it's a design summit discussion14:40
sc`@core thoughts? ^^^14:41
os-chef-bot@j^2 @markvan @mattray @wenchma @jklare @cmluciano @zhiwei @sc` thoughts? ^^^14:41
*** nkrinner has quit IRC14:42
*** zhiwei has quit IRC14:44
*** ChrissKO has quit IRC14:44
markvansc`: I think it should still work well, it's pretty easy to tell between design and general sessions.14:47
sc`based on ttx's response, i'll drop off the second mentioning of the project since it's redundant in the schedule. such as: "Chef OpenStack: Living the dream: refactoring and still going to production"14:49
sc`the last bit mentioning openstack and chef looks redundant in that context14:50
j^2I'd do what feels right to you sc`14:54
sc`word. wanted to get something on the schedule other than 'tbd'14:55
*** zhiwei has joined #openstack-chef14:57
*** zhiwei has quit IRC15:01
sc`pushed a title and a short description - https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Chef%20OpenStack%3A15:03
*** ogny has quit IRC15:09
*** jmb_ has joined #openstack-chef15:09
*** zhiwei has joined #openstack-chef15:13
*** furlongm has quit IRC15:15
*** furlongm has joined #openstack-chef15:16
*** zhiwei has quit IRC15:18
*** aspiers has quit IRC15:37
*** infra-red has joined #openstack-chef15:37
*** aspiers has joined #openstack-chef15:37
*** openstack has joined #openstack-chef15:52
*** ChanServ sets mode: +o openstack15:52
*** openstack has joined #openstack-chef16:09
*** ChanServ sets mode: +o openstack16:09
*** aspiers_ has quit IRC16:15
*** ChrissKO_ has joined #openstack-chef16:17
*** zhiwei has joined #openstack-chef16:19
*** zhiwei has quit IRC16:24
*** zhiwei has joined #openstack-chef16:36
*** zhiwei has quit IRC16:41
*** jmb_ has quit IRC16:42
*** infra-red has joined #openstack-chef16:54
*** zhiwei has joined #openstack-chef16:58
*** zhiwei has quit IRC17:03
*** infra-red has quit IRC17:04
*** aspiers_ has joined #openstack-chef17:05
*** aspiers_ has quit IRC17:14
*** zhiwei has joined #openstack-chef17:15
openstackgerritMark Vanderwiel proposed openstack/cookbook-openstack-image: Cleanup options deprecated for mitaka  https://review.openstack.org/30536817:17
*** zhiwei has quit IRC17:19
*** aspiers_ has joined #openstack-chef17:30
*** zhiwei has joined #openstack-chef17:32
*** zhiwei has quit IRC17:36
*** zhiwei has joined #openstack-chef17:54
*** zhiwei has quit IRC17:58
*** jmb_ has joined #openstack-chef18:07
*** zhiwei has joined #openstack-chef18:10
*** zhiwei has quit IRC18:14
*** ChrissKO has joined #openstack-chef18:27
*** ChrissKO_ has quit IRC18:28
*** zhiwei has joined #openstack-chef18:32
*** zhiwei has quit IRC18:36
*** jmickle has joined #openstack-chef18:37
*** hogepodge has quit IRC18:38
*** openstackgerrit has quit IRC18:39
*** cheeseplus has quit IRC18:41
*** technovangelist has quit IRC18:41
*** cheeseplus has joined #openstack-chef18:44
*** zhiwei has joined #openstack-chef18:48
*** zhiwei has quit IRC18:53
*** cbolt has joined #openstack-chef18:54
*** openstackgerrit has joined #openstack-chef18:55
*** technovangelist has joined #openstack-chef19:02
*** jmb_ has quit IRC19:03
*** jmickle has quit IRC19:05
*** zhiwei has joined #openstack-chef19:05
*** zhiwei has quit IRC19:10
openstackgerritMark Vanderwiel proposed openstack/cookbook-openstack-orchestration: Cleanup options deprecated for mitaka  https://review.openstack.org/30544919:15
*** ChrissKO has quit IRC19:22
*** ChrissKO has joined #openstack-chef19:23
openstackgerritMark Vanderwiel proposed openstack/cookbook-openstack-network: Cleanup options deprecated for mitaka  https://review.openstack.org/30545319:26
*** zhiwei has joined #openstack-chef19:27
*** zhiwei has quit IRC19:31
openstackgerritMark Vanderwiel proposed openstack/cookbook-openstack-block-storage: Cleanup options deprecated for mitaka  https://review.openstack.org/30545919:42
*** The_8th_Dwarf has joined #openstack-chef19:44
*** hogepodge has joined #openstack-chef19:46
*** os-chef-bot has quit IRC19:48
*** furlongm has quit IRC19:48
*** cbolt has quit IRC19:48
*** cheekyhalf has quit IRC19:48
*** Deltachaos has quit IRC19:48
*** jmb_ has joined #openstack-chef19:49
*** zhiwei has joined #openstack-chef19:49
*** furlongm_ has joined #openstack-chef19:52
*** zhiwei has quit IRC19:54
*** cbolt has joined #openstack-chef19:58
*** os-chef-bot has joined #openstack-chef19:58
*** furlongm has joined #openstack-chef19:58
*** cheekyhalf has joined #openstack-chef19:58
*** Deltachaos has joined #openstack-chef19:58
*** ChrissKO has quit IRC20:01
*** The_8th_Dwarf has quit IRC20:01
*** ChrissKO has joined #openstack-chef20:01
openstackgerritMark Vanderwiel proposed openstack/cookbook-openstack-block-storage: Cleanup options deprecated for mitaka  https://review.openstack.org/30545920:04
*** os-chef-bot has quit IRC20:04
*** furlongm has quit IRC20:04
*** cbolt has quit IRC20:04
*** cheekyhalf has quit IRC20:04
*** Deltachaos has quit IRC20:04
*** zhiwei has joined #openstack-chef20:07
*** zhiwei has quit IRC20:11
*** cbolt has joined #openstack-chef20:11
*** os-chef-bot has joined #openstack-chef20:11
*** cheekyhalf has joined #openstack-chef20:11
*** Deltachaos has joined #openstack-chef20:11
*** cheeseplus has left #openstack-chef20:12
openstackgerritMark Vanderwiel proposed openstack/openstack-chef-repo: Give cinder a chnace to create colume  https://review.openstack.org/30547220:14
*** os-chef-bot has quit IRC20:15
*** os-chef-bot has joined #openstack-chef20:15
*** cbolt has quit IRC20:16
*** cheekyhalf has quit IRC20:16
*** Deltachaos has quit IRC20:16
*** jmb_ has quit IRC20:20
*** cheekyhalf has joined #openstack-chef20:21
*** Deltachaos has joined #openstack-chef20:25
*** zhiwei has joined #openstack-chef20:29
*** zhiwei has quit IRC20:33
*** jmickle has joined #openstack-chef20:40
*** ChrissKO has quit IRC20:46
*** ChrissKO_ has joined #openstack-chef20:46
*** zhiwei has joined #openstack-chef21:02
*** zhiwei has quit IRC21:06
markvansc`: I pushed a few nit cleanup patches, but noticing that c7 mariadb is now failing to start, have you seen this?21:12
markvanand we don't seem to capture any logs in this case21:13
*** aspiers_ has quit IRC21:13
*** ChrissKO_ has quit IRC21:15
*** zhiwei has joined #openstack-chef21:19
*** jmickle has quit IRC21:20
markvan@core fyi this one is holding up other patch integration tests: https://review.openstack.org/#/c/305472/21:21
os-chef-bot@j^2 @markvan @mattray @wenchma @jklare @cmluciano @zhiwei @sc` fyi this one is holding up other patch integration tests: https://review.openstack.org/#/c/305472/21:21
*** zhiwei has quit IRC21:23
sc`markvan: haven't seen it before. i did note that we weren't collecting mariadb logs when i was working on integration, so it still needs to be done21:25
openstackgerritSamuel Cassiba proposed openstack/openstack-chef-repo: Capture mariadb logs for centos  https://review.openstack.org/30549421:32
markvansc`: humm, I do see the mysql logs on the c7 side for success cases, but nothing for this restart/startup failure21:33
sc`binlogs, not error logs21:33
markvanah21:33
sc`it captures /var/lib/mysql which is the binlog dir for mariadb/c721:34
sc`the meat of what we need lives in /var/log/mariadb21:34
sc`sorry, /var/log/mysql not /var/lib :)21:34
sc`it's confusing as hell when shifting context21:34
markvank21:35
*** zhiwei has joined #openstack-chef21:36
sc`i doubt it's worth doing a conditional re: /var/log/mysql vs. /var/log/mariadb, so we'll just have some small binlogs saved21:37
sc`i did see that the initial start of mariadb happens, and chef/openstack are happy thus far21:38
markvanyeah, seems to get going, but there's still a restart at the end.  The reason for extra restarts is hard to fun at times...and probably something that should be looked into for the major services21:40
*** zhiwei has quit IRC21:40
markvanfun/find21:40
sc`yeah. even in icehouse we've had problems with service restarts21:40
sc`(ewwww icehouse!)21:40
sc`c7 seems to be a bit more error prone than c6, though21:41
markvanyup, I really hate when the packagers mess with the startups that should be left to the deployers imo.21:41
sc`delayed mariadb restart just happened21:44
sc`still seems to be going thus far21:44
markvanyeah, I think (once again) something has changed in the infra image/server setup that throws timings off...which is probably a good thing so we can catch them.21:46
sc`possible. i know infra has made changes to the chef jobs and not said anything21:46
markvanouch21:46
sc`i only found out because it changed _while i was debugging a ci run_21:47
markvanwow, that's scary.21:47
sc`2016-04-13 21:48:10.436 | Chef Client finished, 428/576 resources updated in 12 minutes 42 seconds21:48
sc`so... transient?21:48
sc`or maybe a deprecated option is tickling something else?21:48
markvandang, would really like to see the logs...  so maybe we should get that repo log patch in, then we can respin the others and see what pops up21:49
sc`i'm going to let this run finish to see what it gathers21:49
markvanyeah, it also happened with the volume timing patch, no deprecation changes,  so don't think those are really involved21:50
markvansc`: don't like the look of this piece of code... https://github.com/sinfomicien/mariadb/blob/master/recipes/server.rb#L7821:52
sc`yup. i don't like it either21:53
sc`too hacky21:53
sc`too error prone21:53
markvanhardcoded 5 second timeout for connect?   while that seems long enough, nothing is that simple for the infra servers21:53
sc`no21:54
sc`i find that the problems we tend to encounter come from the greater chef community21:54
sc`if not packagers21:54
sc`especially weird stuff like this21:54
markvanI know we added a hardcoded sleep for Keystone a while back (not sure if it's still needed anymore),  but we could use that same approach to "help" this check success rate21:54
sc`yeah. that might make sense. problem is, nobody is really banging away with this in production except for maybe jklare's team. downstream operators that still use the pre-refactored cookbooks might get blindsided with some shit like this21:56
sc`if not already21:56
markvangood point.   hey speaking of "production"  do we still have those periodic jobs running?   and if yes, do they include the c7 side?21:57
* markvan yup lazy me just looked and they still appear to be there22:00
markvansc`: we should setup a bot to report the status of those jobs in this channel (if not already done an I'm just missing it...)22:01
sc`we should. i wonder if openstackgerrit can be extended to do so22:07
sc`https://review.openstack.org/#/c/305494/ is all green22:08
sc`which includes mariadb.log22:08
sc`not really seeing a way without hooking up another bot to the gerrit event stream22:17
sc`but periodics don't go through gerrit22:17
sc`markvan: looks like EmilienM started a thread on the ml about this very thing22:25
sc`i asked infra and they referred me there22:25
sc`tl;dr there's a dashboard and (yet another) ml for events22:26
sc`suboptimal to me, but that's what's there22:26
sc`markvan: http://status.openstack.org/openstack-health/#/g/build_queue/periodic?groupKey=build_queue&resolutionKey=hour&end=2016-04-13T22:31:19.256Z22:33
sc`doesn't look like the periodic jobs are listed in openstack-health22:35
sc`not ours anyway22:35
*** ChrissKO_ has joined #openstack-chef22:57
*** ChrissKO_ has quit IRC23:02
« Tuesday, 2016-04-12 Index Thursday, 2016-04-14 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!