*** dfisher has quit IRC | 00:20 | |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: Nothing to see here. https://review.openstack.org/398003 | 00:42 |
---|---|---|
mwhahaha | Hunner: fyi, https://tickets.puppetlabs.com/browse/MODULES-4114 | 00:52 |
*** morazi has quit IRC | 00:53 | |
*** jtriley has joined #puppet-openstack | 00:59 | |
*** jtriley has quit IRC | 01:06 | |
*** fultonj has quit IRC | 01:38 | |
*** xingchao has joined #puppet-openstack | 01:38 | |
openstackgerrit | Merged openstack/puppet-gnocchi: Fix common options position in params.pp https://review.openstack.org/394698 | 02:02 |
Hunner | mwhahaha: ah, we forgot a mergeback after release https://github.com/puppetlabs/puppetlabs-git/pull/77 | 02:19 |
mwhahaha | Cool thanks. I figured be it was something like that | 02:20 |
mwhahaha | s/be // | 02:20 |
*** jtriley has joined #puppet-openstack | 02:25 | |
openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Prepare 6.0.0 (ocata-1) https://review.openstack.org/398046 | 02:38 |
*** zhongshengping has quit IRC | 02:40 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/puppet-glance: Remove old authtoken options https://review.openstack.org/394749 | 02:40 |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/puppet-nova: Remove old authtoken options https://review.openstack.org/394753 | 02:45 |
*** ilbot3 has quit IRC | 02:46 | |
*** ilbot3 has joined #puppet-openstack | 02:48 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/puppet-heat: Remove old authtoken options https://review.openstack.org/394750 | 03:02 |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/puppet-cinder: Use new naming for authtoken variables https://review.openstack.org/398056 | 03:03 |
openstackgerrit | Merged openstack/puppet-magnum: Add magnum::clients class https://review.openstack.org/391807 | 03:18 |
*** xingchao has quit IRC | 03:45 | |
*** xingchao has joined #puppet-openstack | 03:45 | |
*** xingchao has quit IRC | 03:52 | |
*** xingchao has joined #puppet-openstack | 03:53 | |
*** xingchao_ has joined #puppet-openstack | 03:56 | |
*** xingchao has quit IRC | 03:57 | |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: Temporarily include newer PyMySQL https://review.openstack.org/398003 | 04:04 |
*** jtriley has quit IRC | 04:16 | |
*** jtriley has joined #puppet-openstack | 04:35 | |
*** jtriley has quit IRC | 04:40 | |
*** xingchao_ has quit IRC | 04:45 | |
*** xingchao has joined #puppet-openstack | 04:46 | |
*** xingchao has quit IRC | 05:01 | |
*** xingchao has joined #puppet-openstack | 05:45 | |
*** jaosorior has joined #puppet-openstack | 06:16 | |
*** xingchao has quit IRC | 06:20 | |
*** xingchao has joined #puppet-openstack | 06:23 | |
*** markvoelker has quit IRC | 06:28 | |
openstackgerrit | gyani pillala proposed openstack/puppet-cinder: VSA templates Cinder Backend https://review.openstack.org/398125 | 06:30 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Remove explicit hiera calls for heat in keystone profile https://review.openstack.org/398128 | 06:35 |
*** jaosorior has quit IRC | 06:41 | |
*** richm has quit IRC | 06:41 | |
*** jaosorior has joined #puppet-openstack | 06:42 | |
*** snarwade has quit IRC | 06:57 | |
*** xingchao has quit IRC | 06:59 | |
*** snarwade has joined #puppet-openstack | 07:00 | |
*** xingchao has joined #puppet-openstack | 07:06 | |
*** rasca has joined #puppet-openstack | 07:15 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Remove explicit hiera calls for heat in keystone profile https://review.openstack.org/398128 | 07:17 |
*** jaosorior has quit IRC | 07:34 | |
*** jaosorior has joined #puppet-openstack | 07:54 | |
*** shardy has joined #puppet-openstack | 07:56 | |
*** skramaja_ has joined #puppet-openstack | 08:05 | |
*** skramaja has quit IRC | 08:06 | |
*** karthiks has quit IRC | 08:07 | |
*** ccamacho has joined #puppet-openstack | 08:07 | |
*** amoralej|off is now known as amoralej | 08:20 | |
*** karthiks has joined #puppet-openstack | 08:22 | |
*** markvoelker has joined #puppet-openstack | 08:29 | |
*** jpich has joined #puppet-openstack | 08:31 | |
*** jpena|off is now known as jpena | 08:34 | |
*** markvoelker has quit IRC | 08:34 | |
*** saneax-_-|AFK is now known as saneax | 08:36 | |
*** hjensas has joined #puppet-openstack | 08:36 | |
*** saneax is now known as saneax-_-|AFK | 08:55 | |
*** saneax-_-|AFK is now known as saneax | 09:00 | |
openstackgerrit | gyani pillala proposed openstack/puppet-cinder: VSA templates Cinder Backend https://review.openstack.org/398125 | 09:00 |
*** dtantsur|afk is now known as dtantsur | 09:03 | |
*** skramaja has joined #puppet-openstack | 09:06 | |
*** skramaja_ has quit IRC | 09:06 | |
*** karthiks has quit IRC | 09:06 | |
*** saneax is now known as saneax-_-|AFK | 09:13 | |
*** karthiks has joined #puppet-openstack | 09:19 | |
*** dgurtner has joined #puppet-openstack | 09:21 | |
*** dgurtner has quit IRC | 09:21 | |
*** dgurtner has joined #puppet-openstack | 09:21 | |
*** lucas-afk is now known as lucasagomes | 09:32 | |
*** gfidente has joined #puppet-openstack | 09:32 | |
*** dtantsur is now known as dtantsur|afk | 09:33 | |
*** paramite has joined #puppet-openstack | 09:33 | |
*** dtantsur|afk is now known as dtantsur|bbl | 09:33 | |
*** saneax-_-|AFK is now known as saneax | 09:38 | |
*** _nick is now known as yankcrime | 09:38 | |
*** derekh has joined #puppet-openstack | 09:41 | |
*** bogdando has quit IRC | 09:50 | |
*** bogdando has joined #puppet-openstack | 09:52 | |
*** saneax is now known as saneax-_-|AFK | 09:59 | |
*** dgurtner has quit IRC | 10:19 | |
*** markvoelker has joined #puppet-openstack | 10:30 | |
*** markvoelker has quit IRC | 10:35 | |
openstackgerrit | gyani pillala proposed openstack/puppet-cinder: VSA templates Cinder Backend https://review.openstack.org/398125 | 10:42 |
*** xingchao has quit IRC | 11:05 | |
*** xingchao has joined #puppet-openstack | 11:06 | |
*** richm has joined #puppet-openstack | 11:12 | |
*** xingchao has quit IRC | 11:17 | |
*** jaosorior is now known as jaosorior_lunch | 11:25 | |
openstackgerrit | gyani pillala proposed openstack/puppet-cinder: HPLeftHandISCSIDriver support for Cinder Backend https://review.openstack.org/398125 | 11:33 |
*** dtantsur|bbl is now known as dtantsur | 11:44 | |
*** bogdando has quit IRC | 11:55 | |
*** bogdando has joined #puppet-openstack | 11:57 | |
*** dprince has joined #puppet-openstack | 12:01 | |
*** xingchao has joined #puppet-openstack | 12:02 | |
*** fultonj has joined #puppet-openstack | 12:04 | |
iurygregory | good morning | 12:07 |
*** dgurtner has joined #puppet-openstack | 12:08 | |
*** jaosorior_lunch is now known as jaosorior | 12:09 | |
*** lucasagomes is now known as lucas-hungry | 12:11 | |
*** shardy is now known as shardy_lunch | 12:11 | |
*** psanchez has quit IRC | 12:13 | |
*** dprince has quit IRC | 12:15 | |
*** rook-out has quit IRC | 12:15 | |
*** psanchez has joined #puppet-openstack | 12:15 | |
*** dprince has joined #puppet-openstack | 12:16 | |
*** rook has joined #puppet-openstack | 12:18 | |
*** rook is now known as Guest45665 | 12:18 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Sort parameters in keystone profile alphabetically https://review.openstack.org/398323 | 12:21 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Sort parameters in keystone profile alphabetically https://review.openstack.org/398323 | 12:24 |
*** ken_barber has quit IRC | 12:24 | |
*** fxpester has joined #puppet-openstack | 12:40 | |
*** shardy_lunch is now known as shardy | 12:45 | |
*** xingchao has quit IRC | 12:49 | |
*** pradk has joined #puppet-openstack | 12:55 | |
*** morazi has joined #puppet-openstack | 13:00 | |
*** johnmilton has joined #puppet-openstack | 13:01 | |
*** ccamacho is now known as ccamacho|lunch | 13:04 | |
*** morazi has quit IRC | 13:07 | |
*** morazi has joined #puppet-openstack | 13:08 | |
*** ducttape_ has joined #puppet-openstack | 13:08 | |
*** ducttape_ has quit IRC | 13:09 | |
*** ducttape_ has joined #puppet-openstack | 13:10 | |
*** trown|outtypewww is now known as trown | 13:11 | |
*** dgurtner has quit IRC | 13:18 | |
*** lucas-hungry is now known as lucasagomes | 13:20 | |
jaosorior | iurygregory: hey, is CI officially broken? Or should I still recheck? | 13:23 |
*** jayg|g0n3 is now known as jayg | 13:25 | |
*** ken_barber has joined #puppet-openstack | 13:29 | |
iurygregory | jaosorior, recheck =) | 13:31 |
openstackgerrit | gyani pillala proposed openstack/puppet-cinder: HPLeftHandISCSIDriver support for Cinder Backend https://review.openstack.org/398125 | 13:33 |
*** ducttape_ has quit IRC | 13:38 | |
*** ducttape_ has joined #puppet-openstack | 13:38 | |
*** ducttape_ has quit IRC | 13:43 | |
*** jpena is now known as jpena|lunch | 13:43 | |
*** markvoelker has joined #puppet-openstack | 13:44 | |
*** amoralej is now known as amoralej|lunch | 13:52 | |
jaosorior | iurygregory: do you happen to know, in puppet-heat, where would the "beaker tests" be? | 14:01 |
iurygregory | jaosorior, yep spec/acceptance | 14:01 |
iurygregory | https://github.com/openstack/puppet-heat/blob/master/spec/acceptance/basic_heat_spec.rb | 14:02 |
jaosorior | iurygregory: thanks | 14:02 |
iurygregory | jaosorior, np | 14:02 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-heat: Add manifests to deploy APIs over httpd https://review.openstack.org/394835 | 14:05 |
iurygregory | jaosorior, maybe you should create a new acceptance for the wsgi i think =) | 14:06 |
iurygregory | or the deploy in wsgi is mandatory? | 14:07 |
jaosorior | iurygregory: well, EmilienM -1ed the patch cause he wanted it tested there. | 14:07 |
jaosorior | either I could create a new acceptance test, or just run one or two API's over httpd | 14:07 |
iurygregory | yeah | 14:08 |
jaosorior | iurygregory: what would you recommend? | 14:08 |
iurygregory | jaosorior, let's first see the result in the CI | 14:09 |
iurygregory | and when EmilienM come back we can talk :D | 14:09 |
jaosorior | true | 14:10 |
jaosorior | and lets see if it works, I do need a fix I did for heat-templates to be available, and I'm not sure if it's there yet. | 14:10 |
iurygregory | ok =) | 14:11 |
*** Guest45665 is now known as rook | 14:13 | |
*** jperry has joined #puppet-openstack | 14:16 | |
*** ducttape_ has joined #puppet-openstack | 14:20 | |
*** ccamacho|lunch is now known as ccamacho | 14:28 | |
*** ducttape_ has quit IRC | 14:31 | |
*** jtriley has joined #puppet-openstack | 14:31 | |
*** zhongshengpin has joined #puppet-openstack | 14:31 | |
*** ducttape_ has joined #puppet-openstack | 14:32 | |
*** amoralej|lunch is now known as amoralej | 14:32 | |
*** ducttape_ has quit IRC | 14:36 | |
*** jaosorior has quit IRC | 14:41 | |
*** jaosorior has joined #puppet-openstack | 14:42 | |
openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Release 6.0.0 https://review.openstack.org/398387 | 14:43 |
*** jaosorior has quit IRC | 14:47 | |
*** morazi has quit IRC | 14:47 | |
*** agrebennikov has joined #puppet-openstack | 14:48 | |
*** morazi has joined #puppet-openstack | 14:52 | |
*** richm has quit IRC | 14:53 | |
*** ducttape_ has joined #puppet-openstack | 14:54 | |
*** dfisher has joined #puppet-openstack | 14:57 | |
*** rmart04 has joined #puppet-openstack | 14:58 | |
*** jpena|lunch is now known as jpena | 15:01 | |
*** richm has joined #puppet-openstack | 15:09 | |
mwhahaha | ugh looks like scenario004 on ubuntu is broken, http://logs.openstack.org/31/397431/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/749c65e/logs/puppet.txt.gz#_2016-11-15_22_39_40 | 15:19 |
mwhahaha | thanks, openstack. | 15:19 |
dfisher | lol | 15:19 |
mwhahaha | what the hell is URL has an invalid label | 15:20 |
iurygregory | damm it | 15:20 |
mwhahaha | so i guess that's a unicode error | 15:25 |
mwhahaha | the package list is the same from a working and a broken, so it's not an ubuntu change | 15:27 |
* mwhahaha cries | 15:28 | |
iurygregory | D: | 15:28 |
mwhahaha | gotta take the kids to school then i'll poke infra to see if there were any xenial changes overnight | 15:28 |
*** michapma_alt has quit IRC | 15:35 | |
*** ducttape_ has quit IRC | 15:35 | |
*** jtriley has quit IRC | 15:38 | |
openstackgerrit | zhongshengping proposed openstack/puppet-openstack_extras: Modify the annotation for the release parameter https://review.openstack.org/398420 | 15:42 |
*** morazi has quit IRC | 15:44 | |
*** jtriley has joined #puppet-openstack | 15:46 | |
*** fxpester has quit IRC | 15:53 | |
*** morazi has joined #puppet-openstack | 15:57 | |
*** ducttape_ has joined #puppet-openstack | 16:02 | |
dmsimard | mwhahaha: be careful who you blame for the issue | 16:03 |
dmsimard | it might be very well be some ruby gem crap | 16:04 |
mwhahaha | this is a python error, but it's a possible | 16:04 |
mwhahaha | it seems more environmental than code | 16:04 |
mwhahaha | just trying to rule out all the things | 16:04 |
mwhahaha | it's also only senario004 which is weird | 16:05 |
*** danpawlik has quit IRC | 16:06 | |
*** zhongshengpin has quit IRC | 16:06 | |
*** zhongshengpin has joined #puppet-openstack | 16:07 | |
openstackgerrit | Bradley Klein proposed openstack/puppet-monasca: Add support for monasca-notification hipchat and slack https://review.openstack.org/397989 | 16:14 |
*** zhongshengpin has quit IRC | 16:17 | |
*** ducttape_ has quit IRC | 16:22 | |
openstackgerrit | Merged openstack/puppet-tripleo: Remove explicit hiera calls for heat in keystone profile https://review.openstack.org/398128 | 16:22 |
*** ducttape_ has joined #puppet-openstack | 16:23 | |
openstackgerrit | Bradley Klein proposed openstack/puppet-monasca: Add support for monasca-notification hipchat and slack https://review.openstack.org/397989 | 16:26 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: [DONOTMERGE] puppet debug https://review.openstack.org/398465 | 16:29 |
*** hjensas has quit IRC | 16:37 | |
*** rmart04 has quit IRC | 16:39 | |
*** jistr is now known as jistr|biab | 16:48 | |
*** rasca has quit IRC | 16:49 | |
*** paramite has quit IRC | 16:51 | |
openstackgerrit | Merged openstack/puppet-tripleo: Increase haproxy timeouts https://review.openstack.org/397822 | 16:51 |
*** dtantsur is now known as dtantsur|afk | 16:58 | |
mwhahaha | ayoung: are you around? | 16:59 |
ayoung | mwhahaha, will be in a few minutes...finishin a meeting | 17:00 |
mwhahaha | the scenario004 thing seems to be some sort of keystone itneraction | 17:01 |
openstackgerrit | Bradley Klein proposed openstack/puppet-monasca: Add support for monasca-notification hipchat and slack https://review.openstack.org/397989 | 17:01 |
mwhahaha | ayoung: ok i've got a meeting now, had a question about a log entry | 17:01 |
mwhahaha | [Tue Nov 15 22:36:34.403664 2016] [wsgi:error] [pid 30091:tid 140370284582656] 2016-11-15 22:36:34.403 30091 DEBUG keystone.middleware.auth [req-e77274d6-276c-4181-80ee-08bb0bd5f671 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. fill_context | 17:01 |
mwhahaha | /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:210 | 17:01 |
mwhahaha | ayoung: https://www.diffchecker.com/NuNvKZXy from the admin error log | 17:16 |
ayoung | mwhahaha, we'll chat after your meeting | 17:18 |
mwhahaha | since i got stood up, i guess i'm free | 17:18 |
mwhahaha | ayoung: basically we're seeing failures in our scenario004 and it's currently failing when trying to prefetch the project list for the keystone tenant provider | 17:19 |
mwhahaha | ayoung: what i've spotted so far is that diff. it seems that when it was working it was using the admin token rather than a fernet token and it tries to post to tokens but gets a 401 back | 17:20 |
mwhahaha | ayoung: when it was working, it never posted to tokens and just was able to pull the list. so i'm wondering what would posibly generate that middleware error since that seems to be the only error to go on at the moment | 17:21 |
*** rmart04 has joined #puppet-openstack | 17:24 | |
ayoung | mwhahaha, is scenario004 tempest? Link? | 17:27 |
mwhahaha | it's our puppet scenario004 | 17:27 |
mwhahaha | https://github.com/openstack/puppet-openstack-integration#description | 17:28 |
ayoung | mwhahaha, so, if you were using ADMIN_TOKEN before you were in a state of Sin. You know that now, right? | 17:28 |
mwhahaha | so we only use it temporarily i think. but at the moment i'm just trying to figure out wtf is happening | 17:29 |
mwhahaha | worked: http://logs.openstack.org/91/395791/5/gate/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/7f874a0/logs/puppet-20161115_154439.txt.gz#_2016-11-15_15_40_36 | 17:29 |
mwhahaha | failed: http://logs.openstack.org/91/395791/5/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ad7e956/logs/puppet.txt.gz#_2016-11-15_22_36_32 | 17:30 |
mwhahaha | so as far as I can dig in to the apache logs, we don't request a token for that first interaction | 17:30 |
mwhahaha | since the packaging hasn't changed (and this is ubuntu not centos) i can't for the life of me figure out why this only happening here with only that error i pasted previously | 17:31 |
mwhahaha | so what could possibly generate that error | 17:31 |
ayoung | mwhahaha, you do realize I have absolutely no idea what you are trying to actually run here. I could figure it out from the logs, but it would be much nicer to me if you told me, or pointed me at a link to the test code, or something | 17:32 |
mwhahaha | hi, welcome to my world | 17:32 |
ayoung | "This I know from Nothing!" | 17:33 |
mwhahaha | basically scenaio004 is just a way of us deploying stuff, uses the same ways as everything else | 17:33 |
mwhahaha | my question to you is what could generate that error i pasted previously | 17:33 |
mwhahaha | 2016-11-15 22:36:34.403 30091 DEBUG keystone.middleware.auth [req-e77274d6-276c-4181-80ee-08bb0bd5f671 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. fill_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:210 | 17:34 |
ayoung | that is not an error | 17:35 |
*** jistr|biab is now known as jistr | 17:36 | |
mwhahaha | it's a wsgi:error | 17:36 |
ayoung | we just spam it all over the place | 17:36 |
ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/middleware/auth.py#n188 | 17:37 |
ayoung | but since that code gets execute on token request, it trips it up | 17:37 |
ayoung | mwhahaha, don't get me started on the code base...token issue really shoukld be a separate microservice from everything else | 17:37 |
mwhahaha | so somehow we're managing to fail all the previous context seting conditions, http://git.openstack.org/cgit/openstack/keystone/tree/keystone/middleware/auth.py#n174 | 17:39 |
ayoung | mwhahaha, you need to get a token before performing an operation now | 17:40 |
mwhahaha | in ocata sure | 17:40 |
mwhahaha | this is ubuntu newton | 17:40 |
ayoung | and you probably are not setting the appropriate variables for the keystoneauth plugins to do that | 17:40 |
mwhahaha | and nothing has changed between the two errors | 17:40 |
mwhahaha | because ci has been blocked for days now because of cinder | 17:40 |
mwhahaha | so some how it's magically screwing up :/ | 17:40 |
ayoung | not magic, just sufficiently advanced technology, but it is indistinguishabl | 17:41 |
ayoung | e | 17:41 |
mwhahaha | so i'm looking for ideas of what configuration items might cause those conditions to fail now (today) when they were working (yesterday) | 17:41 |
mwhahaha | it might just be a race condition of some sor tbut i'm not sure what to look for | 17:42 |
mwhahaha | this is happening early in the deployemnt | 17:42 |
ayoung | did you switch over to using bootstrap? | 17:44 |
mwhahaha | it's been using bootstrap for a while now | 17:45 |
mwhahaha | if you looked at the puppet log that ran prior to the apache restart | 17:45 |
ayoung | mwhahaha, so my guess ios you were using ADMIN_TOKEN for operations. And now you are not enabling it | 17:45 |
ayoung | do you have a full error log etc | 17:45 |
mwhahaha | full error log of what | 17:46 |
mwhahaha | fails, http://logs.openstack.org/65/398465/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/d3c7f54/logs/apache/keystone_wsgi_admin_error.txt.gz | 17:46 |
mwhahaha | worked, http://logs.openstack.org/91/395791/5/gate/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/7f874a0/logs/apache/keystone_wsgi_admin_error.txt.gz | 17:46 |
ayoung | mwhahaha, yeah, but that does not have all the files | 17:46 |
mwhahaha | error log of what | 17:46 |
*** lucasagomes is now known as lucas-afk | 17:47 | |
*** derekh has quit IRC | 17:47 | |
ayoung | mwhahaha, where is the gerrit review that triggered that? | 17:47 |
mwhahaha | https://review.openstack.org/#/c/395791/ | 17:47 |
ayoung | https://review.openstack.org/#/c/395791 | 17:47 |
ayoung | got it | 17:47 |
ayoung | Heh scenario 4 seems to be the only one passing there | 17:47 |
ayoung | on centos | 17:48 |
mwhahaha | don't get me started with centos | 17:48 |
ayoung | the opposite on xenial | 17:48 |
ayoung | admin_token = a_big_secret | 17:50 |
ayoung | http://logs.openstack.org/91/395791/5/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ad7e956/logs/etc/keystone/keystone.conf.txt.gz | 17:50 |
ayoung | so did we do something else to wipe that out... | 17:50 |
ayoung | http://logs.openstack.org/91/395791/5/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ad7e956/logs/etc/keystone/keystone-paste.ini.txt.gz | 17:50 |
ayoung | still there: | 17:50 |
ayoung | pipeline = cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id admin_token_auth build_auth_context token_auth json_body ec2_extension public_service | 17:50 |
ayoung | admin_token_auth | 17:50 |
mwhahaha | right we remove it later | 17:51 |
mwhahaha | as part of the deployment process | 17:51 |
*** trown is now known as trown|lunch | 17:52 | |
mwhahaha | you know i bet it's something silly like apache restarting slower or faster | 17:55 |
dmsimard | did we merge that apache warmup thing ?& | 17:56 |
mwhahaha | no it broke a bunch of stuff | 17:56 |
ayoung | mwhahaha, sorry RWI | 17:57 |
ayoung | lets see... | 17:58 |
mwhahaha | i'm so irritated at CI this week between the cinder thing and now this | 17:59 |
ayoung | mwhahaha, is that the cause of this failure: https://review.openstack.org/#/c/397381/ ? gate-tripleo-ci-centos-7-ovb-ha | 17:59 |
ayoung | or is this later than the cinder thing? | 18:00 |
*** cdelatte has joined #puppet-openstack | 18:00 | |
ayoung | I think we have all the puppet code for Fernet merged... | 18:00 |
* ayoung goes to check | 18:00 | |
mwhahaha | ayoung: no, your issue was httpd hanging, http://logs.openstack.org/81/397381/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/db9630e/logs/postci.txt.gz#_2016-11-14_23_58_19_000 | 18:01 |
mwhahaha | we have not merged the fernet defaults in openstack puppet, it was previously reverted | 18:01 |
mwhahaha | that being said this scenario that is failing I think uses fernet tokens | 18:01 |
ayoung | mwhahaha, not the defaults, just the mgmgt piece | 18:01 |
openstackgerrit | gyani pillala proposed openstack/puppet-cinder: HPLeftHandISCSIDriver support for Cinder Backend https://review.openstack.org/398125 | 18:02 |
mwhahaha | ayoung: you meanhttps://review.openstack.org/#/c/397391/ | 18:02 |
mwhahaha | sorry https://review.openstack.org/#/c/397391/ | 18:02 |
ayoung | https://review.openstack.org/#/c/334678/ | 18:03 |
mwhahaha | whats super annoying about this failure is that we really need logs from openstack client | 18:03 |
mwhahaha | yea we merged that 4 months ago, like i said nothing really has changed in the last 24 hours in terms of puppet or packaging for xenial that could easily be pointed at for this failure | 18:04 |
mwhahaha | so it seems to totally be a race condition or some sort of odd state that we're now hitting and only hitting on that one scenario | 18:04 |
mwhahaha | but as far as I can tell the configs are the same at the same point | 18:05 |
mwhahaha | something's happening when it tries to fill that context and i don't know enough about it to say what | 18:05 |
mwhahaha | https://www.diffchecker.com/hHqAX4bb | 18:05 |
mwhahaha | it looks like previously we were successfully entering this block of code, http://git.openstack.org/cgit/openstack/keystone/tree/keystone/middleware/auth.py#n174 | 18:07 |
mwhahaha | but we're not longer getting in | 18:07 |
ayoung | Nah...I think it is later | 18:08 |
ayoung | let me see... | 18:08 |
mwhahaha | the reason why i say that block is because of http://logs.openstack.org/91/395791/5/gate/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/7f874a0/logs/keystone/keystone-admin.txt.gz#_2016-11-15_15_40_44_488 | 18:08 |
mwhahaha | where auth_context is {} | 18:08 |
ayoung | Um./..waht about the SSL error at the bottom of the file>? | 18:10 |
*** jperry has quit IRC | 18:10 | |
mwhahaha | that's later in the deployment | 18:10 |
ayoung | revocation_list | 18:10 |
ayoung | OK, that needs to go away | 18:10 |
mwhahaha | yea i spotted that today along with a bunch of other errors while looking at this thing | 18:11 |
mwhahaha | oh we should also be looking at the newton version of this file | 18:11 |
mwhahaha | not the master | 18:11 |
mwhahaha | xenial is newton | 18:11 |
ayoung | I love that we are doing IPv6 | 18:12 |
ayoung | point me at the right place. BUt the lack or return codes/error codes make it hard to debug | 18:13 |
mwhahaha | https://github.com/openstack/keystone/blob/stable/newton/keystone/middleware/auth.py#L182-L213 | 18:13 |
mwhahaha | github is easier to pass links arounds with | 18:13 |
mwhahaha | so that's the conditional block it would be running through | 18:13 |
mwhahaha | so it would either be the first if or the first elif | 18:13 |
mwhahaha | but it's not the first elif because we don't get that deprecation notice | 18:14 |
mwhahaha | so my assumption is that request.env.get(core.CONTEXT_ENV, {}).get('is_admin', False) is no longer evaulating to true | 18:14 |
mwhahaha | but i have no idea wtf that line is | 18:14 |
mwhahaha | where does that stuff come from? apache? | 18:14 |
mwhahaha | mod_wsgi? | 18:14 |
mwhahaha | i guess i could just try disabling the admin_token all together to see if that addresses it | 18:16 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: [DONOTMERGE] puppet debug https://review.openstack.org/398465 | 18:17 |
mwhahaha | we shouldn't be using admin_token in our testing anyway because we support admin_password now instead | 18:17 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: [DONOTMERGE] puppet debug https://review.openstack.org/398465 | 18:19 |
*** jperry has joined #puppet-openstack | 18:24 | |
mwhahaha | so we appear to still be using the auth token for part of the intial setup | 18:29 |
mwhahaha | oh probably cause we don't have an openrc file yet at that point of the deploy | 18:30 |
openstackgerrit | Merged openstack/puppet-tripleo: Prepare 6.0.0 (ocata-1) https://review.openstack.org/398046 | 18:33 |
openstackgerrit | Alex Schultz proposed openstack/puppet-keystone: nothing to see here https://review.openstack.org/398529 | 18:36 |
*** jpich has quit IRC | 18:36 | |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: [DONOTMERGE] puppet debug https://review.openstack.org/398465 | 18:36 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: [DONOTMERGE] puppet debug https://review.openstack.org/398465 | 18:37 |
*** jpena is now known as jpena|off | 18:42 | |
*** jtriley has quit IRC | 18:46 | |
*** rmart04_ has joined #puppet-openstack | 18:48 | |
*** rmart04 has quit IRC | 18:49 | |
*** rmart04_ is now known as rmart04 | 18:49 | |
*** shardy is now known as shardy_afk | 19:01 | |
*** trown|lunch is now known as trown | 19:05 | |
openstackgerrit | Ade Lee proposed openstack/puppet-nova: Added ipaclient class https://review.openstack.org/397836 | 19:05 |
*** amoralej is now known as amoralej|off | 19:12 | |
*** shardy_afk is now known as shardy | 19:16 | |
*** shardy is now known as shardy_afk | 19:17 | |
*** jtriley has joined #puppet-openstack | 19:27 | |
mwhahaha | it's ipv6 | 19:34 |
* mwhahaha throws tables | 19:35 | |
dfisher | haha. | 19:35 |
iurygregory | damm it | 19:35 |
dfisher | ^ the proper response | 19:35 |
iurygregory | =( | 19:35 |
mwhahaha | https://github.com/kennethreitz/requests/blob/master/requests/models.py#L372-L375 | 19:35 |
mwhahaha | the host is '[::1]' | 19:35 |
mwhahaha | which causes the encode to puke | 19:35 |
mwhahaha | https://github.com/kennethreitz/requests/blob/master/requests/models.py#L372-L375 | 19:35 |
mwhahaha | er | 19:36 |
mwhahaha | idna.core.InvalidCodepoint: Codepoint U+005B not allowed at position 1 in u'[::1]' | 19:36 |
mwhahaha | http://paste.openstack.org/show/589502/ | 19:36 |
mwhahaha | i bet we somehow switched to ipv6 which i think only works on centos | 19:37 |
mwhahaha | how the hell did this ever work | 19:38 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: Align stars to fix CI https://review.openstack.org/398003 | 19:42 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: Align stars to fix CI https://review.openstack.org/398003 | 19:44 |
*** johnmilton has quit IRC | 19:56 | |
*** rmart04 has left #puppet-openstack | 20:02 | |
mfisch | mwhahaha: you around? | 20:23 |
mwhahaha | if i must | 20:23 |
mwhahaha | what's up? | 20:23 |
mfisch | lol | 20:23 |
mfisch | wondering should we just remove all eventlet support instead of my change to move stuff around? | 20:24 |
mwhahaha | did we deprecate it? | 20:24 |
mwhahaha | :D | 20:24 |
mfisch | yeah I think in L | 20:24 |
mwhahaha | when did keystone drop the eventlet support, was that part of newton? | 20:24 |
mfisch | I will look into it | 20:24 |
mwhahaha | like it's just not a thing anymore | 20:24 |
mwhahaha | right? | 20:25 |
mfisch | [blueprint removed-as-of-newton] Removed [eventlet_server] and [eventlet_server_ssl] sections from the keystone.conf. | 20:26 |
mfisch | removed in N | 20:26 |
mfisch | [blueprint removed-as-of-newton] Removed support for running keystone under eventlet. It is recommended to run keystone in an HTTP server. | 20:26 |
mwhahaha | yea if it's just not there anymore, let's pull it | 20:26 |
mfisch | ok | 20:26 |
mfisch | I will update | 20:26 |
mfisch | hopefully I can get the unit tests functional again on my mac | 20:27 |
mfisch | something broke them | 20:27 |
mfisch | mwhahaha: have you seen this before? | 20:46 |
mfisch | /Users/mfischer/code/puppet/puppet-keystone/spec/spec_helper.rb:6:in `require': cannot load such file -- puppet-openstack_spec_helper/facts (LoadError) | 20:46 |
mfisch | when running rake spec | 20:46 |
mwhahaha | yea | 20:47 |
mwhahaha | nuke your Gemfile.lock and try rerunning bundle install | 20:47 |
*** rcernin has joined #puppet-openstack | 20:58 | |
mwhahaha | this whole ci thing is so frustrating, i managed to fix scenario004 and get pymysql for centos only to get timed out | 21:08 |
iurygregory | =( | 21:09 |
*** dprince has quit IRC | 21:13 | |
*** morazi has quit IRC | 21:14 | |
openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Remove Combination alarms support https://review.openstack.org/398579 | 21:15 |
openstackgerrit | Merged openstack/puppet-heat: Don't default metadata server URLs to localhost https://review.openstack.org/397644 | 21:19 |
openstackgerrit | Ade Lee proposed openstack/puppet-nova: Added ipaclient class https://review.openstack.org/397836 | 21:20 |
*** jayg is now known as jayg|g0n3 | 21:25 | |
*** jtriley_ has joined #puppet-openstack | 21:30 | |
*** gfidente has quit IRC | 21:31 | |
openstackgerrit | Matt Fischer proposed openstack/puppet-keystone: Drop all eventlet support https://review.openstack.org/396475 | 21:32 |
*** jtriley has quit IRC | 21:32 | |
*** fultonj_ has joined #puppet-openstack | 21:33 | |
*** fultonj_ has quit IRC | 21:33 | |
mfisch | mwhahaha: thats a Work in progress ^ | 21:33 |
mfisch | I'll ping you when ready | 21:33 |
*** cdelatte has quit IRC | 21:34 | |
*** jtriley_ has quit IRC | 21:34 | |
mwhahaha | ok | 21:35 |
openstackgerrit | Matt Fischer proposed openstack/puppet-keystone: Drop all eventlet support https://review.openstack.org/396475 | 21:50 |
*** trown is now known as trown|outtypewww | 21:52 | |
*** ayoung has quit IRC | 21:55 | |
mwhahaha | dmsimard: https://review.openstack.org/#/c/398003/ plz4me | 21:58 |
dmsimard | I saw, getting to it soon | 22:00 |
dmsimard | I was going to nit about setting ipv6 to false in both if and else | 22:00 |
dmsimard | but ... yeah | 22:00 |
dmsimard | :P | 22:00 |
mwhahaha | ugh | 22:00 |
mwhahaha | at this point i just want to merge all my 10.0.0 patches so i can get the stupid release out the door | 22:01 |
openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration: Fix scenario004 ipv6 https://review.openstack.org/398603 | 22:02 |
dmsimard | mwhahaha: yeah I was mostly kidding | 22:02 |
mwhahaha | there ya go as a followon since we won't be able to cleanly revert that patch anyway | 22:02 |
dmsimard | I'll point it out anyway cause I'm like that | 22:03 |
iurygregory | mwhahaha, you already have the release patches topic? | 22:17 |
mwhahaha | https://review.openstack.org/#/q/topic:ocata-1 | 22:17 |
iurygregory | tks | 22:17 |
openstackgerrit | Ade Lee proposed openstack/puppet-nova: Added ipaclient class https://review.openstack.org/397836 | 22:25 |
openstackgerrit | Alex Schultz proposed openstack/puppet-ec2api: Prepare 10.0.0 release https://review.openstack.org/398627 | 22:27 |
openstackgerrit | Alex Schultz proposed openstack/puppet-octavia: Prepare 10.0.0 release https://review.openstack.org/398628 | 22:27 |
*** fultonj has quit IRC | 22:27 | |
openstackgerrit | Alex Schultz proposed openstack/puppet-vswitch: Prepare 6.0.0 release https://review.openstack.org/397437 | 22:36 |
*** jperry has quit IRC | 22:42 | |
*** ayoung has joined #puppet-openstack | 22:44 | |
*** pradk has quit IRC | 22:48 | |
*** ducttape_ has quit IRC | 23:16 | |
patrickeast | hello, this might be a dumb question, but is there a "right" way to pin my puppet manifests to install certain releases of openstack? I went to add some new compute nodes to my mitaka based cloud and am getting errors like http://paste.openstack.org/show/589532/ which looks like its getting conflicts with newton packages | 23:22 |
*** dfisher has quit IRC | 23:24 | |
mwhahaha | patrickeast: it's hard to say without seeing your repo configuration, it seems that you've got newton yum repos configured which would be a problem | 23:27 |
openstackgerrit | Bradley Klein proposed openstack/puppet-monasca: Add support for monasca-notification hipchat and slack https://review.openstack.org/397989 | 23:34 |
patrickeast | mwhahaha: oooh yea, sure enough i had both newton and mitaka rdo repos, removing the newton one fixes the issues... guess i just need to track back which script installed the latest ones too | 23:40 |
patrickeast | mwhahaha: thanks for the pointer | 23:40 |
mwhahaha | np | 23:40 |
*** cdelatte has joined #puppet-openstack | 23:49 | |
*** rcernin has quit IRC | 23:52 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!