Monday, 2017-07-31

« Sunday, 2017-07-30 Index Tuesday, 2017-08-01 »
*** masber has joined #openstack-sahara00:05
*** dave-mcc_ has joined #openstack-sahara00:05
*** dave-mccowan has quit IRC00:07
*** deep-book-gk has joined #openstack-sahara01:39
*** deep-book-gk has left #openstack-sahara01:42
openstackgerritHongtao Zhang proposed openstack/sahara master: Enable some off-by-default checks  https://review.openstack.org/48737901:59
*** tuanluong has joined #openstack-sahara02:38
*** ukaynar has quit IRC02:44
*** ukaynar has joined #openstack-sahara02:46
*** ukaynar has quit IRC02:48
*** ukaynar has joined #openstack-sahara02:49
openstackgerritzhongshengping proposed openstack/puppet-sahara master: Update openstackdocstheme>=1.16.0  https://review.openstack.org/48906503:03
*** shuyingya has joined #openstack-sahara04:55
*** pgadiya has joined #openstack-sahara05:10
*** Poornima_K has joined #openstack-sahara05:11
*** dhellmann has quit IRC05:28
*** dhellmann has joined #openstack-sahara05:28
*** Poornima_K has quit IRC05:41
*** GK1wmSU has joined #openstack-sahara06:02
openstackgerritShu Yingya proposed openstack/sahara master: Fix UnicodeEncoding Error  https://review.openstack.org/48838706:03
*** GK1wmSU has left #openstack-sahara06:03
*** rcernin has joined #openstack-sahara06:11
*** _GK1wmSU has joined #openstack-sahara06:16
*** _GK1wmSU has left #openstack-sahara06:18
*** pcaruana has joined #openstack-sahara06:19
*** tuanluong has quit IRC06:19
*** Poornima_K has joined #openstack-sahara07:00
*** Poornima_K has quit IRC07:06
*** Poornima_K has joined #openstack-sahara07:08
*** Poornima_K has quit IRC07:09
*** Poornima_K has joined #openstack-sahara07:10
*** ukaynar has quit IRC07:17
*** tuanluong has joined #openstack-sahara07:18
*** Poornima_K has quit IRC07:23
*** dhellmann has quit IRC07:30
*** dhellmann has joined #openstack-sahara07:31
*** esikachev has joined #openstack-sahara07:38
*** Poornima_K has joined #openstack-sahara08:39
*** Poornima_K has quit IRC08:40
*** Poornima_K has joined #openstack-sahara08:42
*** Poornima_K has quit IRC08:43
*** pgadiya has quit IRC09:17
*** tosky has joined #openstack-sahara09:21
*** pgadiya has joined #openstack-sahara09:33
*** Poornima_K has joined #openstack-sahara09:37
*** Poornima_K has quit IRC09:50
*** esikachev has quit IRC10:09
*** esikachev has joined #openstack-sahara10:09
*** esikachev has quit IRC10:14
*** pgadiya has quit IRC10:36
*** esikachev has joined #openstack-sahara10:46
*** zemuvier has joined #openstack-sahara10:48
*** pgadiya has joined #openstack-sahara10:49
*** tuanluong has quit IRC11:08
*** ukaynar has joined #openstack-sahara11:09
*** ukaynar has quit IRC11:19
*** ukaynar has joined #openstack-sahara11:19
*** ukaynar has quit IRC11:23
*** ukaynar has joined #openstack-sahara11:23
*** ukaynar has quit IRC11:25
*** ukaynar has joined #openstack-sahara11:26
*** ukaynar has quit IRC11:30
*** ltosky[m] has quit IRC12:01
*** shuyingya has quit IRC12:35
*** ukaynar has joined #openstack-sahara12:57
*** shuyingya has joined #openstack-sahara12:59
*** lucasxu has joined #openstack-sahara13:02
*** shuyingya has quit IRC13:04
*** shuyingya has joined #openstack-sahara13:08
*** jeremyfreudberg has joined #openstack-sahara13:10
*** shuyingya has quit IRC13:11
*** ukaynar has quit IRC13:12
*** ukaynar has joined #openstack-sahara13:12
*** shuyingya has joined #openstack-sahara13:13
*** ukaynar has quit IRC13:17
*** ukaynar has joined #openstack-sahara13:21
*** shuyingya has quit IRC13:23
*** esikachev has quit IRC13:28
*** shuyingya has joined #openstack-sahara13:39
*** shuyingya has quit IRC13:43
*** pgadiya has quit IRC13:56
*** ssmith has joined #openstack-sahara13:57
toskyI almost nailed the sahara-dashboard failure: it looks like an issue in horizon14:27
jeremyfreudbergtosky, nice14:27
toskyhorizon bug coming, with fix and dependency14:27
toskyand then discussion with the horizon team (which I already nagged :)14:28
ssmithGood morning gentlemen.  We now have ephemeral volumes working and the instances are launching but the heat stacks are getting stuck.  Any hints on what we can look at? http://imgur.com/R4Nx3sa14:31
jeremyfreudberghey ssmith, glad you finally got the storage working... anything in heat-engine logs?14:32
*** openstackgerrit has quit IRC14:33
*** ltosky[m] has joined #openstack-sahara14:39
*** shuyingya has joined #openstack-sahara14:40
*** shuyingya has quit IRC14:44
*** openstackgerrit has joined #openstack-sahara14:52
openstackgerritLuigi Toscano proposed openstack/sahara-dashboard master: Switch render() arguments to the new way  https://review.openstack.org/48842514:52
jeremyfreudbergtosky, nice work14:53
toskywhack-a-mole and yak shaving to the top14:54
jeremyfreudberg:D14:54
*** zemuvier has quit IRC14:55
*** ukaynar has quit IRC15:01
*** rcernin has quit IRC15:03
*** pcaruana has quit IRC15:04
*** ukaynar has joined #openstack-sahara15:06
*** ukaynar_ has joined #openstack-sahara15:08
*** ukaynar has quit IRC15:11
*** esikachev has joined #openstack-sahara15:14
*** lucasxu has quit IRC15:26
*** shuyingya has joined #openstack-sahara15:29
*** shuyingya has quit IRC15:34
toskytellesnobrega, jeremyfreudberg, esikachev: the horizon patch is going to land, so https://review.openstack.org/488425 should unlock sahara-dashboard15:44
toskyand then we can ninja merge the request from the update bot for example15:44
toskynow I can take a bre.. no, the OTHER patch15:44
tosky:D15:44
*** jeremyfreudberg has quit IRC15:46
*** ukaynar_ has quit IRC15:48
*** ukaynar has joined #openstack-sahara15:49
tomtomtomI'm seeing this in the sahara logs, 2017-07-31 15:37:24.876 223 INFO keystonemiddleware.auth_token [-] Rejecting request  anyone else experience this issue and solve it?  According to my keystone auth section the username(sahara) and password are correct.15:52
*** ukaynar has quit IRC15:53
toskytomtomtom: master?15:53
toskytomtomtom: how did you deploy it?15:53
tomtomtomthis is an openstack newton version deployed with openstack-ansible.15:54
toskyso, ‎jeremyfreudberg‎ have seen that message, and I've seen it too, but in different context15:54
toskyin my case (master deployment), haproxy tries to contact periodically (not sure if it's misconfigured or not)15:54
toskyI don't remember what was the issue in Jeremy's deployment15:55
toskyhow much frequently do you see it?15:55
tomtomtomevery 10 seconds or so.15:56
toskyas mentioned, I'15:56
toskygrmf15:56
toskyI'm still not sure if it's an issue in my case (haproxy); I'd suggest you to wait for Jeremy and see if he has more details15:57
tomtomtomok15:57
*** lucasxu has joined #openstack-sahara16:10
*** ukaynar has joined #openstack-sahara16:11
*** ukaynar has quit IRC16:18
*** shuyingya has joined #openstack-sahara16:19
*** shuyingya has quit IRC16:23
*** shuyingya has joined #openstack-sahara16:34
*** shuyingya has quit IRC16:39
*** esikachev has quit IRC16:44
ssmithNo errors in heat logs and everything stops after creating the servers.  Turning on heat debug logging17:04
*** jeremyfreudberg has joined #openstack-sahara17:14
jeremyfreudbergtomtomtom, about your issue17:15
jeremyfreudbergthe "Rejecting request" is not actual an error or misconfiguration of sahara itself17:15
jeremyfreudbergas tosky said, in his case it has haproxy sending a request to the sahara service, in my case it was nagios17:16
jeremyfreudbergbut these request are not wellformed for a service using keystonemiddleware for auth17:16
jeremyfreudberghence, rejecting request17:16
jeremyfreudbergnothing really to worry about17:16
ssmithSo it's stuck in the image above with heat message "Task create from HeatWaitCondition "sparkslave-wc-waiter" Stack "Hellboy5d8ed36df-sparkslave-wrdnopuuf7zq-0-sjjl2ht437x7" " repeating over and over17:17
jeremyfreudbergssmith, try heat_enable_wait_condition=False in DEFAULT of sahara.conf17:19
ssmithIs there anyway to see what it's waiting on?17:19
jeremyfreudbergssmith, wait condition is used for getting status of vm (has it booted, is ssh ready) but in my deployment it never worked.... on the other hand, we knew of some deployments where it was necessary17:20
*** ukaynar has joined #openstack-sahara17:21
*** shuyingya has joined #openstack-sahara17:23
jeremyfreudbergssmith, your cloud is newton?17:24
*** shuyingya has quit IRC17:28
*** ukaynar has quit IRC17:29
*** tosky has quit IRC17:32
*** esikachev has joined #openstack-sahara17:41
*** esikachev has quit IRC17:45
*** ukaynar has joined #openstack-sahara17:50
*** esikachev has joined #openstack-sahara17:52
*** tosky has joined #openstack-sahara17:56
*** tosky has quit IRC17:56
*** esikachev has quit IRC17:57
*** tosky has joined #openstack-sahara17:57
*** tosky has quit IRC17:58
*** tosky has joined #openstack-sahara17:58
*** ukaynar has quit IRC17:59
*** ukaynar has joined #openstack-sahara17:59
openstackgerritEvgeny Sikachev proposed openstack/sahara-ci-config master: [wip]Added ability to deploy zuul  https://review.openstack.org/48928918:01
*** shuyingya has joined #openstack-sahara18:03
*** ukaynar has quit IRC18:03
*** shuyingya has quit IRC18:07
ssmithjeremyfreudberg: Yes, it's Newton18:08
ssmithNow all Heat stack complete but status in Sahara is "Waiting"18:08
jeremyfreudbergssmith, ok, if it's newton there was some bugfixes regarding heat condition which should be there (if you were mitaka or earlier, could be a problem). how long has sahara been on "Waiting" ? waiting means can't ssh in18:09
ssmithI was in a 1/2 hour meeting and started it before then so 35 minutes18:11
ssmithI can ssh into it with the supplied key (which is mine on the OS system) with user ubuntu18:12
jeremyfreudbergssmith, can you check instance log (see if there is a key with comment "Generated-By-Sahara" in cloud-init logs)18:13
ssmithNo there is not just "Generated-by-Nova "18:14
* jeremyfreudberg thinks18:19
jeremyfreudbergssmith, not exactly sure, but as a first step, might as well check authorized_keys file of ubuntu user instead of cloud-init logs18:20
jeremyfreudbergjust to cover all our bases18:20
ssmithJust one on the instance "generated-by-nova"18:21
*** ukaynar has joined #openstack-sahara18:21
jeremyfreudbergok, that's unfortunate, but at least consistent18:21
jeremyfreudbergare you using floating IP or internal IP ?18:21
jeremyfreudbergin sahara configuration, I mean18:22
ssmithboth18:22
jeremyfreudbergso use_floating_ips is True, but not all instances have floating? just to clarify18:22
ssmithOne master, one slave both with a private tenant IP both with Floating IPs18:23
jeremyfreudbergok18:25
ssmithWhich key should be assigned to the stack? I used one that was generated for myself on the pulldown list18:26
jeremyfreudbergssmith, generally there would be two18:26
jeremyfreudbergone that sahara creates for itself18:26
jeremyfreudbergand one that is the user's18:27
jeremyfreudbergright now, i'm trying to remember how the "Generated-By-Sahara" keypair gets inserted, i know the user's keypair comes through the heat template itself, but the management one is something else18:27
jeremyfreudbergso here's the answer about that18:30
jeremyfreudberguser keypair is the keypair that nova knows about18:31
jeremyfreudbergmanagement keypair is inserted as part of the "User data" script18:31
jeremyfreudbergso i guess to diagnose18:32
jeremyfreudbergsee if you can launch a regular instance with a script in "User data"18:32
jeremyfreudberg^ ssmith18:34
ssmithBefore trying that I was just looking through the logs and see " Failed fetching userdata from url http://169.254.169.254/2009-04-04/user-data18:41
ssmith" so could that be the issue?18:41
jeremyfreudbergssmith, that's the issue18:41
jeremyfreudbergto clarify, some problem with you cloud-init / nova metadata server18:46
toskyand/or network setup18:47
jeremyfreudbergtosky - true, ssmith ^18:47
ssmithOn machines with non-floating sets 169.254.169.254 | 10.0.128.10 and on this tenant with both sets to 169.254.169.254 | 172.16.1.118:49
*** ukaynar has quit IRC18:50
*** ukaynar has joined #openstack-sahara18:50
jeremyfreudbergssmith, what exactly are you referring to? sorry18:53
*** ukaynar has quit IRC18:54
*** ukaynar has joined #openstack-sahara18:55
ssmithNVM, I ssh into the spark master and it can telnet to 169.254.169.254 8018:57
*** ukaynar has quit IRC19:06
*** shuyingya has joined #openstack-sahara19:12
*** shuyingya has quit IRC19:16
*** esikachev has joined #openstack-sahara19:33
esikachevjeremyfreudberg: ping19:35
ssmithjeremyfreudberg: would you have something that would be a good test for the user data script?19:35
jeremyfreudbergwoah, two pings at once19:35
esikachev:)19:35
jeremyfreudbergssmith, just a shell script with a valid shebang and echo hello world19:35
jeremyfreudbergif you can run that you can run anything19:35
jeremyfreudbergesikachev, what's up?19:36
esikachevjeremyfreudberg: everything is ok with HDDs on cloud for sahara-ci?19:36
esikachevis too slow :(19:36
jeremyfreudbergesikachev, probably it's not ok :(19:37
esikachevthis is happend periodicaly19:37
jeremyfreudbergi see some filesystem junk messages in your instance log...19:38
ssmithIf I do a curl from the cluster sparkmaster I get this curl http://169.254.169.254/2009-04-0419:38
ssmith --> meta-data/19:38
ssmith no /user-data19:38
jeremyfreudbergesikachev, unfortunately it happens a lot19:38
jeremyfreudbergit can all be recovered and fixed, but it might happen a gain19:38
openstackgerritEvgeny Sikachev proposed openstack/sahara-ci-config master: [wip]Added ability to deploy zuul  https://review.openstack.org/48928919:40
jeremyfreudbergesikachev, i sent a note for it to be recovered, hopefully fixed soon19:40
jeremyfreudbergssmith, very interesting19:41
esikachevok. thank you. i finished for today with instance19:41
jeremyfreudbergssmith, investigating19:43
ssmithWait, yes there is....it's just down on the input line  curl http://169.254.169.254/2009-04-0419:43
ssmithmeta-data/19:43
ssmithuser-dataubuntu@hellboy-6-sparkmaster-0:~$19:43
jeremyfreudbergyep, there's no newline so hard to see19:44
ssmithThen it just hangs there  curl -v http://169.254.169.254/2009-04-04/user-data/19:44
ssmith* Hostname was NOT found in DNS cache19:44
ssmith*   Trying 169.254.169.254...19:44
ssmith* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)19:44
ssmith> GET /2009-04-04/user-data/ HTTP/1.119:44
ssmith> User-Agent: curl/7.35.019:44
ssmith> Host: 169.254.169.25419:44
ssmith> Accept: */*19:44
ssmith>19:44
jeremyfreudbergssmith, not sure how to diagnose, really not something i'm very familiar with19:47
ssmithLaunched server from Horizon with a config script and user-data works19:56
*** esikachev has quit IRC19:57
*** shuyingya has joined #openstack-sahara20:01
jeremyfreudbergssmith, hmm, then i would expect it to work on sahara instances too20:02
*** shuyingya has quit IRC20:05
ssmithjeremyfreudberg: this is in the DB so looks like it's there. Looks hashed but does it look normal? http://imgur.com/j94gyaJ20:43
jeremyfreudbergssmith, you got this from SELECT management_private_key FROM <sahara db name>.clusters; ?20:44
jeremyfreudbergor something else?20:44
ssmithselect user_data from instance20:47
jeremyfreudbergssmith, oh, the nova table20:47
jeremyfreudbergnot familiar with what it's supposed to look like in there20:47
jeremyfreudbergand this is the entry referring to sahara-created instance?20:48
ssmithIs there a way to paste in a pre-generated key pair on the horizon UI?20:48
jeremyfreudbergssmith, there is20:48
jeremyfreudbergaccess and security -> key pairs -> import key pair20:49
jeremyfreudbergnot sure if that will help you much20:49
jeremyfreudbergany way, checked my own nova db, my user_data column also looks like yours20:51
jeremyfreudbergso not to worry about that specifically, ssmith20:51
jeremyfreudbergssmith, sorry have to go afk now.20:53
*** jeremyfreudberg has quit IRC20:53
ssmithok, thanks20:53
*** shuyingya has joined #openstack-sahara21:00
*** lucasxu has quit IRC21:02
*** shuyingya has quit IRC21:04
ssmithSo we've discovered that the spark instance can reach user-data if the mtu is lowered to 1500.  We run on 10G networks so have set the mtu on the servers to 9000.  But dang if we can find out where the metadata service is set to 150021:58
*** shuyingya has joined #openstack-sahara21:59
*** shuyingya has quit IRC22:03
toskyin most of the case, sahara issues are $some_other_service issue22:17
ssmithWierd that highest mtu it will work with is 734022:24
*** shuyingya has joined #openstack-sahara22:38
*** shuyingya has quit IRC22:43
*** ukaynar has joined #openstack-sahara22:53
openstackgerritMerged openstack/sahara-dashboard master: Switch render() arguments to the new way  https://review.openstack.org/48842523:30
toskyuh, I haz the power23:38
tellesnobregatosky, you do :)23:38
tellesnobregaI was going to ask you to check if you have +2 powers already23:38
tellesnobregabut thought you wouldn't be online now23:38
toskyehm, yeah, I'm usually late23:39
tellesnobregaI had the worse trip of my life today23:39
toskyoh23:39
tellesnobregafrom my parent's trip to mine, it is usually a 1:30h drive23:40
tellesnobregait took 7h today23:40
toskythe time of an intercontinental flight23:40
toskytraffic jam?23:40
tellesnobregathere was a protest due to an accident that happened friday on the highway23:40
toskyI see23:41
tellesnobregaI was stuck from 11:30h until 17h23:41
toskyand a lot of people with you, I guess23:42
tellesnobregayes23:42
toskya bad mishap, but at least it's over23:56
toskythe power seems to work, I ninja approved one change (maybe two) coming from the bots now that sahara-dashboard works23:56
toskyand thanks!23:56
* tosky disappears23:56
*** tosky has quit IRC23:56
*** shuyingya has joined #openstack-sahara23:57
*** jeremyfreudberg has joined #openstack-sahara23:57
tellesnobregasee ya23:59
« Sunday, 2017-07-30 Index Tuesday, 2017-08-01 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!