| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-addons master: Gate: migrate to zuul v3 https://review.openstack.org/519591 | 00:22 |
|---|---|---|
| *** felipemonteiro_ has quit IRC | 01:03 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Cinder: set ownership of co-ordination backend for remaining services https://review.openstack.org/545187 | 01:51 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Kolla Newton Gate https://review.openstack.org/544021 | 01:53 |
| *** unicell has quit IRC | 02:00 | |
| openstackgerrit | Merged openstack/openstack-helm master: Cinder: set ownership of co-ordination backend for backup service https://review.openstack.org/545126 | 02:12 |
| *** unicell has joined #openstack-helm | 02:45 | |
| *** unicell has quit IRC | 02:50 | |
| openstackgerrit | Ganesh Maharaj Mahalingam proposed openstack/openstack-helm-infra master: WIP - Add support for managing generic loopback devices. - Setup loopback devices - Ensure devices are available after reboot - Easy to create generic loopback devices https://review.openstack.org/539019 | 03:06 |
| *** unicell has joined #openstack-helm | 03:47 | |
| *** yamamoto has joined #openstack-helm | 04:05 | |
| *** yamamoto_ has joined #openstack-helm | 04:51 | |
| *** yamamoto has quit IRC | 04:55 | |
| *** coolsvap has joined #openstack-helm | 06:03 | |
| *** openstack has joined #openstack-helm | 06:28 | |
| *** ChanServ sets mode: +o openstack | 06:28 | |
| *** yamamoto_ has quit IRC | 07:40 | |
| *** yamamoto has joined #openstack-helm | 07:42 | |
| *** yamamoto has joined #openstack-helm | 07:43 | |
| *** yamamoto has quit IRC | 07:48 | |
| *** yamamoto has joined #openstack-helm | 08:44 | |
| *** unicell1 has joined #openstack-helm | 08:46 | |
| *** unicell has quit IRC | 08:47 | |
| *** yamamoto has quit IRC | 08:50 | |
| *** yangyapeng has joined #openstack-helm | 08:57 | |
| *** yamamoto has joined #openstack-helm | 08:59 | |
| *** yamamoto has quit IRC | 09:15 | |
| *** unicell1 has quit IRC | 09:16 | |
| *** MarkBaker has joined #openstack-helm | 09:59 | |
| *** yamamoto has joined #openstack-helm | 10:15 | |
| *** MarkBaker_ has joined #openstack-helm | 10:18 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** yamamoto has quit IRC | 10:20 | |
| *** MarkBaker has quit IRC | 10:22 | |
| *** MarkBaker_ has quit IRC | 10:32 | |
| *** MarkBaker_ has joined #openstack-helm | 10:47 | |
| *** yamamoto has joined #openstack-helm | 10:51 | |
| *** MarkBaker_ has quit IRC | 10:55 | |
| *** MarkBaker_ has joined #openstack-helm | 11:10 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** yamamoto has joined #openstack-helm | 11:59 | |
| *** yamamoto has quit IRC | 12:08 | |
| *** yamamoto has joined #openstack-helm | 12:11 | |
| *** MarkBaker_ has quit IRC | 12:14 | |
| *** julim has quit IRC | 12:20 | |
| *** MarkBaker has joined #openstack-helm | 12:22 | |
| *** yamamoto has quit IRC | 12:41 | |
| *** yamamoto has joined #openstack-helm | 13:15 | |
| *** coolsvap has quit IRC | 13:22 | |
| *** julim has joined #openstack-helm | 13:36 | |
| *** julim has quit IRC | 13:37 | |
| *** julim has joined #openstack-helm | 13:38 | |
| *** openstackgerrit has joined #openstack-helm | 13:51 | |
| openstackgerrit | dave kormann proposed openstack/openstack-helm master: Ceph device naming by physical path https://review.openstack.org/545298 | 13:51 |
| *** felipemonteiro_ has joined #openstack-helm | 14:30 | |
| mattmceuen | Merry Office Hours everyone | 14:30 |
| mattmceuen | @gmmaha @renmak I too am testing https://review.openstack.org/#/c/539019/ on my laptop (ubuntu 1604), and have been running into issues | 14:34 |
| *** felipemonteiro__ has joined #openstack-helm | 14:34 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Kolla Newton Gate https://review.openstack.org/544021 | 14:35 |
| mattmceuen | Here's what I see: | 14:36 |
| mattmceuen | 1) run loopback creation the first time, works fine, devices can be used | 14:36 |
| mattmceuen | 2) upon reboot, devices are gone, and the loopback creation script hangs (I believe due to iscsi issues but i haven't dug more yet) | 14:36 |
| mattmceuen | 3) the iscsi and target systemd services won't start anymore | 14:36 |
| mattmceuen | The only way I've resolved #3 is reinstalling the os, but then I once again seem to get one shot at loopback devices before reboot kills it all again | 14:36 |
| *** felipemonteiro_ has quit IRC | 14:37 | |
| openstackgerrit | Merged openstack/openstack-helm-addons master: Gate: migrate to zuul v3 https://review.openstack.org/519591 | 14:39 |
| mattmceuen | @gmmaha @renmak here are my logs for the iscsi services: | 14:39 |
| mattmceuen | https://www.irccloud.com/pastebin/uHUlf4Xg/ | 14:39 |
| *** dansmith is now known as superdan | 14:43 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-addons master: Remove elasticsearch, fluentd, kibana from osh-addons https://review.openstack.org/540508 | 14:45 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP: Update rabbitmq dashboard to support multiple deployments https://review.openstack.org/545350 | 14:49 |
| *** jistr is now known as jistr|mtg | 14:52 | |
| *** felipemonteiro__ has quit IRC | 15:06 | |
| *** felipemonteiro_ has joined #openstack-helm | 15:06 | |
| *** Mdamkot has joined #openstack-helm | 15:06 | |
| *** Mdamkot has quit IRC | 15:10 | |
| d|k | mattmcuen: probably obvious, but i suspect this _has_ to be related to the problem you're having: | 15:11 |
| d|k | mattmceuen: |ExecStartPre=/sbin/iscsiadm -m discovery -t st -p 0.0.0.0 (code=exited, status=4) | 15:11 |
| d|k | that ip address is certainly wrong. | 15:12 |
| mattmceuen | d|k agree - I believe that's the iscsi service trying to connect to the target service (which is down) | 15:12 |
| d|k | seems likely that the listen-address for the target service is being used by the initiator as its destination address. dunno why that would be. | 15:12 |
| d|k | ... even if the target service weren't down, that connection would fail, of course. | 15:13 |
| *** jistr|mtg is now known as jistr | 15:15 | |
| d|k | owait, i take it back. guess that connection to 0.0.0.0 works if the service is running, sorry. | 15:17 |
| mattmceuen | ok whew - looks like it's hard coded here :) https://review.openstack.org/#/c/539019/37/tools/gate/playbooks/loopback-devices-create/templates/ubuntu-open-iscsi.service.j2@22 | 15:17 |
| d|k | it seems like a _bit_ of an odd choice -- you'd think maybe localhost would be a better choice. but hey, if it normally works ... | 15:19 |
| *** eeiden has joined #openstack-helm | 15:23 | |
| d|k | mattmceuen: did you happen to notice if the target kernel modules were loaded when the target and initiator wouldn't start? | 15:23 |
| *** felipemonteiro__ has joined #openstack-helm | 15:26 | |
| *** felipemonteiro_ has quit IRC | 15:29 | |
| openstackgerrit | Merged openstack/openstack-helm-addons master: Remove elasticsearch, fluentd, kibana from osh-addons https://review.openstack.org/540508 | 15:35 |
| *** MikeG451_ has joined #openstack-helm | 15:38 | |
| osh-chatbot | <renmak> @mattmceuen Matt, Ganesh and i retested that PS yesterday and found an issue with order in which services are being started after reboot. We did tested in two env and merged a small fix. However CI gate was not properly functioning last night so waiting to rerun check and create output. | 15:40 |
| osh-chatbot | <renmak> let me kick of CI gate again for OSH-infra | 15:40 |
| *** felipemonteiro__ has quit IRC | 16:02 | |
| *** seaneagan has joined #openstack-helm | 16:05 | |
| *** renmak_ has joined #openstack-helm | 16:05 | |
| *** renmak__ has joined #openstack-helm | 16:05 | |
| mattmceuen | thanks renmak. Using the patch prior to the one you & gmmaha did last night, I'm able to recover from my loopback hell if I delete /var/lib/iscsi-loopback. I'll give your latest patch a try now! | 16:08 |
| osh-chatbot | <renmak> oh wait | 16:10 |
| portdirect | renmak: any chance you'd be able to have a look at: https://review.openstack.org/#/c/540630/ soon? | 16:11 |
| portdirect | would be great to get this closer to the line, and consolated to a single helm toolkit macro ala https://github.com/openstack/openstack-helm/blob/master/helm-toolkit/templates/manifests/_job-ks-user.yaml.tpl | 16:12 |
| osh-chatbot | <renmak> yes Pete, In process of testing my changes. I was out in morning yesterday but that PS is my priority. | 16:12 |
| portdirect | awesome, thanks dude | 16:12 |
| osh-chatbot | <renmak> sure np | 16:12 |
| osh-chatbot | <renmak> that PS should be ready for another review soon | 16:13 |
| osh-chatbot | <renmak> @mattmceuen If you are recreating loopback devices, you have to delete that directory. I would suggest to use function make dev-deploy loopback-devices-create make dev-deploy loopback-devices-validate and when you want to create new devices make dev-deploy loopback-devices-remove | 16:14 |
| d|k | portdirect, renmak: in case you're interested, hardware path device targeting patch is here: https://review.openstack.org/#/c/545298/ | 16:15 |
| mattmceuen | @renmak the only reason I needed to recreate them was because they weren't surviving reboot. If they survive reboot, I'm happy leaving them as-is across redeployments. But I'll keep that in mind, I'm sure I'll need to delete them occasionally. | 16:16 |
| osh-chatbot | <renmak> the loopback devices (using targetcli) sometime provide different results in diff env. I have been monitoring PS CI gate and notice that devices are surviving reboot. @mattmceuen can you give latest PS a try please? | 16:18 |
| mattmceuen | Yep, will let you know in 5-10 mins using the latest code | 16:19 |
| mattmceuen | loopback devs not surviving reboot, using the latest PS. What diagnosis can I do that would help renmak? | 16:28 |
| *** felipemonteiro has joined #openstack-helm | 16:45 | |
| *** MikeG451_ has quit IRC | 16:45 | |
| *** MarkBaker has quit IRC | 17:00 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Fix Ceph Grafana dashboard https://review.openstack.org/545389 | 17:03 |
| gmmaha | mattmceuen: just got into work | 17:04 |
| gmmaha | the first error where target.service was not able to shutdown was an issue that got resolved once we upgraded the kernel using the OSH play | 17:05 |
| gmmaha | mattmceuen: for the latest, can i get anything you have in dmesg, and the systemctl status for target, iscsid and iscsi | 17:05 |
| gmmaha | d|k: mattmceuen: i hard-coded to 0.0.0.0 cause for some fantastic reason (i wish i knew why) that IP worked on fedora and centos but not 127.0.0.1 | 17:06 |
| gmmaha | glad to change it to local host if you think that will help | 17:06 |
| gmmaha | mattmceuen: and wierdly with the latest PS i tested my env and devices survive post boot and renmak_ mentioned that he was able to follow zuul logs and he saw the same in with CI | 17:07 |
| d|k | gmmaha: 0.0.0.0 _does_ work so it seems silly to say it should change, but given we know we want to connect to localhost, it might make sense to use that instead | 17:09 |
| gmmaha | d|k: as long as it works i am fine changing it to localhsot | 17:10 |
| gmmaha | let me see if i can test it on a fedora VM | 17:10 |
| *** felipemonteiro has quit IRC | 17:10 | |
| *** MarkBaker has joined #openstack-helm | 17:13 | |
| d|k | kewl.... though like i said, given using 0 works, it's prolly not particularly important | 17:13 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Update rabbitmq dashboard to support multiple deployments https://review.openstack.org/545350 | 17:18 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Update rabbitmq dashboard to support multiple deployments https://review.openstack.org/545350 | 17:25 |
| gmmaha | d|k: fedora didn't play nice with changing 0.0.0.0 to localhost | 17:37 |
| portdirect | was it ok with 127.0.0.1? | 17:38 |
| portdirect | https://review.openstack.org/#/c/484982/25/tools/gate/funcs/common.sh | 17:38 |
| gmmaha | d|k: http://paste.openstack.org/show/674726/ | 17:38 |
| gmmaha | portdirect: it didn't play nice with 127.0.0.1 either. that's why i set it to 0.0.0.0 | 17:39 |
| gmmaha | but i can tst it again with 127.0.0.1 | 17:39 |
| gmmaha | to the best of my recollcetion, it didn't. won't hurt trying it out | 17:39 |
| mattmceuen | gmmaha thanks -- here are the logs after rebooting, using the latest -- https://pastebin.com/a1Ettjws | 17:39 |
| d|k | gmmaha: i do recall that there was some issue when doing the listener bind in targetcli that on some platforms it didn't bind localhost unless you forced it to. | 17:39 |
| portdirect | if you looks at 76-77 in that ps I linked to i had to monkey around to get it to work | 17:39 |
| * gmmaha goes to look at the PS | 17:40 | |
| d|k | yah, what i'm recalling is undoubtedy lwhat @portdirect is describing | 17:40 |
| gmmaha | mattmceuen: do you have anything listening on port 3260? | 17:42 |
| gmmaha | portdirect: aah.. so ubuntu no play nice with 0.0.0.0 :| | 17:43 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Remove grafana etcd dashboard https://review.openstack.org/545401 | 17:43 |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-addons master: Add artifactory chart https://review.openstack.org/539333 | 17:44 |
| gmmaha | portdirect: d|k: sorry read it the other way around. duh! if ubuntu cannot play nice with 127.0.0.1 and fedora seems to work fine with 0.0.0.0, won't it be alright to just keep 0.0.0.0 and not wrangle to be 127.0.0.1 | 17:46 |
| gmmaha | i am not an expert in this in any form. please feel free to correct me if i am completely wrong about this | 17:47 |
| d|k | gmmaha: as i said, if 0.0.0.0 works (and it pretty clearly does) in both cases i don't think there's a strong reason to change. i'm just not used to seeing that as a destination address | 17:48 |
| gmmaha | d|k: got it. :) | 17:49 |
| d|k | gmmaha: i do think if you're intending to talk on the loopback interface you should explicitly use that address, but maybe in this case all you care is taht it actually connects. | 17:49 |
| gmmaha | was just trying to make sure i am not missing anything obvious using 0.0.0.0 | 17:49 |
| mattmceuen | gmmaha - nothing running on 3260 | 17:49 |
| osh-chatbot | <renmak> Matt is having similar issue as what i have been experience. After reboot, no one is listening on 3260 | 17:50 |
| gmmaha | mattmceuen: thanks. this is bizzare. | 17:50 |
| d|k | are the kernel modules there? | 17:50 |
| mattmceuen | which ones? | 17:50 |
| gmmaha | d|k: when i was checking renmak_ machine yesterday, they were all there. | 17:50 |
| mattmceuen | https://www.irccloud.com/pastebin/yMUZpPby/ | 17:50 |
| d|k | should at least be iscsi_target_mod | 17:50 |
| gmmaha | all the usual suspects are there mattmceuen | 17:51 |
| mattmceuen | yup, got that one | 17:51 |
| gmmaha | that and iscsi_tcp (which was a pain on non-ubuntu hosts) | 17:51 |
| d|k | yeah, looks reasonable. | 17:51 |
| openstackgerrit | Merged openstack/openstack-helm master: Cinder: set ownership of co-ordination backend for remaining services https://review.openstack.org/545187 | 18:06 |
| gmmaha | mattmceuen: i am trying to create this all in GCE to see if i can reproduce the error. | 18:26 |
| gmmaha | my usual test env isn't showing me this which is totally bizzarre.. | 18:26 |
| mattmceuen | my test env is a laptop fwiw - fresh install of ubuntu desktop | 18:28 |
| osh-chatbot | <renmak> k Matt, can you try this command please `targetcli ls` | 18:28 |
| osh-chatbot | <renmak> what do you get? | 18:28 |
| mattmceuen | https://www.irccloud.com/pastebin/bWE9d3w2/ | 18:29 |
| gmmaha | mattmceuen: thanks and that is interesting./ | 18:29 |
| osh-chatbot | <renmak> k then please try following. `sudo targetcli saveconfig` | 18:29 |
| mattmceuen | https://www.irccloud.com/pastebin/Y2Q9pOLv/ | 18:29 |
| osh-chatbot | <renmak> say Y on prompt | 18:29 |
| mattmceuen | yikes | 18:30 |
| mattmceuen | https://www.irccloud.com/pastebin/Mux6HVFO/ | 18:30 |
| osh-chatbot | <renmak> parse error? | 18:30 |
| mattmceuen | yup | 18:30 |
| osh-chatbot | <renmak> yeap so targetcli is not able to save config | 18:30 |
| osh-chatbot | <renmak> that is due to package version | 18:30 |
| mattmceuen | aha! | 18:30 |
| mattmceuen | of targetcli? | 18:30 |
| osh-chatbot | <renmak> k i have made this change before | 18:31 |
| osh-chatbot | <renmak> let me find it and we can test in your env | 18:31 |
| * gmmaha 's hatred towards targetcli just went hulk mode | 18:31 | |
| osh-chatbot | <renmak> k Matt, ``` # in my environment, latest version of pyparsing was causing parsing error. I had # downgrade version in order to make it work. sudo -H -E pip uninstall --yes pyparsing sudo -H -E pip install -U pyparsing==2.0.3 ``` | 18:31 |
| osh-chatbot | <renmak> Matt, please run above two commands and then create loopback devices | 18:32 |
| mattmceuen | awesome - going to reboot and try. | 18:33 |
| osh-chatbot | <renmak> k i am testing in my env too | 18:34 |
| *** eeiden has quit IRC | 18:34 | |
| * gmmaha just noticed that his pyparsing is 2.0.3 by default | 18:37 | |
| osh-chatbot | <renmak> no i had higher version | 18:37 |
| osh-chatbot | <renmak> yessss | 18:37 |
| osh-chatbot | <renmak> devices are surviving reboot :slightly_smiling_face: | 18:37 |
| osh-chatbot | <renmak> atleast in my env | 18:38 |
| osh-chatbot | <renmak> let's see if Matt can confirm as well | 18:38 |
| osh-chatbot | <renmak> i had `pyparsing-2.2.0` | 18:39 |
| mattmceuen | d'oh, the ansible script re-installs pyparsing-2.2.0 | 18:44 |
| *** eeiden has joined #openstack-helm | 18:46 | |
| osh-chatbot | <renmak> yeah so before you run create devices, you can downgrade version as above. | 18:46 |
| osh-chatbot | <renmak> i am writing a Ansible task to do that as part of device create | 18:46 |
| *** unicell has joined #openstack-helm | 18:47 | |
| gmmaha | mattmceuen: i never thought targetcli would be the culprit. my bad. | 18:56 |
| gmmaha | glad it all worked out | 18:56 |
| osh-chatbot | <renmak> Well let's see if Matt does confirm that it works. | 18:59 |
| osh-chatbot | <renmak> pushing changes to add a task to update pip module version | 18:59 |
| mattmceuen | Yep that resolved it :) | 19:01 |
| mattmceuen | oh no worries gmmaha - appreciate your help and glad my problem was helpful | 19:01 |
| gmmaha | anytime.. | 19:02 |
| osh-chatbot | <renmak> :+1: great! | 19:02 |
| * gmmaha watches for renmak_' | 19:03 | |
| * gmmaha watches for renmak_ 's change to test it on fedora and ubuntu | 19:03 | |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm-infra master: WIP - Add support for managing generic loopback devices. - Setup loopback devices - Ensure devices are available after reboot - Easy to create generic loopback devices https://review.openstack.org/539019 | 19:04 |
| osh-chatbot | <renmak> go ahead @gmmaharaj | 19:04 |
| *** unicell has quit IRC | 19:22 | |
| gmmaha | renmak_: mattmceuen: tested the latest patch on both fedora and ubuntu and they seem to be working just fine | 19:32 |
| gmmaha | though my env isn't really going to help | 19:34 |
| osh-chatbot | <renmak> yeah gate is also showing devices are there after reboot | 19:36 |
| osh-chatbot | <renmak> waiting on gate to finish | 19:36 |
| gmmaha | and fedora timedout again. i need to check with them to see what can be done. | 19:40 |
| osh-chatbot | <renmak> k i am going to commit one more update. I am going to reduce number of disks being create and also size | 19:55 |
| osh-chatbot | <renmak> just wondering if space is an issue | 19:56 |
| gmmaha | renmak_: hold-up. just talkoing to the infra guys | 19:56 |
| osh-chatbot | <renmak> k | 19:56 |
| gmmaha | they made a recent change in the way timeout happens on the CI jobs and that might be the reason for it. | 19:56 |
| gmmaha | they are working on a patch to fix it and then we can re-test the patch. | 19:56 |
| gmmaha | doubt size is an issue given we have been able to create the disks just fine | 19:56 |
| osh-chatbot | <renmak> yeah even other PS are timing out in gate | 19:57 |
| osh-chatbot | <renmak> k then won't make any changes | 19:57 |
| gmmaha | yeah. all of openstack-helm-infra is hitting that problem | 19:57 |
| *** renmak__ has quit IRC | 19:57 | |
| *** renmak_ has quit IRC | 19:57 | |
| gmmaha | if it helps, http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-02-16.log.html#t2018-02-16T19:45:12 | 20:03 |
| srwilkers | yeah, just need to give it time at this point | 20:14 |
| srwilkers | http://grafana.openstack.org/dashboard/db/zuul-status, for what its worth | 20:14 |
| openstackgerrit | Merged openstack/openstack-helm master: mariadb: by default don't cluster https://review.openstack.org/543007 | 20:19 |
| gmmaha | srwilkers: thanks.. that is some fancy chart | 21:48 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Kolla Newton Gate https://review.openstack.org/544021 | 22:04 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Images: Move default to LOCI and Kolla newton gate https://review.openstack.org/544021 | 22:07 |
| *** eeiden has quit IRC | 22:07 | |
| gmmaha | renmak: d|k: mattmceuen: the system is failing cause of disk space.. atleast on fedora. http://logs.openstack.org/19/539019/38/check/openstack-helm-infra-fedora/c484630/job-output.txt.gz#_2018-02-16_22_36_15_079822http://logs.openstack.org/19/539019/38/check/openstack-helm-infra-fedora/c484630/job-output.txt.gz#_2018-02-16_22_36_15_079822 | 22:44 |
| gmmaha | err, sorry http://logs.openstack.org/19/539019/38/check/openstack-helm-infra-fedora/c484630/job-output.txt.gz#_2018-02-16_22_36_15_079822 | 22:44 |
| mattmceuen | oh my | 22:47 |
| gmmaha | and we definitely have to increase the timeout for our jobs. we are not finishing much in the allocated 30 mins.. will throw a patch up with it. | 22:48 |
| gmmaha | and wierdly the space isn't an issue on centos nand i bet on ubuntu too.. | 22:48 |
| portdirect | gmmaha: i think the jobs have a much longer timeout than 30? | 22:48 |
| portdirect | eg: https://review.openstack.org/#/c/543553/ | 22:49 |
| portdirect | and for osh-prime we explicity set it to 7200 seconds | 22:49 |
| portdirect | gmmaha: for some host profiles you'll want to use a device thats not the root dev | 22:51 |
| portdirect | http://logs.openstack.org/19/539019/38/check/openstack-helm-infra-fedora/c484630/job-output.txt.gz#_2018-02-16_22_35_50_180401 | 22:51 |
| portdirect | with the size of loopback devices your trying to make | 22:52 |
| portdirect | though for simplicity, could we not just reduce these to much lower limits, ie 1gb ? | 22:52 |
| gmmaha | portdirect: yeah that is what i am thinking, just wondering if ceph will work with a smaller journal and disk.. it should. | 22:53 |
| gmmaha | portdirect: and for the timeout i think they recently changed the timeout to being 30 mins and that is what made all our infra jobs fail which led to them making an additonal change to still get us the logs even if it times out | 22:54 |
| portdirect | ah - that makes sense | 22:55 |
| portdirect | lets get them set to 3600 secs then in infra | 22:55 |
| gmmaha | cool.. let me go ahead and make a patch | 22:55 |
| osh-chatbot | <renmak> ah so disk space was the issue. Let me update vars.yaml to only create 1 device for each osd, journal and swift | 22:58 |
| gmmaha | renmak: even with that i think we have to drop the sizes down.. | 22:58 |
| gmmaha | maybe like portdirect said, 2G for osd, journal and swift. | 22:58 |
| gmmaha | and i wonder if journal will work with 2G disk or will it complain that is low on space | 22:59 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Add network policy https://review.openstack.org/539753 | 22:59 |
| openstackgerrit | Ganesh Maharaj Mahalingam proposed openstack/openstack-helm-infra master: Update timeout for infra jobs https://review.openstack.org/545481 | 23:00 |
| osh-chatbot | <renmak> so when we specify 2GB, current code will create disks with 3GB space right. | 23:00 |
| gmmaha | right | 23:00 |
| osh-chatbot | <renmak> yeah i am not sure what is minimum OSD and Journal space is needed | 23:01 |
| gmmaha | portdirect: https://review.openstack.org/545481 | 23:01 |
| osh-chatbot | <renmak> i have seen Ceph spitting out an error if disk size is smaller | 23:01 |
| osh-chatbot | <renmak> i can't remember what that size was | 23:01 |
| osh-chatbot | <renmak> what? `OSDs should have plenty of hard disk drive space for object data. We recommend a minimum hard disk drive size of 1 terabyte` | 23:03 |
| gmmaha | renmak: maybe best to test it locally and then push the patch if you are doing it./. | 23:03 |
| gmmaha | wish i can do it.. but i have to shutdown all my dev envs.. building electrical upgrade over the long weekend :( | 23:03 |
| gmmaha | portdirect: it seems like fedora is the only one with the small root partition. ubuntu and centos is sort of fine. using an alternate drive would be good, only that it is not formatted and mounted by default | 23:05 |
| gmmaha | atleast from what lsblk is saying | 23:05 |
| portdirect | it depends on the cloud provider the node is spawned on | 23:05 |
| gmmaha | aaah.. | 23:05 |
| osh-chatbot | <renmak> yeah k i will test it probably over the weekend with 3GB disks and latest ceph chart | 23:06 |
| portdirect | we only need 1gb for a smoke/gate test renmac | 23:06 |
| portdirect | *renmak | 23:06 |
| portdirect | actually - lets make it 5 | 23:06 |
| portdirect | but 1tb is way too much :) | 23:07 |
| *** seaneagan has quit IRC | 23:07 | |
| * gmmaha seconds portdirect on the size of disk. | 23:07 | |
| gmmaha | 5G for osd, 5G for journal snd 2G for swift would be good.. | 23:07 |
| gmmaha | portdirect: was also thnking that making one disk per node should be good and give us the 3 replica that we might need won't it? | 23:08 |
| gmmaha | do we need to run 9 osds for the test cluster? | 23:08 |
| gmmaha | unless the goal is to mainly test multi-osd on each node. | 23:08 |
| gmmaha | then maybe bring it back to 2 osds per node | 23:08 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: move etcd chart to use etcd-operator https://review.openstack.org/545489 | 23:26 |
| * gmmaha thinks scaling loopback to 2 per node and 5G osd, 5G journal and 1G swift drives will be able to fit inside the fedora VM without having to look for the alternate disk, format, mounting and using it | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!