Monday, 2020-08-03

*** ryohayakawa has joined #openstack00:01
*** born2bake has quit IRC00:06
*** cyberworm54 has quit IRC00:23
*** cyberworm54 has joined #openstack00:24
*** markvoelker has joined #openstack00:29
*** markvoelker has quit IRC00:33
*** Arsenick has quit IRC00:44
*** Arsenick has joined #openstack00:48
*** psahoo has joined #openstack01:12
*** markvoelker has joined #openstack01:12
*** markvoelker has quit IRC01:17
*** benfelin has quit IRC01:22
*** skyraven has joined #openstack01:25
*** mvorwerk has joined #openstack01:29
*** mvorwerk has quit IRC01:34
*** skyraven has quit IRC01:35
*** arnoldoree has quit IRC01:37
*** arnoldoree has joined #openstack01:39
*** mvorwerk has joined #openstack01:43
*** mvorwerk has quit IRC01:47
*** mvorwerk has joined #openstack02:00
*** mvorwerk has quit IRC02:05
elico*: I want to run OpenStack on a single node AIO. Is there any specific thing to know about such a setup?02:07
*** markvoelker has joined #openstack02:28
*** jistr has quit IRC02:28
*** jistr has joined #openstack02:28
*** Arsenick has quit IRC02:30
*** Arsenick has joined #openstack02:30
*** markvoelker has quit IRC02:32
*** ueha has joined #openstack02:44
*** ewalt has joined #openstack03:08
*** slaweq_ has joined #openstack03:22
*** slaweq has quit IRC03:24
*** waxfire7 has joined #openstack03:26
*** waxfire has quit IRC03:26
*** waxfire7 is now known as waxfire03:26
*** maharg101 has quit IRC03:28
*** maharg101 has joined #openstack03:29
*** drznpy0 has joined #openstack03:29
*** drznpy has quit IRC03:30
*** drznpy0 is now known as drznpy03:30
*** skyraven has joined #openstack03:32
*** skyraven has quit IRC03:37
*** markvoelker has joined #openstack04:28
*** rcernin has quit IRC04:31
*** markvoelker has quit IRC04:33
*** ewalt has quit IRC04:36
*** ewalt has joined #openstack04:36
*** gmoro has quit IRC04:37
*** Goneri has quit IRC04:44
*** Warped has quit IRC04:50
*** markvoelker has joined #openstack04:52
*** rcernin has joined #openstack04:52
*** markvoelker has quit IRC04:56
*** ewalt has quit IRC05:05
*** ircuser-1 has quit IRC05:35
*** skyraven has joined #openstack05:39
*** links has joined #openstack05:44
*** ymasson has quit IRC06:00
*** miloa has joined #openstack06:11
*** ryo_hayakawa has joined #openstack06:12
*** avivgt has joined #openstack06:13
*** ryohayakawa has quit IRC06:14
*** mvorwerk has joined #openstack06:28
*** rhayakaw__ has joined #openstack06:32
*** mvorwerk has quit IRC06:32
*** mvorwerk has joined #openstack06:33
*** ryo_hayakawa has quit IRC06:34
*** malevolent has quit IRC06:51
*** markvoelker has joined #openstack06:53
*** jcapitao has joined #openstack06:58
*** markvoelker has quit IRC06:58
*** nightmare_unreal has joined #openstack07:03
*** bengates has joined #openstack07:08
*** slaweq_ has quit IRC07:13
*** slaweq has joined #openstack07:13
*** ntt__ has joined #openstack07:16
*** tesseract has joined #openstack07:16
*** suuuper has joined #openstack07:18
*** Lucas_Gray has joined #openstack07:19
*** slaweq has quit IRC07:21
*** slaweq has joined #openstack07:22
*** slaweq has quit IRC07:22
*** slaweq has joined #openstack07:22
*** frickler_pto is now known as frickler07:26
*** yaawang has quit IRC07:32
*** yaawang has joined #openstack07:32
*** shyamb has joined #openstack07:38
*** sajoupa has quit IRC07:47
*** sajoupa has joined #openstack07:47
*** sshnaidm|afk is now known as sshnaidm07:58
*** ttx has quit IRC08:00
*** ttx has joined #openstack08:01
*** yaawang has quit IRC08:02
*** yaawang has joined #openstack08:03
*** mlycka has joined #openstack08:08
*** pkopec has joined #openstack08:17
*** rgogunskiy has quit IRC08:22
*** imega has joined #openstack08:28
*** shyamb has quit IRC08:32
*** malevolent has joined #openstack08:34
*** Lucas_Gray has quit IRC08:35
*** bolg has joined #openstack08:36
*** ntt has joined #openstack08:36
*** ntt__ has quit IRC08:37
*** geaaru has joined #openstack08:46
*** bengates has quit IRC08:47
*** Lucas_Gray has joined #openstack08:48
*** shyamb has joined #openstack08:49
*** markvoelker has joined #openstack08:55
*** markvoelker has quit IRC08:59
*** shyam89 has joined #openstack09:07
*** pcaruana has joined #openstack09:09
admin0elico, nothing .. just follow the relevant document ( from openstack-ansible or kolla-ansible)09:09
*** shyamb has quit IRC09:11
*** jangutter has joined #openstack09:11
*** Lucas_Gray has quit IRC09:13
*** jangutter_ has quit IRC09:14
recycleheroguys. I suspected a failed hdd on my openstack controller and ddrescued it with 100% success.09:15
recycleheronow when I swaped the hdd with the cloned one I get the same errors09:16
recycleheroeverything is failing09:16
recycleherobut I guess mariadb is the main reason09:16
*** jcap has joined #openstack09:16
recycleherowhat should I do? where should I inspect09:16
*** jcap has quit IRC09:16
*** jcapitao has quit IRC09:17
*** jcapitao has joined #openstack09:17
admin0you had a single disk or multiple controllers ?09:18
recycleheroadmin0: single disk09:23
recycleherothe hdd is ok now I guess I dd rescued it09:23
admin0and there was no daily or hourly mariadb backup ?09:23
recycleheronone09:24
admin0if the database is not corrupt, then you might be able to get back by doing a reinstall09:24
recycleheroI think the problem was mayb a sudden shutdown and its side effects on mariadb09:24
recycleheroshould I go to mariadb channel?09:25
recycleherowhy a reinstall?09:25
recycleheroreinstall of the controller?09:25
*** Lucas_Gray has joined #openstack09:25
recycleheroAug 03 13:57:45 controller mysql-check-socket[13104]: Socket file /var/lib/mysql/mysql.sock exists.09:25
recycleheroAug 03 13:57:45 controller mysql-check-socket[13104]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, so it will be removed automatically.09:26
admin0except database, everything in openstack can be recovered09:26
recycleheroAug 03 13:57:45 controller mysql-prepare-db-dir[13128]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.09:26
recycleheroAug 03 13:57:45 controller mysql-prepare-db-dir[13128]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.09:26
recycleheroAug 03 13:57:45 controller mysqld[13165]: 2020-08-03 13:57:45 139990674802880 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as process 13165 ...09:26
admin0please don't paste here09:26
admin0mariadb or mysql channel will be best09:26
recycleheroadmin0: everything except mariadb is in place I guess09:28
recycleherowhats the consequence of losing the database?09:28
admin0the instances, cinder volumes will be zombies09:29
admin0your mapping from a user -> instance is gone isn't it ?09:29
*** yaawang has quit IRC09:29
recycleheroQ for later! If I have my needed instances running and dont need horizion ar any change to state can I power off the controller?09:30
recycleheroadmin0: dont know09:30
recycleherotring mariadb channel09:30
admin0with controllers totally down, the instances will still work . but only till they are running09:31
admin0if they are rebooted for any reason, they might not get thier old ip address09:31
admin0it depends on how affected your controller is and where your network agents live09:31
*** yaawang has joined #openstack09:32
*** stingrayza has joined #openstack09:34
recycleheroadmin0: got it09:35
recycleherothe machine is still unstable and I dont know why09:36
recycleherocan u guide me what should I get out to have a resintall09:36
admin0how was it installed ?09:37
*** also_stingrayza has quit IRC09:37
recycleheroit was a while ago but its openstack queens and manual install on the machine.09:37
admin0first thing you have to do is make sure if you have your database back or not09:38
recycleherook I will do that.09:38
*** jangutter has quit IRC09:45
*** jangutter has joined #openstack09:46
*** bengates has joined #openstack09:47
*** shyam89 has quit IRC09:48
*** benfelin has joined #openstack09:57
*** shyamb has joined #openstack09:59
*** miloa has quit IRC10:04
*** miloa has joined #openstack10:05
*** rcernin has quit IRC10:13
*** tkajinam has quit IRC10:13
*** rcernin has joined #openstack10:14
*** Relih_ has joined #openstack10:25
*** shyamb has quit IRC10:28
*** markvoelker has joined #openstack10:32
*** markvoelker has quit IRC10:39
*** shyamb has joined #openstack10:51
*** shyam89 has joined #openstack11:00
*** rcernin has quit IRC11:02
*** shyamb has quit IRC11:03
*** wpp has quit IRC11:04
*** rhayakaw__ has quit IRC11:05
*** wpp has joined #openstack11:09
*** jangutter has quit IRC11:11
*** jangutter has joined #openstack11:12
*** born2bake has joined #openstack11:12
*** jv_ has joined #openstack11:13
*** Lucas_Gray has quit IRC11:19
*** mlycka has quit IRC11:27
*** Lucas_Gray has joined #openstack11:31
*** gameon has joined #openstack11:38
*** psahoo has quit IRC11:45
*** mlycka has joined #openstack11:49
*** makije has joined #openstack11:51
recycleheroadmin0: I think there is a bug in mariadb/mysql causing my problem.11:55
admin0until  you fix and get your data back, your recovery will look like a new greenfield deployment11:55
recycleherocan I upgrade my centos controller host with no problems causing to opeanstack ?11:55
recycleheroi didnt update since install11:56
admin0without your database, your deployment will look like new ..11:56
admin0so better ssh to the running hosts and take backup11:56
admin0and most imp, set a root pass so that you can login to console and do manual ip assignment in case you want to salvage it later11:56
recycleherosome of the importent ones are not working and non-important one are working!!!11:56
recycleheroso I need to take the controlller back11:57
admin0what is the status on the db ?11:57
admin0it cannot be restored ?11:57
admin0it cannot be used ?11:57
*** psahoo has joined #openstack11:57
recycleherono cannot connect to mysql deamon, not running11:57
recycleherowhat happens if i update with yum11:58
recycleherowhy should I lose my database with yum update?11:58
admin0can you install the same version mysql in a docker/container and then copy the /var/lib/mysql and see if it brings your database to a state where you can take a backup/restore11:58
admin0no, you make a backup of your mysql, put it in a docker/lxc container and then do the yum update there11:58
recycleheronice, admin. I will try that.11:59
admin0unless you have gone through the upgrade codes yourself or written them yourself, can you guarantee it will not make the situation worse11:59
admin0so no, i would not be working on a failed system like this .. i would be taking backup and then trying update/upgrade in a more controlled environment with copied data11:59
recycleheroadmin0: no, i cant. you are right. your approcah is way more safe11:59
admin0are you on shared storage ?12:00
recycleherobut this mariadb/mysql distinction is going to kill me I guess.12:00
admin0like ceph12:00
recycleherono for the controller host its a regular hdd12:00
admin0no no12:01
recycleheroi use nfs + raid for instances12:01
admin0the instance file / nova i meant12:01
*** jcapitao is now known as jcapitao_lunch12:01
admin0so the disks are on /var/lib/nova/instances ?12:01
recycleherono they are on a diffrent host outside of openstack and provided via a nfs share to openstack12:01
admin0ok12:02
admin0they are still disk files right ? and not ceph blobs12:02
*** sbetrium has joined #openstack12:02
recycleherothat a + for my situation, right?12:02
*** sbetrium has quit IRC12:02
admin0yes12:02
recycleherono ceph things12:02
recycleheroi will go try to make a docker file now. thanks!12:03
admin0i use lxd .. find it much easier12:06
recycleheroit was my plan not to jump to the docker hype and use lxd but learning jenkins led me to using docker for now.12:09
recycleheroit cannot scp mysql.sock12:09
recycleheroit is a socket and have suid bit set I guess12:10
*** rcernin has joined #openstack12:11
admin0gitlab is taking over jenkins ..12:12
admin0its taking over a lot of stuff tbh12:12
admin0for good12:12
*** rcernin has quit IRC12:15
*** ueha has quit IRC12:15
recycleheroyou mean my investment on learning jenkins is not portable? I was setting up jenkins in docker on my laptop and had my  git repostories in my opestack when the controller died on me:D12:16
recycleheroswap git+jenkins for gitlab?12:16
*** shyam89 has quit IRC12:20
*** mlycka has quit IRC12:20
*** lyarwood has joined #openstack12:30
*** gameon has quit IRC12:35
*** belmoreira has joined #openstack12:54
*** morazi has quit IRC13:00
*** morazi has joined #openstack13:08
*** ueha has joined #openstack13:13
*** BloqueNegro has quit IRC13:19
*** jcapitao_lunch is now known as jcapitao13:20
*** avolkov has joined #openstack13:20
*** BloqueNegro has joined #openstack13:31
*** rcernin has joined #openstack13:37
*** makije has quit IRC13:41
*** makije has joined #openstack13:44
*** jangutter_ has joined #openstack13:46
*** jangutter has quit IRC13:49
*** haleyb has joined #openstack13:53
*** jangutter has joined #openstack13:54
*** jv_ has quit IRC13:55
*** jangutter_ has quit IRC13:57
*** JamesBenson has joined #openstack14:02
fungigitlab does not replace git, in fact it depends heavily on it14:13
*** jhesketh has quit IRC14:14
*** ntt has quit IRC14:15
*** jhesketh has joined #openstack14:16
recycleheroadmin0: I am tring with mariadb docker image for recovery. my version is 10.1.20. first i tried with mariadb:latest it says it doesnt allow upgrade after crash14:21
recycleherothen with 10.1-bionic image it shows the same error as before14:22
admin0yo will get help from the maria/mysql channel itself14:22
recycleherook14:22
admin0related to mysql and how to recover it14:22
admin0once its recovered, then can help you14:22
*** rcernin has quit IRC14:32
*** matt_kosut has joined #openstack14:35
*** ntt has joined #openstack14:42
DHEI've seen this with mysql. you can recover a crashed innodb image, or you can upgrade a clean shutdown to the newer versions. Not both at the same time. have the old version repair the damage, shut down cleanly, then upgrade.14:44
*** rcernin has joined #openstack14:45
recycleheroadmin0: i got the database back with innodb_force_recovery=414:50
recycleheroi am investigating what that does to the data14:50
*** masterpe has joined #openstack14:52
*** arnoldoree has quit IRC14:55
*** jathan has joined #openstack15:01
*** pkopec has quit IRC15:05
*** mvorwerk has quit IRC15:06
*** Warped has joined #openstack15:06
*** belmoreira has quit IRC15:17
*** ntt has quit IRC15:20
*** ddstreet has quit IRC15:21
*** ddstreet has joined #openstack15:21
*** ewalt has joined #openstack15:22
*** malevolent has quit IRC15:25
*** ewalt has quit IRC15:29
*** miloa has quit IRC15:30
*** jv_ has joined #openstack15:43
*** bengates has quit IRC15:45
*** bengates has joined #openstack15:46
*** npalladium has joined #openstack15:47
*** links has quit IRC15:49
*** q-u-a-n2 has quit IRC15:49
*** zbsarash1 has joined #openstack15:49
*** tesseract has quit IRC15:50
*** bengates has quit IRC15:50
*** pkopec has joined #openstack16:00
*** rcernin has quit IRC16:15
*** hamalq has joined #openstack16:16
*** jv_ has quit IRC16:21
*** jcapitao has quit IRC16:22
*** ymasson has joined #openstack16:31
*** rcernin has joined #openstack16:50
*** sshnaidm has quit IRC16:55
*** rcernin has quit IRC16:59
*** jonaspaulo has joined #openstack17:07
*** ueha has quit IRC17:11
*** markvoelker has joined #openstack17:11
*** b0urne has quit IRC17:12
*** b0urne has joined #openstack17:14
*** __ministry1 has joined #openstack17:16
*** markvoelker has quit IRC17:16
*** __ministry1 has quit IRC17:16
*** psahoo has quit IRC17:25
*** Lucas_Gray has quit IRC17:26
*** nightmare_unreal has quit IRC17:56
*** pkopec has quit IRC18:06
*** sshnaidm has joined #openstack18:14
*** gyee has joined #openstack18:17
*** sdmitriev has joined #openstack18:20
*** avolkov has quit IRC18:20
recycleherocan I shutdown my controller node after I have started my instances?18:26
*** elico has quit IRC18:27
*** Sauvin has quit IRC18:29
*** ewalt has joined #openstack18:38
recycleherois there a way to correct a out of sync linux agent or should I restart the whole infra18:43
*** malevolent has joined #openstack18:53
admin0just restart the proceses18:54
admin0if the db is back and eveything is connected to the db, and rabbitmq is also up, then restarting the process will make them in sync18:55
*** slaweq has quit IRC19:08
*** KeithMnemonic has joined #openstack19:21
*** slaweq has joined #openstack19:24
*** slaweq has quit IRC19:29
*** JamesBen_ has joined #openstack19:38
*** JamesBenson has quit IRC19:41
*** ewalt has quit IRC19:43
*** avolkov has joined #openstack19:43
*** cyberworm54 has quit IRC19:47
*** cyberworm54 has joined #openstack19:48
*** ewalt has joined #openstack19:48
*** marbindrakon has quit IRC20:07
*** marbindrakon has joined #openstack20:16
*** aedc has joined #openstack20:16
*** zbsarash1 has quit IRC20:24
*** slaweq has joined #openstack20:25
*** zbsarash1 has joined #openstack20:26
*** spiette has joined #openstack20:30
*** Lucas_Gray has joined #openstack20:31
*** rcernin has joined #openstack20:56
*** rcernin has quit IRC21:01
*** jonaspaulo has quit IRC21:06
*** mmercer has joined #openstack21:16
*** slaweq has quit IRC21:29
*** maevoureik has joined #openstack21:33
*** ewalt has quit IRC21:36
*** ewp has joined #openstack21:36
*** maevoureik has quit IRC21:39
*** maevoureik has joined #openstack21:39
*** avivgt has quit IRC21:41
*** slaweq has joined #openstack21:41
*** avivgt has joined #openstack21:41
*** ewp has quit IRC21:42
*** slaweq has quit IRC21:45
*** spiette has quit IRC21:53
*** benfelin has quit IRC21:54
*** tkajinam has joined #openstack21:59
*** marquezini has joined #openstack22:00
*** born2bake has quit IRC22:00
*** marquezini has quit IRC22:05
*** skyraven has quit IRC22:06
*** ewp has joined #openstack22:08
*** maevoureik has quit IRC22:11
*** qchris has quit IRC22:22
*** skyraven has joined #openstack22:23
*** ewp has quit IRC22:24
*** jidar has quit IRC22:33
*** jidar has joined #openstack22:33
*** qchris has joined #openstack22:35
*** avivgt has quit IRC22:56
*** pramchan has joined #openstack23:05
*** skyraven has quit IRC23:08
*** avivgt has joined #openstack23:08
*** rcernin has joined #openstack23:11
*** skyraven has joined #openstack23:14
*** Lucas_Gray has quit IRC23:14
*** rcernin has quit IRC23:15
*** rcernin has joined #openstack23:16
*** Lucas_Gray has joined #openstack23:19
*** skyraven has quit IRC23:21
*** avivgt has quit IRC23:29
*** skyraven has joined #openstack23:31
*** skyraven has quit IRC23:36
*** avivgt has joined #openstack23:37
*** avivgt has quit IRC23:45
*** npalladium has quit IRC23:49
*** hamalq has quit IRC23:50

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!