Monday, 2020-11-09

*** sboyron has quit IRC00:08
*** tosky has quit IRC00:41
*** DSpider has quit IRC01:59
ianw#status log collecting netconsole logs from regionone.linaro-us mirror on temp host @
openstackstatusianw: finished logging03:12
ianwinfra-root: ^ should be able to login there.  hopefully see if we can capture something if this shuts itself off03:13
*** mtreinish has quit IRC03:15
*** slaweq has quit IRC03:40
*** slaweq has joined #opendev03:42
*** amotoki has quit IRC03:56
*** amotoki has joined #opendev03:57
*** ykarel has joined #opendev04:46
openstackgerritIan Wienand proposed opendev/system-config master: Add all backup hosts to borg backups
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove entry-point for element-info
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove dib-block-device console entrypoint
openstackgerritIan Wienand proposed opendev/system-config master: Add all backup hosts to borg backups
*** ykarel_ has joined #opendev06:27
*** ykarel has quit IRC06:30
*** marios has joined #opendev06:30
*** ysandeep|away is now known as ysandeep06:39
*** ysandeep is now known as ysandeep|ruck06:39
*** sboyron has joined #opendev06:41
*** whoami-rajat__ has joined #opendev06:50
*** lpetrut has joined #opendev07:11
*** hashar has joined #opendev07:50
*** iurygregory has quit IRC07:58
*** fressi has joined #opendev08:01
*** andrewbonney has joined #opendev08:10
openstackgerritMerged openstack/project-config master: Run publish-openstack-artifacts on ubuntu-focal
*** ysandeep|ruck is now known as ysandeep|lunch08:21
*** rpittau|afk is now known as rpittau08:30
*** iurygregory has joined #opendev08:30
*** ykarel_ is now known as ykarel08:37
*** ysandeep|lunch is now known as ysandeep|ruck09:15
*** sshnaidm_ is now known as sshnaidm|rover09:24
*** ralonsoh has joined #opendev09:33
*** iurygregory has quit IRC09:35
*** iurygregory has joined #opendev09:36
*** fressi has quit IRC09:40
*** fressi has joined #opendev09:47
openstackgerritMerged zuul/zuul-jobs master: Decrease MTU to account for IPv6 header
*** fressi has quit IRC09:53
*** fressi has joined #opendev09:57
openstackgerritzbr proposed opendev/elastic-recheck master: Make file writing atomic
*** hashar has quit IRC10:31
*** iurygregory_ has joined #opendev10:32
*** iurygregory has quit IRC10:35
openstackgerritThierry Carrez proposed opendev/irc-meetings master: Update Large Scale SIG meeting time
openstackgerritMerged opendev/elastic-recheck master: Make file writing atomic
*** tosky has joined #opendev11:09
*** mtreinish has joined #opendev11:14
*** DSpider has joined #opendev11:35
*** dtantsur|afk is now known as dtantsur12:32
*** iurygregory_ is now known as iurygregory13:01
*** hashar has joined #opendev13:07
*** priteau has joined #opendev13:17
*** d34dh0r53 has quit IRC13:46
*** d34dh0r53 has joined #opendev13:52
*** ykarel has quit IRC14:08
*** TheJulia has joined #opendev14:09
*** cloudnull has quit IRC14:11
*** ykarel has joined #opendev14:18
*** cloudnull has joined #opendev14:18
openstackgerritzbr proposed zuul/zuul-jobs master: More E208 (22)
openstackgerritzbr proposed zuul/zuul-jobs master: More E208 (final)
openstackgerritzbr proposed openstack/project-config master: Add pytest-dev/pytest-infra project
openstackgerritzbr proposed opendev/system-config master: Account for testinfra project rename
*** lpetrut has quit IRC15:02
*** ysandeep|ruck is now known as ysandeep|away15:25
*** hashar has quit IRC15:28
mordredTIL: in bash v4 - shopt -s globstar15:38
mordredls **/*.conf15:38
mordredtotally a thing15:38
mordredand it DTRT with paths containing spaces15:38
*** iurygregory has quit IRC15:43
openstackgerritMerged zuul/zuul-jobs master: More E208 (22)
fungirecursive globs here i come15:52
*** rpittau is now known as rpittau|bbl15:52
mordredright? it's not every day I hear about a thing in bash and immediately get excited about cleaner shell scripts15:53
*** hashar has joined #opendev15:54
*** roman_g has joined #opendev15:54
clarkbfwiw gerrit 3.3 won't release until the monday prior to our planned upgrade. For this reason I think we should stick with 3.2 as the upgrade target15:55
*** cloudnull has quit IRC15:55
clarkbgetting to 3.3 shouldn't take long once we are at 3.2 if the 3.0 -> 3.1 -> 3.2 upgrades are any indication so I'm not too worried about it15:55
mordreddoing a followup 3.3 ... yeah15:55
mordredshould be easy15:55
openstackgerritzbr proposed openstack/project-config master: Add pytest-dev/pytest-testinfra
*** cloudnull has joined #opendev16:02
*** iurygregory has joined #opendev16:03
*** mlavalle has joined #opendev16:06
*** jgwentworth is now known as melwitt16:12
openstackgerritzbr proposed openstack/project-config master: Rename testinfra project
roman_gGood morning, team. We are observing increased number of NODE_FAILURE error on attempts to launch jobs utilizing 16GB and 32GB VMs. Could you, please, have a look at Zuul logs and confirm that it's just capacity issue and VMs couldn't be scheduled due to lack of resources? Thank you.16:28
roman_gWe are primarily concerned of 16GB jobs, as 32GB are only temporary ones (not merged and probably would not be merged).16:29
clarkbroman_g: we have to turn off the openedge cloud due to a dying router and I expect most of your successful launches were against that cloud16:29
clarkbroman_g: now you're only launching against teh citycloud provider and that one has always had a higher fail rate16:29
roman_gclarkb, But openedge was down for quite a while, no?16:30
clarkbit was down for a while then turned back on for a while and now off again16:30
donnydYea my edge router is dying16:31
roman_gclarkb all right. Thank you.16:31
clarkbI'm 99% certain this is going to be citynetwork issues we've seen before but I've not checked the logs16:31
roman_gdonnyd thank you for your services!16:31
*** diablo_rojo_phon has joined #opendev16:32
donnydIf i can find a new edge machine, OE will get turned back on, but short of one falling from the sky...16:32
roman_gdonnyd I wish I could help.16:36
*** slaweq has quit IRC16:37
*** slaweq has joined #opendev16:40
clarkbmelwitt: should be the first file indexed by your udpated log processor script16:43
clarkbmordred: is second16:44
clarkber melwitt ^16:44
clarkbmelwitt: I think the next step for us is to check that those contents ended up in elasticsaerch as expected and if so we can fix the python requests package install on your change then land the change and get it out everywhere16:44
clarkbI can help with the package install side since that is puppetry and I'm not sure how interested in figuring that out you are16:44
melwittclarkb: thanks! makes sense. +1 to your review comment to hold off on updates until we see how the test run does. I'm up for trying if you point me to an example or something, if that would be helpful for me to try it16:46
clarkbmelwitt: is another16:46
clarkbmelwitt: oh I've just yolo'd it on production since we have 80 of these processes running16:46
*** marios is now known as marios|out16:47
clarkbmelwitt: what you can do though is check that those three log files show up in logstash for you16:47
clarkbfor those specific job runs16:47
melwitt:) ok, can do. thanks16:47
clarkbmelwitt: and another. That is probably a big enough sample now where if those are happy we can roll ahead16:49
*** marios|out has quit IRC16:50
*** hashar has quit IRC17:00
*** ykarel has quit IRC17:04
clarkbmelwitt: build_short_uuid:"a6fd710" AND filename:"controller/logs/screen-keystone.txt" I think that shows the keystone example17:08
clarkbdon't forget to expand the default time range, but can you confirm that you think the data is getting indexed?17:08
melwittyeah, I've been doing similar. I find: build_change:752006 AND filename:"controller/logs/screen-etcd.txt" AND build_short_uuid:dc896cf has index 40 lines and the file has 40 lines, so that's a good sign17:09
*** hamalq has joined #opendev17:11
*** hashar has joined #opendev17:12
*** hashar has quit IRC17:12
*** hamalq has quit IRC17:14
*** hamalq has joined #opendev17:15
melwittfor the second sample, I'm not seeing all of the beginning lines in logstash when I sort by ascending timestamp filename:"controller/logs/screen-neutron-api.txt" AND build_short_uuid:cd4444117:15
clarkbmelwitt: it should drop all the DEBUG lines, could that be it?17:16
clarkbor maybe if they don't have a timestamp prefix they are getting associated to a prior log line/file17:16
clarkb(fwiw I don't think issues like that will be due to your change, if we get any lines I expect we're good as far as not regressing via your change)17:16
melwittI see. yeah not yet sure what I'm seeing17:17
clarkbI'll work on updating the puppet and removing the StringIO line in a bit if you'd prefer to not learn puppet :)17:18
melwittsure, if you don't mind :) I'm not opposed to learning puppet, I didn't want to create more work for you, so if it would help for me to do it, I'm willing to try17:21
melwittdo you know if lines like this "Nov 09 16:25:14.440258 ubuntu-bionic-limestone-regionone-0021671450 systemd[1]: Started Devstack devstack@neutron-api.service." get categorized as DEBUG?17:22
*** rpittau|bbl is now known as rpittau17:22
melwittI see that file missing in logstash but like you said, maybe unlikely to be a regression17:23
clarkbmelwitt: I'm not sure what it does if there is no level. but I wouldn't be surprised if that is the underlying issue17:23
openstackgerritClark Boylan proposed opendev/puppet-log_processor master: Stream log files instead of loading full files into memory
clarkbmelwitt: ^ I think that will do it17:24
melwittthat same line shows up in the first example file, but not in the second one17:24
melwitta-ha cool thanks!17:24
clarkbthey use different logstash rules I think17:25
clarkbsince etcd isn't an openstack service17:25
clarkbI think we expect log level in openstack service logs17:25
clarkbbut we're probably more relaxed with more generic log files17:25
clarkbfungi: if you have time to rereview again that would be great. Its being tested on logstash-worker01 and other than needing to install python-requests it seems happy17:26
clarkbmelwitt: and thank you for having patience with me while I found time to finish this up17:26
melwittnp at all, thank you for testing it out. I definitely wanted to do everything we could to help ensure it wouldn't break anything, so I appreciate it17:29
*** ralonsoh has quit IRC17:34
melwittI found that line btw in the neutron file in logstash with filename:"controller/logs/screen-neutron-api.txt" AND build_short_uuid:cd44441 AND message:"Started Devstack"17:34
melwittso there must be something funny with the timestamp sorting on the kibana dashboard I guess17:35
fungiclarkb: melwitt: 759492 lgtm17:36
melwittyeah I can see the "@timestamp" field isn't in the same order as the lines appear in the file, example: 2020-11-09T08:43:24.534-08:00    Nov 09 16:25:14.440258 ubuntu-bionic-limes17:41
melwittis the "Started Devstack" line17:42
*** hjensas__ has joined #opendev17:43
clarkbif we can't parse the timestamp in the input we use the current parse time timestamp17:43
clarkbwhich may explain it17:43
melwittwhereas this line "2020-11-09T08:25:16.610-08:00[-] Logging enabled!" appears later in the file but got "@timestamp" earlier in time17:43
*** hjensas_ has quit IRC17:47
openstackgerritClark Boylan proposed opendev/puppet-log_processor master: Stream log files instead of loading full files into memory
clarkbfungi: ^ I had to move the requests install from worker.pp to init.pp because worker.pp is a define and called multiple times for each worker daemon install and we end up with duplicate package conflicts17:49
clarkbthe init.pp is a class and run once so should be fine there17:49
fungiyep, good idea17:50
fungicleaner anyway17:50
fungiconflict avoidance with ifdef would have been slightly messier i guess17:51
*** andrewbonney has quit IRC18:09
*** roman_g has quit IRC18:20
*** dtantsur is now known as dtantsur|afk18:24
*** roman_g has joined #opendev18:28
openstackgerritMerged openstack/project-config master: Add pytest-dev/pytest-testinfra
fungittx: i've requested some clarification on 761751 just to make sure expectations are correctly set18:35
clarkbI've removed logstash-worker01 from the emergency hosts file since melwitt's change looks like it will merge soon18:35
clarkbonce it has merged the daemons won't auto restart so I'll likely go through and reboot them all as its a good opportunity for that18:36
fungithat's ought to be a huge stability improvement18:36
*** mlavalle has quit IRC18:43
*** mlavalle has joined #opendev18:44
*** rpittau is now known as rpittau|afk18:45
openstackgerritMerged opendev/puppet-log_processor master: Stream log files instead of loading full files into memory
clarkbI need to figure out lunch but then I'll check ^ has applied then restart as necessary18:57
*** whoami-rajat__ has quit IRC19:00
*** roman_g has quit IRC19:09
*** larainema has quit IRC19:12
*** hashar has joined #opendev19:41
*** tosky has quit IRC20:17
yoctozeptodear infra, may I get chanserv auth to be able to op myself on #openstack-masakari?20:20
fungiyoctozepto: sure, just a sec, my wifi is acting a little flaky out here on the deck20:26
yoctozeptofungi: sure, take your time, it's not a priority :-)20:27
*** tosky has joined #opendev20:28
clarkbI'm restarting logstash workers to pick up melwitts change (puppet has applied but they dont' restart services)20:28
fungi#status log added founder access to #openstack-masakari for ptl yoctozepto20:29
openstackstatusfungi: finished logging20:29
fungiyoctozepto: looks like samP was the only person outside opendev irc volunteers with access to control that channel (seems to have been the one to initially register it)20:30
yoctozeptofungi: I see, thanks20:31
ianwfungi: i think the new mirror-update reprepro is working ok, if you have time to double check the removal @ i can cleanup the old stuff20:35
ianwclarkb: and adds all host into borg-backup; i think we're ready to go with that, ethercalc has been working good20:36
clarkbianw: ah cool I'll take a look20:37
clarkbmelwitt: all 80 logstash workers are now running with your update. Thanks! let us know if you notice anything off with that20:38
*** whoami-rajat__ has joined #opendev20:38
melwittclarkb: sweet, will do, I'll keep an eye on it20:38
clarkbs/workers/worker daemons/20:39
openstackgerritMerged opendev/system-config master: Remove mirror-update server and related puppet
*** mgoddard has quit IRC21:25
clarkbinfra-root I've updated the meeting agenda for tomorrow on the wiki. Is there anything else that we should add before sending it?21:28
fungii can't think of anything21:31
fungi(at all, not really agenda-related, i'm basically just braindead at this point)21:32
openstackgerritMerged openstack/project-config master: Rename testinfra project
clarkbianw: lookingat the borg change, it occurs to me that we probably want to exclude afs too21:43
clarkbianw: or wait do we do that already because / isn't in the list?21:44
clarkbif that is the case do we need to worry about other things like /opt being missed?21:44
clarkbmaybe we should backup / then be more aggressive on the excludes?21:44
clarkbI think that is how bup did it21:44
*** hashar has quit IRC21:46
*** sboyron has quit IRC21:46
ianwclarkb: yeah; i did more to more a include process.  do we keep data on opt?21:52
clarkboff the top of my head we don't, but we do use opt for random things so we may need to audit that on the backed up hosts?21:53
fungimy recollection is we put important data in /home or /srv or /var/lib in some cases, but stuff in /opt is generally ephemeral?21:54
ianwask lists storyboard translate it seems no21:59
ianwgitea review-dev review also all code22:00
fungiwe could stand to settle on a more formal policy for where we keep important stateful data, but it seems like we've been doing a pretty good job at being consistent so far22:01
ianwetherpad has an /opt/db but i think that might be a manual thing?22:01
ianwyeah, it looks like it was a dump made by mordred in april22:02
ianw /dev/xvda1                        39G   39G     0 100% /22:02
ianwhowever, that is not good22:02
fungilooks like it ran out of space while rotating db backups?22:04
clarkbthere were old stale backups on there that I was supposed to clear out then the world melted down22:04
clarkb/var/backups/etherpad-mariadb/etherpad-mariadb.sql.gz.2.gz can be deleted since it is the old dobule gzipped file22:05
clarkbI'm rm'ing it now22:05
clarkbthat gets us to 2.7GB free22:06
clarkbwhich is about how large a db backup is, maybe we should reduce retention to less than 7 days locally22:06
clarkband rely on offsite backups more22:06
ianwthat seems big22:06
clarkbreview and zuul also had large .bups22:07
clarkbI think we decided that you can safely rm it and bup rebuilds?22:07
ianwi guess it's still going because the db volume has enough space22:08
*** stevebaker has joined #opendev22:12
ianwseems like it's still not going to have enough space to dump22:12
clarkbianw: I think the two things we can do are to clean out the .bup dir, them manually rerun bup to rebuild it and reduce logrotate retention from 7 days to say 2 for the local db backups22:16
clarkbwe've also got a etherpad/etherpad image we can delete22:16
clarkbthat image isn't very large though so thats a small win22:17
ianwok, i'll move .bup to /opt and re-run to make sure it work22:17
ianwok, it's re-buping22:20
openstackgerritIan Wienand proposed opendev/system-config master: etherpad: reduce backup rotations
*** slaweq has quit IRC22:39
ianwhrm, bup failed ->
ianwwith an out of disk error22:42
ianwi'm going to move a bunch of the dump backups to /opt22:46
openstackgerritMerged openstack/diskimage-builder master: yum-minimal: Add centos-stream-repos package for centos-8-stream
clarkbI've run `sudo docker image rm etherpad/etherpad:1.8.0` for completeness on etherpad22:58
clarkbianw: ^ fyi22:59
*** fressi has quit IRC22:59
ianwok, i'm trying this bup backup again with more free space on /23:00
ianw39G   29G   11G23:01
ianwi'll see if it goes down23:01
clarkbit was 39G   25G   15G a couple minutes before your paste23:01
ianwit's now at 9gb23:02
ianwso ... yeah :/23:02
clarkbiirc the .bup index is tracking all the files23:02
clarkbthat is why we excluded the job runtime files on zuul beacuse t here are many of them23:02
clarkbI wonder if we've got another place with many files we might want to exclude on etherpad23:02
clarkbperhaps the mariadb content?23:02
ianwdown to 6.5gb now23:02
ianwi wonder if we should just get this on borg and not spend too much time debugging this23:03
clarkbya that may be the best path forward at this point23:03
ianwthe only really important thing here is the .sql dumps23:03
ianwyeah, it's down to 2g free space, it's going to run out23:04
*** iurygregory has quit IRC23:04
ianwi'll just roll-out 761855 and watch it23:04
clarkbnow it says 25GB free ?23:04
fungii'm in favor of pushing ahead with etherpad's assimilation23:05
clarkbianw: did you stop the bup run? or did it resolve that on its own?23:05
fungiit'll happen eventually anyway (resistance is futile)23:05
clarkbmaybe it was using /tmp for spooling?23:05
ianwyeah, it just died23:05
ianwok, will let the borg rollout apply then run that manually so we know we've got some good offsite backups of it23:07
clarkbianw: I wonder if we put /var/etherpad/db/* in the bup excludes if it would be happier23:08
clarkbpossible we want to add ^ to be borg excludes?23:08
clarkbbut ya  Ithink we can focus on borg now23:08
ianwyes, we probably do want that actually, as we just want to store the daily dumps i guess23:08
ianwno point storing an in-flight db23:08
clarkbyup and the inflicht db probably isn't something we can recover from23:09
clarkboh also for review we mounted the old snapshot23:09
clarkbI wonder if we back that up :/23:09
*** lourot has quit IRC23:09
clarkbfungi: ^ should we go ahead and unmount and detach that volume now?23:10
* fungi checks23:10
fungii've umounted /mnt/2020-20-01_snapshot just now23:10
clarkb/mnt/2020-20-01_snapshot is where we mounted that and /mnt/* is in our excludes23:10
fungiyeah, i was about to surmise we might already exclude /mnt for precisely this reason23:11
clarkbfungi: thanks, we should probably proceed with detaching the volume too? what is the process for that again? we have to unload it form lvm somehow?23:11
openstackgerritIan Wienand proposed opendev/system-config master: etherpad: ignore live db for borg backups
openstackgerritIan Wienand proposed opendev/system-config master: etherpad: ignore live db for borg backups
clarkbthere are two "main"s in vgs23:12
clarkbwhich seems like something we don't want to get wrong :/23:12
fungii just ran `sudo lvchange -an main/test-gerrit`23:12
fungilvs no longer shows the active flag for it23:12
fungiunfortunately it's in the same vg as the production volume23:13
ianwoh, there was also that extra volume i added23:13
fungihopefully cinder/nova will let us detach it23:13
ianwthat was no lvm'd, just for scratch space.  once you've cleaned up that i'll remove that one oto23:13
clarkbfungi: I don't think it is in the same vg? they are listed separately23:14
*** iurygregory has joined #opendev23:16
clarkbfungi: `sudo lvs -o lv_name,lv_uuid,vguuid` should show the differentiation I think23:16
clarkbLu4enc-MUnC-ToPA-z7zc-ZrMJ-GP6g-QAIhpz is the vg to disable based on that23:16
fungioh neat23:16
fungii just looked at the names and assumed it merges them23:17
fungiVolume group "Lu4enc-MUnC-ToPA-z7zc-ZrMJ-GP6g-QAIhpz" not found23:17
fungidoesn't look from the manpage like vgchange has a way to take a vg uuid23:18
clarkbfungi: internet says we can rename the vg with uuid23:18
clarkbthen use vgchange on the new name23:18
fungioh, yeah that should work23:18
clarkbthat is nice because we can confirm we got the right one with vgs prior to stopping too23:19
fungisudo vgrename Lu4enc-MUnC-ToPA-z7zc-ZrMJ-GP6g-QAIhpz notmain23:19
fungi  Volume group "main" successfully renamed to "notmain"23:19
fungithat works23:19
fungi0 logical volume(s) in volume group "notmain" now active23:19
fungiokay, should really be ready for detachment now i guess23:20
clarkbfungi: does vgs not show that it is inactive?23:21
fungithere's no active flag for volume groups, just volumes23:21
clarkbfungi: are you working on detaching it next or should I do that?23:22
clarkb(and are we ready to try that?)23:23
fungi`vgchange -a n` deactivates all volumes in a vg, rather than `lvchange -a n foo/bar` which deactivates volume foo in vg bar23:23
fungiso the `vgchange -a n notmain` was just a no-op anyway23:23
fungiand yeah, we can work on detaching now, i'll give it a go23:24
clarkbfungi: I see, but we ran the vgchange -a n notmain?23:24
fungiyeah, sorry, `vgchange -a n foo` deactivates all volumes in vg "foo"23:25
fungii had already deactivated the only lv in that vg, so deactivating the whole vg was a no-op23:25
fungionly logical volumes are active or inactive, not volume groups23:26
clarkbgot it23:26
fungialso i checked and the pv for the notmain vg is /dev/xvdd123:26
clarkbya and that seems to line up with what cinder is saying for what I believe is the volume23:27
clarkbxcdc is the prod one which lines up with vgs23:28
fungiargh, can't use osc for looking at the volume list in rax23:28
clarkbohya use my osc23:28
fungii found my outdated venv and it's working23:29
clarkbheh don't know where the 4 came from23:29
*** tosky has quit IRC23:33
fungiokay, it detached and now shows as available in volume list23:34
fungithere's another similar available volume i suspect is the one ianw mentioned creating23:34
clarkbno that other one is from review-test's /home/gerrit2 that we replaced with the old snapshot23:35
ianwfungi: umm, the volume i created for the backup restore should't be available, that's mounted at /backup23:35
clarkbya its the one we moved aside for review-test in order to start from the old state and build forward again23:35
fungiahh, okay, i missed we had anything mounted there23:35
clarkbwe swapped it out with the snapshot23:35
clarkbI figure we'll clean up all the review-test stuff post upgrade. maybe after testing a 3.3 upgrade23:36
fungioh, i see, /backup is a raw blockdev not a pv23:36
fungishall i unmount it too?23:36
clarkbno I think ianw is still curating content on it23:36
fungioh okay23:36
clarkbvgs looks good to me now23:37
clarkbthanks for helping with that. My lvm foo is lacking23:37
ianwyeah, let's get what we need off it onto the main disk then we can remove23:37
openstackgerritMerged opendev/system-config master: Add all backup hosts to borg backups
fungiokay, i've deleted volume 765225a9-ab7c-40a3-a0ef-01f33bf498a8 now, the one we created from the snapshot23:40
*** lourot has joined #opendev23:43

Generated by 2.17.2 by Marius Gedminas - find it at!