Friday, 2020-10-23

*** DSpider has quit IRC00:27
*** hamalq_ has quit IRC02:45
*** el1x1r has joined #opendev03:19
*** el1x1r has quit IRC03:25
*** d34dh0r53 has quit IRC03:44
*** ysandep|away is now known as ysandep|ruck03:52
*** bhagyashris is now known as bhagyashris|sick05:09
*** fressi has joined #opendev05:16
*** fressi has quit IRC05:39
*** marios has joined #opendev05:46
*** Tengu has quit IRC06:03
*** Tengu has joined #opendev06:05
cgoncalvesianw, kevinz: hey. any news on Linaro being down?06:13
*** rpittau|afk is now known as rpittau06:21
*** mkalcok has joined #opendev06:23
*** ysandep|ruck is now known as ysandep|afk06:37
*** eolivare has joined #opendev06:37
*** slaweq has joined #opendev06:51
*** ralonsoh has joined #opendev07:02
*** andrewbonney has joined #opendev07:13
*** icey has quit IRC07:17
*** icey has joined #opendev07:19
*** sboyron has joined #opendev07:20
*** sboyron has quit IRC07:26
*** ysandep|afk is now known as ysandep|ruck07:28
*** roman_g has joined #opendev07:32
*** tosky has joined #opendev07:36
ysandep|ruck#opendev hey i noticed some jobs are with mirror issues:-07:44
ysandep|ruck2020-10-23 07:36:26.814749 | primary | Errors during downloading metadata for repository 'AppStream':07:44
ysandep|ruck2020-10-23 07:36:26.814847 | primary |   - Status code: 403 for (IP: 2607:ff68:100:54:f816:3eff:feb5:4635)07:44
ysandep|ruck2020-10-23 07:36:26.814939 | primary | Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried07:44
ysandep|ruckfungi, clarkb ^^ fyi ..07:48
*** sboyron has joined #opendev07:49
*** sboyron has quit IRC07:51
*** sboyron has joined #opendev07:52
*** ysandep|ruck is now known as ysandeep|ruck07:59
*** DjeufackZane has joined #opendev08:00
*** ysandeep|ruck is now known as ysandeep|lunch08:10
*** Tengu has quit IRC08:34
*** Tengu has joined #opendev08:45
*** slaweq has quit IRC08:52
*** slaweq has joined #opendev08:52
*** marios has quit IRC08:54
*** sboyron has quit IRC09:04
*** ysandeep|lunch is now known as ysandeep|ruck09:18
*** DjeufackZane has quit IRC09:28
openstackgerritlikui proposed openstack/diskimage-builder master: update tox
*** fressi has joined #opendev09:55
*** marios has joined #opendev09:55
yoctozeptomorning infra09:55
yoctozeptois there a chance to get valid https on ?09:56
*** sboyron has joined #opendev10:18
*** sboyron has quit IRC10:23
*** Eighth_Doctor has quit IRC10:24
*** mordred has quit IRC10:24
*** Eighth_Doctor has joined #opendev10:33
frickleryoctozepto: we can (and should IMO) do this, but I'm not sure we'll be able to finish that before the upcoming PTG10:37
*** ysandeep|ruck is now known as ysandep|ruck|afk10:38
*** Tengu has quit IRC10:40
*** Tengu has joined #opendev10:41
*** mkalcok has quit IRC10:46
*** mordred has joined #opendev10:49
fricklerinfra-root: I need to run to an appointment, but limestone mirror seems broken, we need to either fix it or disable the region for now10:59
*** mkalcok has joined #opendev11:18
*** ysandep|ruck|afk is now known as ysandeep|ruck11:25
yoctozeptofrickler: yeah, it's 2020 so it's obligatory TLS nowadays11:45
ysandeep|ruck#opendev hello guys o/ ,  Intermittently some jobs are failing with retry_limit after mirror issues.11:45
ysandeep|ruckIs this the right channel to this issue?11:45
ysandeep|ruck2020-10-23 07:36:26.814749 | primary | Errors during downloading metadata for repository 'AppStream':11:45
ysandeep|ruck2020-10-23 07:36:26.814847 | primary |   - Status code: 403 for (IP: 2607:ff68:100:54:f816:3eff:feb5:4635)11:46
ysandeep|ruck2020-10-23 07:36:26.814939 | primary | Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried11:46
ysandeep|ruckIs this the right channel to report* this issue?11:46
yoctozeptoysandeep|ruck: the problem is known, yes11:46
ysandeep|ruckyoctozepto, good to know its already known issue, many thanks! , do we have any bug or patch for this issue?11:47
yoctozeptoysandeep|ruck: waiting for infra-root for that11:48
ysandeep|ruckyoctozepto, okay, thanks!11:48
yoctozeptoysandeep|ruck: the mirror will be fixed or disabled11:48
yoctozeptoyoctozepto: you are welcome11:49
*** sboyron has joined #opendev11:51
AJaegerysandeep|ruckyoctozepto, as discussed in #openstack-infra, the mirror has been rebooted. If you encounter problems with new jobs (starting now or later), please report again.11:55
ysandeep|ruckAJaeger, awesome thanks!11:56
AJaegerysandeep|ruck: fungi did all the work ;)11:59
fungiyep, sorry, was just waking up and checked #openstack-infra before here12:04
fungithe console showed hung cpu tasks, i recorded a copy to ~fungi/limestone-mirror.console on bridge.o.o12:05
fungithere you go12:08
fungilogan-: ^ anything unusual going on in the limestone cloud around 09:00 utc yesterday?12:09
fungiaha, more like 08:0012:11
fungiwe have them in syslog in this case, it seems to have been able to log to disk just couldn't communicate over the network12:12
fungithe first stall was logged at Oct 22 08:01:2812:13
*** kleini has left #opendev12:38
*** qchris has quit IRC12:49
*** qchris has joined #opendev12:56
*** fressi has quit IRC13:11
*** DSpider has joined #opendev13:22
*** eolivare_ has joined #opendev13:27
*** diablo_rojo has quit IRC13:28
*** eolivare has quit IRC13:28
*** priteau has quit IRC13:28
openstackgerritzbr proposed zuul/zuul-jobs master: Allow test-setup to perform a connection reset
*** manpreet has joined #opendev13:34
*** d34dh0r53 has joined #opendev14:00
*** rpittau is now known as rpittau|afk14:06
*** ysandeep|ruck is now known as ysandeep|away14:07
*** ricolin has joined #opendev14:14
openstackgerritzbr proposed zuul/zuul-jobs master: Add test_setup_reset_connection setting
openstackgerritzbr proposed zuul/zuul-jobs master: Add test_setup_reset_connection setting
*** sboyron has quit IRC14:22
kevinzcgoncalve,ianw: our colo facilities are being maintained, will recover soon14:22
openstackgerritzbr proposed zuul/zuul-jobs master: Add test_setup_reset_connection setting
*** sboyron has joined #opendev14:37
*** sboyron has quit IRC14:41
clarkbkevinz: thank you for the update14:49
openstackgerritJeremy Stanley proposed openstack/project-config master: Use StoryBoard for sandbox repos
kevinzclarkb: np15:04
*** adsy has joined #opendev15:10
openstackgerritzbr proposed zuul/zuul-jobs master: Add test_setup_reset_connection setting
*** ralonsoh has quit IRC15:20
*** ralonsoh has joined #opendev15:20
*** Tengu has quit IRC15:23
*** mkalcok has quit IRC15:25
*** Tengu has joined #opendev15:25
*** DjeufackZane has joined #opendev15:44
*** dmellado has quit IRC15:46
*** dmellado has joined #opendev15:47
dmsimardbtw seeing what looks like timeouts when trying to submit logstash/subunit jobs
clarkbdmsimard: we've had persistent issues with very large log files clogging up the works. Which causes gearman to backup and eventually fall over15:54
clarkbmelwitt is looking at having the tools automatically discard large log files so that it will be more reliable15:54
dmsimardworks for me, wanted to point it out in case it wasn't a known issue15:55
clarkbthanks for checking15:56
*** adsy has quit IRC16:01
melwittclarkb: I'm working on streaming the data instead of loading entire large file into memory as a first step. and then see how it does. if you think that won't be enough, I can stack another change on top to not send too large files for indexing (I don't yet know which part of this code initiates indexing tho)16:05
clarkbmelwitt: oh no I think thats great too16:06
clarkbbasically do our best to index the large logs and if it still fails figure it out from there16:07
clarkbfungi: I'm trying to remember what the rough plan for jvbs was. I think we said we'd add one more for a total of two jvb hosts?16:11
clarkbwhich would be 60% of previous ptg capacity (and it seemed we had plenty last time?)16:11
fungiright, and we already run a jvb on the primary too yeah?16:12
fungiso we'll end up with three jvb processes in total16:12
clarkbyup so last time we had primary + 4 jvbs which is a total of 5 and this time adding one more would be total of 316:12
fungithat matches my recollection of the discussion from the meeting a couple weeks ago16:12
clarkbshoudl I start booting a jvb02 then? or does someone else want to give it a go? (I'm happy to do it but offering in case others are interested in seeing how meetpad is put together)16:13
openstackgerritzbr proposed zuul/zuul-jobs master: Add test_setup_reset_connection setting
*** eolivare_ has quit IRC16:29
kevinzclark, fungi: Linaro US nodepool recovered16:33
kevinzplease help to check if it works16:33
fungithanks kevinz!!!16:33 is up and running an image build16:33
kevinzOK, np16:34
kevinzSorry for inconvenience16:34
clarkbthe other thing to check for is if jobs can run on that cloud again16:34
fungiseems i can browse
clarkboh ya the mirror that too :)16:35
*** ralonsoh has quit IRC16:36
clarkbok /me doesn't see anyone demanding to boot jvb02 :) I'll do that shortly16:38
*** marios is now known as marios|out16:43
*** hillpd_ has joined #opendev16:50
*** DjeufackZane has quit IRC16:51
*** ianw has quit IRC16:51
*** persia_ has joined #opendev16:51
*** mordred has quit IRC16:57
*** auristor has quit IRC16:57
*** hillpd has quit IRC16:57
*** persia has quit IRC16:57
*** ajya has quit IRC16:57
*** hillpd_ is now known as hillpd16:57
clarkbjvb02 server is booting now16:58
fungiyep, sorry, as if one security problem wasn't enough, the rest of my afternoon is being spent trying to catch up on openstack vmt stuff i put off all week16:58
clarkbno worries, I had intended on doing it was just checking no one else wanted to give it a go first16:59
clarkbthe scale up should be basically automagic at this point though since xmpp is used to coordinate16:59
*** mordred has joined #opendev17:04
*** andrewbonney has quit IRC17:06
*** marios|out has quit IRC17:06
*** mlavalle has joined #opendev17:07
*** hamalq has joined #opendev17:17
*** hamalq_ has joined #opendev17:20
*** hamalq has quit IRC17:23
*** ianw has joined #opendev17:23
openstackgerritmelanie witt proposed opendev/puppet-log_processor master: Stream log files instead of loading full files into memory
openstackgerritClark Boylan proposed opendev/system-config master: Add jvb02 prior to the PTG
clarkbI'm sorting out sshfp records then will push up the dns update too17:35
melwittclarkb: ^ first intelligible pass at streaming log files. note that I did not fully test it in that I hacked main to create a LogRetriever and call _open_log_file_url and loop to _retrieve_log_line manually. I'm not sure how completely zuul will test it. if you think there's a better test I should do, lmk17:39
*** auristor has joined #opendev17:41
clarkbmelwitt: I'm not sure zuul will test it, but what we can do is take one of the worker nodes out of config management and test it that way. Its not like things are functional in that system right now anyway17:41
clarkb(we do have some logstash worker tests but not sure we test the file retrieval there)17:41
melwittcool, I wondered if we could do that. and I hope I don't have some dumb bug in the _handle_event method -_-17:42
melwittif I do, I apologize in advance17:42
fungiwell, there are 20 workers, so if one is buggy it's not the end of the world17:42
fungi20 worker servers i mean... with... 4? worker processes each17:42
openstackgerritClark Boylan proposed opendev/ master: Add jvb02
clarkbfungi: ya 4x2017:43
clarkbwe've leaked ansible playbook processes again due to servers being unreachable and jobs timing out on bridge17:51
clarkbI've rebooted the sad server (logstash worker04) and am cleaning up processes now17:51
clarkbdoes anyone know if we can have a zuul timeout result in less leaking in our cd jobs?17:51
clarkband done17:58
*** roman_g has quit IRC17:59
clarkbinfra-root and are two meetpad scale up changes to prep for next weeks PTG17:59
clarkbreviews welcome17:59
melwittmy change failed the legacy linter but I'm not sure whether it's related. it failed during some package installs and I don't see the words "puppet-log_processor" in it
clarkbmelwitt: its a puppet issue. I'll put a fix under your change18:10
melwittoh ok, thanks18:11
openstackgerritClark Boylan proposed opendev/puppet-log_processor master: Stream log files instead of loading full files into memory
openstackgerritClark Boylan proposed opendev/puppet-log_processor master: Fix puppet linter complaints about :: prefixes
clarkbmelwitt: if you're curious old puppet required the :: prefix to root the namespace scope. Newer puppet made it optional then at some point they decided that even though the old version is still correct it shouldn't be used :/18:14
* clarkb finds lunch18:15
melwittclarkb: a-ha, now the log file makes sense to me. thanks!18:17
fungipip 20.2.4 was just released, 20.3 will turn on the dep solver probably next week18:37
fungithey're saying wednesday or thursday is likely for that18:37
funginow the beaker-rspec job doesn't like 75949218:42
fungiGem::RemoteFetcher::UnknownHostError: timed out18:50
fungilooks like a connectivity problem, or maybe is having trouble18:51
fungii'll recheck it18:51
clarkbfungi: thanks19:00
openstackgerritMerged opendev/puppet-log_processor master: Fix puppet linter complaints about :: prefixes
clarkbfungi: I'm thinking we may not get a second reviewer for the jvb stuff today. Should we go ahead and approve it now or ask ianw to review during australia monday morning?19:24
fungiclarkb: i figured i'd approve those in a little while if another infra-root doesn't happen along19:24
fungiworking on some stir fry now but will circle back around to those once i'm done19:28
clarkbnow I want stir fry19:33
fungiit's deep-fry/stir-fry... i'm practicing my skills at doing mongolian beef19:35
clarkbI had some leftover curry for lunch which was good19:37
*** cloudnull has quit IRC19:47
*** cloudnull has joined #opendev19:47
*** cloudnull has quit IRC19:49
*** cloudnull has joined #opendev19:58
*** cloudnull has quit IRC20:04
*** cloudnull has joined #opendev20:05
mordrednow I want stir-fry too20:07
corvusfungi: hope you can keep up with all the orders!20:11
fungii actually can't20:21
fungidredging all the wafer-thin slices of steak is something i need to get faster at20:21
*** roman_g has joined #opendev20:24
*** bhagyashris has joined #opendev20:47
*** ianw has quit IRC20:50
*** prometheanfire has quit IRC20:50
*** bhagyashris|sick has quit IRC20:50
*** SotK has quit IRC20:50
*** ianw has joined #opendev20:50
*** prometheanfire has joined #opendev20:50
*** SotK has joined #opendev20:50
*** roman_g has quit IRC20:50
openstackgerritMerged opendev/ master: Add jvb02
clarkbfungi looks like failed on tox linters due toa timeout21:32
clarkbshould we reenqueue to the gate or just recheck?21:33
clarkb(lookingat the timed out job it seems to have just been slow)21:33
* clarkb enqueues because running out of day21:33
fungiyeah, sorry, please do21:35
fungigot sidetracked prepping pizza dough for tomorrow21:35
*** mlavalle has quit IRC21:48
prometheanfireare we still being scraped?21:54
* prometheanfire can't connect to gerrit via gertty21:54
clarkbdid you set a new api token?21:55
prometheanfireoh, that needed? ok21:56
prometheanfireI need a new http-password set?21:57
clarkbyes, we unset all of them21:57
prometheanfireah, kk done21:57
fungiyeah, gerrit 2.13 still keeps them in plaintext in the db22:04
fungiwhat we're upgrading to will put kdf (bcrypt) hashes in there instead22:04
fungidownside is that gerrit will no longer be able to show you the key once set, other than ephemerally when it's generated22:05
fungiso if you forget it you can't look it up, you'll just have to regenerate it22:05
fungiultimately much safer though22:06
prometheanfireya, I prefer that anyway :P22:06
fungiso do we22:06
clarkbI think that tox linters job may fail again22:08
clarkbit takes 10 minutes just to run tox without any tests? that installs deps?22:08
clarkbwhy is that so slow22:08
fungithe times i watched, it looked like it was taking forever on ansible-lint22:10
clarkbya its slow there too22:10
clarkbfungi: do you think we should just increase the timeout? I'm not sure how much slowness dbeugging I want to do right now22:17
clarkb unfortunately doesn't have timing info for the pip install steps22:20
fungimmm, maybe22:25
openstackgerritClark Boylan proposed opendev/system-config master: Add jvb02 prior to the PTG
clarkbthat bumps the tox-linters timeout22:28
fungimaybe sometime soon we can figure out how to speed that up22:29
clarkbsupposedly not using find makes it go faster because then you don't have python startup overhead over and over22:30
clarkbbut in the past it hasn't found all the files when we did that so may need double checking22:30
fungii have a feeling this is part of the problem:
fungiyeah, what you just said22:31
fungiwe basically incur the startup overhead 300x22:31
fungiand that multiplier is only going to continue to increase as we port more systems from puppet to ansible22:32
*** slaweq has quit IRC22:33
fungi switched us back to using find in march, suggesting "ansible-lints ability to find ansible files is less than good"22:34
clarkbya we tried swithicng and it dind't work at all iirc22:34
clarkbso we reverted22:34
*** qchris has quit IRC22:41
*** qchris has joined #opendev22:55
*** sean-k-mooney has quit IRC23:20
*** sean-k-mooney has joined #opendev23:21
*** whoami-rajat__ has quit IRC23:36
openstackgerritMerged opendev/system-config master: Add jvb02 prior to the PTG
fungiclarkb: ^23:37
*** owalsh_ has joined #opendev23:40
*** hamalq_ has quit IRC23:43
*** owalsh has quit IRC23:43
*** owalsh has joined #opendev23:45
*** tosky has quit IRC23:46
clarkbchecking ti see if ansible ran happily23:47
clarkboh it hasn't run yet because it updated inventory os runs all the jobs23:48
clarkbI'll check on it a bit later then23:48
*** owalsh_ has quit IRC23:48
fungibut at least it merged after upping the linters timeout23:49

Generated by 2.17.2 by Marius Gedminas - find it at!