Tuesday, 2020-03-24

*** tosky has quit IRC00:01
mnaserso this is weird but i was just scrolling through opendev00:04
mnaserand noticed 4 nodes that were in rax-ord which are 3 days old00:04
mnaserthey're also ubuntu-bionic so its not like its a super unpopular nodeset00:05
mnaser(they're "ready" too)00:05
clarkbpossible that nodepool isn't properly reusing those if they've been originally procurred for a dropped request?00:06
clarkbtypically older nodes would be due to deletes that refuse to work or holds00:07
clarkbbeing ready implies neither of those cases are the case00:07
mnaseryeah, that's why it semed curious to me00:14
ianwwell, the problem starts @ https://opendev.org/openstack/python-openstackclient/src/branch/master/openstackclient/volume/v2/volume.py#L737 ... but after that i'm not so sure00:29
ianw#status log removed old files02 and static.openstack.org servers00:43
openstackstatusianw: finished logging00:43
*** mlavalle has quit IRC00:54
mordredianw: bother. so - it's entirely possible osc isn't pulling things from clouds.yaml fully properly here. might be worth trying volume_endpoint_override as well .. or there's also potentially just a bug01:30
ianwmordred: volume_endpoint_override seems to give the same error01:31
mordredianw: blerg01:32
ianwauth_config_hook(): ... 'volume_api_version': '2', 'volume_endpoint_override': 'https://dfw.blockstorage.api.rackspacecloud.com/v2/',01:32
ianw(same for block_) ... so it gets into the config that far01:32
mordredianw: try setting --os-endpoint-override=https://dfw.blockstorage.api.rackspacecloud.com/v2/ on the command line? I'm betting it's an internal mapping issue01:33
mordredianw: (since osc is ultimately instantiating python-cinderclient, some stuff can go to the bad place)01:33
ianwumm, it doesn't like os-endpoint-override01:34
mordredok. well - I'll swing back around to it in the morning if you haven't magically solved it :)01:34
ianw--os-endpoint doesn't work01:35
*** diablo_rojo_phon has joined #opendev02:29
ianwok wow i got a volume list out but OMG this is all so so so terrible ...03:07
*** diablo_rojo has quit IRC04:27
ianwmordred: in short; i have filed https://storyboard.openstack.org/#!/story/2007459.  it seems that cinder does not pass the api endpoint correctly, which i have prosed a fix for in https://review.opendev.org/71457005:13
ianwmordred: however, it still seems that openstacksdk/openstackclient doesn't correctly pass the os-endpoint information on to cinderclient correctly.  i will leave that to you as i'm out of brain power for now :)05:14
ianwmordred: details in task #3913705:15
*** DSpider has joined #opendev05:50
*** diablo_rojo has joined #opendev07:02
*** dpawlik has joined #opendev07:16
*** diablo_rojo has quit IRC07:35
*** roman_g has quit IRC07:48
*** ralonsoh has joined #opendev08:13
*** rpittau|afk is now known as rpittau08:26
*** tosky has joined #opendev08:41
*** roman_g has joined #opendev10:59
*** dpawlik has quit IRC11:33
*** dpawlik has joined #opendev11:34
mordredianw: awesome - thanks!11:44
*** roman_g has quit IRC12:03
mordredianw: https://review.opendev.org/714630 Update Rackspace vendor profile for cinder v212:57
mordredianw: https://review.opendev.org/714624 Honor endpoint override from config for volume12:57
mordredclarkb: ^^12:57
openstackgerritMonty Taylor proposed openstack/project-config master: Stop running against devel of ansible/ansible  https://review.opendev.org/71463113:01
mordredconfig-core: could you please approve that real quick ^^?13:02
mordredcorvus: when you're up - we should maybe try to debug why zuul ran a job on https://github.com/ansible/ansible/pull/68122/files13:04
*** hashar has joined #opendev13:05
mnasermordred: +213:05
mordredmnaser: thanks. I went ahead and clicked +A since right now we're making noise on ansible PRs13:05
openstackgerritMerged openstack/project-config master: Stop running against devel of ansible/ansible  https://review.opendev.org/71463113:20
mordredmnaser: since you're using python-base and python-builder images, https://review.opendev.org/#/c/714532/ is worth being aware of13:35
mordredmnaser: as well as https://review.opendev.org/#/c/711073/13:35
*** hashar has quit IRC13:37
openstackgerritArtom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA  https://review.opendev.org/71447813:47
corvusmordred: did anything regarding that sdk job change recently?13:55
corvusmordred: (or anything in the sdk repo?  other branches?)13:55
mnasermordred: both lgtm :)13:56
openstackgerritSagi Shnaidman proposed openstack/project-config master: Remove devel branch for ansible openstack jobs  https://review.opendev.org/71464713:59
mordredcorvus: not since jan 2013:59
mordredlemme check other branches13:59
mordredcorvus: nope. nothing in .zuul.yaml changed in sdk repo14:01
corvusmordred: any new branches created or deleted?14:01
mordrednot in sdk14:01
mordredthere were some branch shuffles in ansible - but that was mostly a branch creation and deletion in the ansible repo14:02
mordredcorvus: and then all of the files which match the matchers we had in that definition were deleted yesterday14:02
mordredor maybe a little while before that - but recently-ish14:07
corvusmordred: since the zuul configuration has been removed, i'm not sure i can determine the cause14:15
mordredcorvus: ok. do you think it's worth re-adding the config for a bit to help with that? (I could update the job content to stop failing so nobody on the ansible side would notice)14:16
corvusmordred: the logs don't have any clues.  what i would do next is to re-equeue that change with the repl debug console open and inspect it.14:16
*** hashar has joined #opendev14:16
openstackgerritSabeel Ansari proposed openstack/project-config master: Add cert-manager to StarlingX  https://review.opendev.org/71465214:19
corvusmordred: or you could change the 3pc pipeline not to report to github14:19
mordredcorvus: in other news ... the jeepyb change is successfully running system-config upload changes14:20
mordredcorvus: well - we still want to run and report on changes to stable branches14:20
mordredupload jobs14:21
corvusmordred: if we want to debug this, i would suggest we turn off all github reporting14:22
mordredah. nod14:22
corvusi think that's what we should do.14:22
mordredcorvus: ok. I'll turn off reporting and re-enable that job14:23
AJaegerare we still in freeze regarding repo changes and new reps?14:23
corvusmordred: ++ i think that can be one change?14:23
openstackgerritMonty Taylor proposed openstack/project-config master: Disable github reporting and re-add devel ansible job  https://review.opendev.org/71465414:25
mordredAJaeger: yes - but we're _really_ close14:25
mordredcorvus: does that look right?14:25
AJaegermordred: thanks14:30
openstackgerritMonty Taylor proposed openstack/project-config master: Revert "Disable github reporting and re-add devel ansible job"  https://review.opendev.org/71466014:33
mordredand there's a followup revert revert14:33
mordredinfra-root: mirror.iad looks like it's having sads14:34
mordredI can't ssh to it14:35
* mordred is logging in to the web console to check there14:35
fungiis that mirror.iad.rax.openstack.org or mirror.iad.rax.opendev.org? both seem broken but in slightly different ways14:37
mordredfungi: .openstack14:37
mordredhttps://zuul.opendev.org/t/openstack/build/61b24d96e0ec4fc7b1fe67841c64dbd6/log/job-output.txt#459 is a job log output14:37
mordredbut I've seen it in a couple now14:37
mordredthere is an "apache task hung" message on the console14:38
mordredshould we give it a cloud reboot?14:38
fungiyeah, cacti says it ceased responding to snmp polls around 01:41z14:39
mordredkk. rebooting14:39
fungithe opendev.org version has apparently been offline since around 05:00z on 2020-02-0614:40
corvusmordred: lgtm; AJaeger can you push https://review.opendev.org/714654 through?14:41
mordredk. back up14:42
fungistatus log it?14:42
fungior i can14:42
AJaegercorvus: checking...14:43
mordred#status log rebooted mirror.iad.rax.openstack.org because it was hung14:43
openstackgerritMerged openstack/project-config master: Disable github reporting and re-add devel ansible job  https://review.opendev.org/71465414:55
corvusmordred: ^ i'll manually enqueue and debug that after breakfast14:59
mordredcorvus: cool15:06
openstackgerritMerged opendev/jeepyb master: Clean up some more python3 things  https://review.opendev.org/71431815:08
clarkbre reboots it seems like if live migrations fail the servers end up in those weird states15:08
clarkbmordred: the sdk changes to fix volume things lgtm15:40
clarkbre the ansible thing its due to pulling all the useful bits out of devel aiui15:42
clarkbI assume we'll want to fix that job once upstream stablizes as we'll need to do that anyway to upgrade15:42
clarkbinfra-root wasn't there a change up of some sort to update the text at opendev.org's root page? I thought there was and it would be a good candidate for monitoring the new gitea restart procedures15:49
clarkbI'm thinking we (I) should do that sooner than later to ensure it isn't buggy before we need it15:49
openstackgerritMerged openstack/project-config master: Revert "Disable github reporting and re-add devel ansible job"  https://review.opendev.org/71466015:52
corvusmordred: ^ maybe shoulda put a wip on that, eh?15:53
corvusi literally just connected to the repl to start debugging15:53
mordredcorvus: crappit15:54
corvusi'm reverting15:54
clarkboh sorry15:54
clarkbthe commit message said you were done15:54
mordredyeah - sorry - I wrote it to be right for when we landed it - my bad, I should have marked it WIP15:55
openstackgerritJames E. Blair proposed openstack/project-config master: Revert "Revert "Disable github reporting and re-add devel ansible job""  https://review.opendev.org/71468515:55
fungiclarkb: the only one i was aware of already merged15:57
fungi(updating the irc channel listed there)15:57
clarkbfungi: that may have been it. I suppose there is likely some other edit that makes sense I can come up with :)15:58
clarkbmordred: did you see ianw's comment on https://review.opendev.org/#/c/714553/2 that has me reconsidering if the sdk changes are correct15:58
fungihas anybody put any thought into a new discussion ml for opendev yet?15:58
clarkbmordred: I think the most correct thing is to keep using v1 volume api based on that15:58
fungilike service-discuss@ maybe15:58
clarkbfungi: we probably should. I've got a meeting agenda item to discuss similar things (meeting location in particular)15:59
fungiawesome. will remember to bring it up again in a few hours15:59
openstackgerritSabeel Ansari proposed openstack/project-config master: Add nginx-ingress-controller armada app to StarlinX  https://review.opendev.org/71468616:00
mordredclarkb: the v2 api is there - I have used it just fine16:06
mordredclarkb: the v2 api does not have an accessible version discovery document16:06
clarkbmordred: is that the only difference? the rax doc implies its not a full v2 api16:06
clarkbthe doc seems to imply there are quite a few deltas16:07
clarkbits possible they don't matter muc though16:07
clarkbmordred: rereading it though I'm not sure their v1 cinder api is any better?16:09
clarkbthis document is comparing rax's api to cinder v2, I assume cinder v1 has similar deltas?16:09
mordredprobably? it's a great question tbh16:10
clarkbits unfortunate that its rax api v116:11
clarkbwhich may be confusing if that doesn't map onto cinder api v116:11
openstackgerritMerged openstack/project-config master: Revert "Revert "Disable github reporting and re-add devel ansible job""  https://review.opendev.org/71468516:20
openstackgerritSabeel Ansari proposed openstack/project-config master: Add Cert-Manager Armada app to StarlingX  https://review.opendev.org/71468916:25
mordredclarkb: also - with my patch to openstackclient- openstack volume list works just fine16:32
mordredclarkb: we should maybe test whatever other osc commands we use16:32
mordredclarkb: (well, with the sdk patch and the osc patch)16:32
openstackgerritClark Boylan proposed opendev/system-config master: Note about gitea functionality on opendev.org  https://review.opendev.org/71469116:33
clarkbinfra-root ^ maybe something like that to exercise the gitea restart changes16:33
mordredclarkb: sounds good16:37
openstackgerritTobias Henkel proposed opendev/gear master: Move handleDisconnect into BaseClientServer  https://review.opendev.org/71470916:44
openstackgerritMonty Taylor proposed opendev/system-config master: Add jeepyb to file matchers for gerrit-base  https://review.opendev.org/71471116:48
mordredclarkb: sigh ^^16:48
mordredclarkb: we built images on the jeepyb patch - but we didn't build the gerrit-base image which is the thing that actually includes the jeepyb content :)16:48
clarkbmordred: hrm that won't rebuild if top level files of jeepyb change?16:50
clarkbthats a weird one. The fix is definitely an improvement but probably won't catch all cases16:50
mordredclarkb: yeah - but it's probably ok for the time being I hope16:50
mordredclarkb: we could also put in more file matchers in teh project-config pipeline definition16:50
clarkbmordred: maybe we can add something like ^[^/]+$16:54
clarkbI think that will match all top level files16:54
mordredclarkb: yeah - but I don't think we want that to be true when we run it from system-config16:54
clarkbya so would be in the project-config definition for the jeepyb application? though maybe there we can just say .*16:55
openstackgerritMonty Taylor proposed openstack/project-config master: Add more file matchers for jeepyb  https://review.opendev.org/71471316:57
mordredclarkb: like that ^^16:57
clarkbmordred: can we reduce it to ^.*$ there?16:58
mordredclarkb: worth a shot17:00
openstackgerritMonty Taylor proposed openstack/project-config master: Always build gerrit images on jeepyb changes  https://review.opendev.org/71471317:01
mordredclarkb: ^^17:01
*** mlavalle has joined #opendev17:06
*** rpittau is now known as rpittau|afk17:18
*** hashar has quit IRC17:29
AJaegermordred: +317:38
AJaegerconfig-core, a few reviews for your consideration: https://review.opendev.org/679082 https://review.opendev.org/714506 https://review.opendev.org/71454317:41
*** openstackstatus has quit IRC17:45
*** openstackstatus has joined #opendev17:46
*** ChanServ sets mode: +v openstackstatus17:46
clarkbfungi on https://review.opendev.org/#/c/714506/ (noted above) can you ack or nack that? I wasn't quite clear on whether you were saying that is ok yesterday17:46
openstackgerritMerged openstack/project-config master: Always build gerrit images on jeepyb changes  https://review.opendev.org/71471317:46
fungiclarkb: ahh, i was talking about the related change for zuul's encrypt_secret.py but sure, i'll review this too17:48
openstackgerritMerged opendev/base-jobs master: Remove env var for tarballs mirror  https://review.opendev.org/71454317:59
openstackgerritMonty Taylor proposed opendev/jeepyb master: Just use flake8 instead of hacking  https://review.opendev.org/71472918:00
mordredclarkb, fungi: that's not a very meaningful change - but should exercise our latest project-config update18:01
openstackgerritMerged zuul/zuul-jobs master: Trim whitespace from uri password for docker promote  https://review.opendev.org/71450618:06
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed opendev/lodgeit master: Add lodgeit-db script  https://review.opendev.org/71473218:16
*** diablo_rojo has joined #opendev18:42
mordredclarkb, ianw: https://review.opendev.org/714624 has merged - so even without the rax profile update in the sdk change - it should allow testing the use of block_storage_endpoint_override in clouds.yaml with osc18:49
mordred(it works for me on my laptop - but obviously further testing isn't a bad idea)18:49
*** dpawlik has quit IRC18:53
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed opendev/lodgeit master: Add lodgeit-db script  https://review.opendev.org/71473218:57
*** ralonsoh has quit IRC19:07
mordredianw: when you get a chance, if you can confirm that master of osc works for you I'll get a release cut19:21
ianwmordred: will it work without the cinderclient changes?19:22
mordredat least - it worked for me without those locally19:22
mordredyou'll need the cinderclient changes if you want to use cinderclient cli with an override I believe19:22
ianwok interesting, so it uses something other than cinderclient keystone session?  or it sets up its own?19:23
mordredit's .... complicated :)19:23
ianwhaha yes, that i will agree on!19:23
mordredbut yeah- it constructs a Client object from cinderclient from its own parameter processing19:23
mordredthat processing itself is too complex because it's also double-processing things that sdk will do for it for historical reasons19:24
mordredand I haven't unwound that yet19:24
mordredianw: I *almost* rage-removed cinderclient from osc this morning ... but I figured I should just fix it in place as-is :)19:27
ianwmordred: sorry, so should "--os-cloud=openstackci-rax  --os-region=DFW --os-volume-api=2 --os-endpoint=https://dfw.blockstorage.api.rackspacecloud.com/v1/...... volume list" work *without* block_storage_endpoint_override in clouds.yaml?19:30
mordredjust the block_storage_endpoint_override in clouds.yaml19:33
mordredianw: the other stuff is a much larger rats-nest to unwind19:33
mordred(also - be sure it's /v2/$project_id19:34
mordredianw: /v2/61027519:34
mordredor whatever it is19:34
openstackgerritMonty Taylor proposed opendev/system-config master: Actually write out the projects.ini config file  https://review.opendev.org/71476019:39
mordredclarkb, corvus: ^^ silly mistake19:39
ianwmordred: huh, ok .. so "/v2" actually works, despite not being documented as an endpoint either in their catalog or documentation?19:42
mordredit's in the service discovery doc19:43
*** hashar has joined #opendev19:43
mordredif you do:19:43
mordredianw: http://paste.openstack.org/show/791101/19:45
mordredianw: you can see the version discovery document19:45
mordredianw: acutlaly - it's curl-able19:45
mordredcurl https://dfw.blockstorage.api.rackspacecloud.com/19:45
mordred{"versions": [{"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "id": "v1.0", "links": [{"href": "http://dfw.blockstorage.api.rackspacecloud.com/v1/", "rel": "self"}]}, {"status": "CURRENT", "updated": "2012-11-21T11:33:21Z", "id": "v2.0", "links": [{"href": "http://dfw.blockstorage.api.rackspacecloud.com/v2/", "rel": "self"}]}]}19:45
ianwwell, TIL ...19:52
clarkbmordred: corvus https://review.opendev.org/#/c/714691/1 that job failure was due to connectivity problems to dockerhub. We don't use our caches there bceause its using the system-config roles not the zuul roles19:53
clarkbI'm not sure if there is a good fix for that, but wanted to call it out19:53
ianwmordred: i can confirm volume_endpoint_override: https://dfw.blockstorage.api.rackspacecloud.com/v2/610275 makes it work for me with master openstackclient19:53
mordredianw: woot!19:53
mordredI'll get a release cut19:53
openstackgerritMonty Taylor proposed opendev/jeepyb master: Just use flake8 instead of hacking  https://review.opendev.org/71472920:00
clarkbcorvus: to be clear I Think the meetpad idea is an excellent one and will likely capture the process of inperson brainstorming and discussion at PTGs and design summits far better than many of the other options available to us. I'm simply anticipating that as plans congeal there will be more askes for tools or tweaks to existing tools20:02
corvusclarkb: ok, i wasn't sure if you knew of other efforts that might impact or relate to this20:03
clarkbI do not20:03
corvusi thought it may be useful, but don't want to needlessly spin wheels or step on other peoples toes20:03
corvusbut sounds like we're good :)20:03
clarkbI think the first lareg community brainstorm is happening next week?20:03
clarkbfungi: ^ would probably know for sure20:04
openstackgerritMonty Taylor proposed opendev/jeepyb master: Trigger new images  https://review.opendev.org/71476320:04
clarkbas far as anticipating specific things I know video recording for zoom calls are a thing some communities (starlingx) rely on. I also expect that scheduling due to time zones may be a tricky problem people look to tooling to help solve20:05
mordredclarkb, corvus: cool - that seems to trigger the right things ^^20:05
fungithere's a "community meeting" (conference call) which i think it going to be announced for 15:00z on thursday april 220:05
openstackgerritIan Wienand proposed opendev/system-config master: Override RAX block storage endpoint  https://review.opendev.org/71455320:05
mordred(we were missing a job config the last time we built images- we dont' actually have new content, but we need new images built)20:05
fungiclarkb: not sure if that's what you're referring to20:05
ianwmordred: ^^ i think that's now right to update configs20:06
clarkbfungi: ya I think that is the one20:06
mordredianw: ++20:06
clarkbianw: one small fix on that change20:08
clarkb(noted it inline and now I'm finding lunch)20:08
clarkband probably a short indoor bike ride because its cold outside20:08
* mordred is going to take an outdoor bike ride - because it is not cold outside20:11
fungii love that it must now be explicitly differentiated from an "indoor20:17
fungi" bike ride20:17
mordredclarkb, corvus, fungi, ianw: if anybody wants to +A https://review.opendev.org/#/c/714763/ - we should have new images by the time I'm back20:17
fungithat's... intruiging. is it really a diffless commit, or is gertty lying to me?20:20
mordredfungi: really a diffless commit20:20
mordredI just did a git commit --allow-empty20:20
fungiyeah, gerrit's webui confirms20:20
fungii'm approving just because i want to see this "merge"20:21
AJaegermordred: will jobs run - of will filematcher not find any files?20:21
mordredAJaeger: it looks like it's running all of them20:21
mordred(including gerrit-base which is what was missing before)20:22
openstackgerritIan Wienand proposed opendev/system-config master: Override RAX block storage endpoint  https://review.opendev.org/71455320:22
* mordred afks20:22
AJaegermordred: indeed - http://zuul.opendev.org/t/openstack/status/change/714763,120:22
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Fixing results from ZAP Scanning Report  https://review.opendev.org/71435120:33
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Fixing results from ZAP Scanning Report  https://review.opendev.org/71435120:40
corvusclarkb, fungi: who's organizing that community meeting?20:40
openstackgerritMerged opendev/system-config master: OpenStackID 3.0.6 production deployment  https://review.opendev.org/71426320:50
fungicorvus: allison price is working on an agenda for it which she says she hopes to send to project mailing lists tomorrow20:54
corvusfungi: thanks20:55
fungitentative agenda is the openstack project 10th anniversary activities and an events update, plus a lot of q&a time for anyone on the call who has input/suggestions/ideas20:55
fungiso sounds like virtual ptg is on topic20:55
*** hashar has quit IRC21:08
mordredianw: sad: https://review.opendev.org/#/c/71455321:33
openstackgerritIan Wienand proposed opendev/system-config master: Override RAX block storage endpoint  https://review.opendev.org/71455321:35
ianwthis time for sure! :)21:35
clarkbianw: yup +221:55
openstackgerritMerged opendev/jeepyb master: Trigger new images  https://review.opendev.org/71476321:56
mordredclarkb, ianw: https://review.opendev.org/#/c/714760/ ?22:03
mordredclarkb, ianw: also Release 5.1.0 of python-openstackclient  https://review.opendev.org/71478122:03
clarkbmordred: 760 done22:04
openstackgerritMerged opendev/system-config master: Note about gitea functionality on opendev.org  https://review.opendev.org/71469122:04
openstackgerritMerged opendev/puppet-openstackid master: Fixing results from ZAP Scanning Report  https://review.opendev.org/71435122:04
clarkbI'll watch 69122:05
corvusmordred: i'm not sure what to do about that ansible issue22:18
mordredcorvus: me either. maybe we'll learn something when gundalow replies22:19
corvusmordred: at this point, i think it's going to take quite a bit of debugging.... do we want to do that, or write this off as a blip since we're stopping testing on that repo anyway?22:19
mordredcorvus: I think I'm leaning more towards writing it off as a blip22:19
corvusmordred: i want to find the right balance of being supportive but not imposing on the ansible project22:21
corvusmordred: i notice from the debug logs that we used to see a lot of github traffic related to ansible/ansible, and i think it may have stopped yesterday22:21
corvusmordred: did you log in using the opendev superuser access to investigate the app?22:22
mordredcorvus: I did - but I couldn't find anywhere to see where it was installed22:23
corvusmordred: 2020-03-23 19:23:30,044 was the last time we saw a GithubTriggerEvent from ansible/ansible22:24
corvuswe have seen them from kata since then22:24
mordredcorvus: there are _many_ payloads in recent deliveries since that date/time - but spot checking I don't see any from ansible/ansible22:26
corvusmordred: i'm leaning toward thinking they've uninstalled it22:27
mordredI'm leaning that way too22:27
corvusmordred: hopefully gundalow can confirm; and if that's the case, that at least explains the invalid change thing22:27
corvusthen the question is, do we ask to reinstall it so we can debug the original error, or just say 'nevermind' and revert the revert revert revert?22:28
mordredyeah - although not so much why it went weird in the first place :(22:28
mordredthat's a good question - and one for which I don't know that I have an answer for22:29
corvusokay, well, we can pick this up again tomorrow if we want; i'll stand down for today22:30
mordredcool - I'll let you know what I hear from gundalow when I do22:31
corvusmordred: thx22:31
clarkbcurrently gitea nodes are getting the base server playbook applied to them22:33
clarkbshould update the deployment soon I expect22:33
clarkbI'm going to take the zuul user survey now for OpenDev22:34
*** bolg has joined #opendev22:39
clarkbcorvus: we aren't using the pagure driver are we?22:40
openstackgerritMerged openstack/diskimage-builder master: Use rpm -e instead of dnf for cleaning old kernels  https://review.opendev.org/71398822:42
clarkbcodesearch says no22:42
*** DSpider has quit IRC22:44
openstackgerritMerged opendev/system-config master: Actually write out the projects.ini config file  https://review.opendev.org/71476022:46
openstackgerritMerged opendev/system-config master: Override RAX block storage endpoint  https://review.opendev.org/71455322:46
clarkbps shows gitea-web process started after sshd but ansible syslog logs show things running in the correct sequence. I'm going to guess there is some sort of forking related there that may throw off the ps timestamp?22:49
clarkbreplication around that timeframe seems to have worked though22:52
clarkbthe rax block storage endpoint change above is an example https://gitea01.opendev.org:3000/opendev/system-config22:52
clarkbthat started about 40 seconds after the gitea-ssh container was started though so its not quite in the bit we think might have problems22:52
clarkbat the very least this doesn't seem to regress worse than the behavior we had before22:53
clarkbwhich was my biggest concern and wanted to rule out before it bit us later unexpectedly. So yay this seems happy22:53
mordredclarkb: nope22:55
mordredclarkb: sorry - lag - I was answering "we arent' using pagure"23:01
openstackgerritIan Wienand proposed opendev/infra-specs master: Move static.openstack.org replacement spec to completed  https://review.opendev.org/71478923:02
mordredclarkb: yay!23:02
clarkbmordred: ya I figured that was what you were responding to23:04
mordredclarkb, corvus: could I interest you in https://review.opendev.org/#/c/714532/ ?23:07
clarkboh ya that was on my list yesterday then it got lost23:08
clarkbmordred: left a thought and a clarification for other reviews but looks great to me23:19
mordredclarkb: setting version arg is a good idea23:24
mordredcorvus: ansible app was disabled - is reinstalled23:24
mordredcorvus: s/ansible/github/23:24
corvusmordred: thanks!23:25
clarkbcorvus: rereading the meetpad spec, it might be a good idea to make note of what changes we're carrying locally that prevent us from usign the upstream images (since we want to switch to the upstream images at some point)23:28
corvusclarkb: there is exactly one two-line patch23:28
clarkbcorvus: we can probably do that on the change to use the images though not necessarily in the spec23:28
corvusthere's a comment in the change about it23:29
clarkbah ok I must've missed it when looking over the meetpad changes23:29
clarkbis it in the docker image change?23:29
corvusclarkb: https://review.opendev.org/71450523:29
corvuseasy :)23:30
corvusnow, getting redhat to agree to the cla -- i dunno23:30
openstackgerritMerged opendev/infra-specs master: Move static.openstack.org replacement spec to completed  https://review.opendev.org/71478923:31
mordredmaybe clarkb could clean-room write the same patch?23:33
clarkbI haven't looked at the PR yet at least23:33
corvuswell, that might be an option if the cla situation is different for clarkb23:35
corvusi've sent off an email to redhat legal to see about setting the wheels in motion23:35
clarkbI can ask jbryce if you don't make progress23:37
clarkbcorvus: in the docker-compose file you've got ${CONFIG} as a var. Where is that value set?23:38
clarkbis docker-compose reading the .env file?23:38
clarkb(I'm assuming so and thats an interesting feature I had not come across before yet)23:38
corvusclarkb: yes in https://review.opendev.org/71423823:38
clarkbyup thats what I'm looking at. Ok making sure I grok what is going on there23:39
clarkband that must just be a docker-compose behavior23:39
corvusclarkb: and yes, it is interesting.  i like it because it's nice separation of inputs.  i don't like that it's a dotfile and i don't see a way to change that.23:39
openstackgerritMonty Taylor proposed opendev/system-config master: Start making 3.8 python images  https://review.opendev.org/71453223:41
mordredcorvus, clarkb: updated per clarkb's comment23:41
mordredclarkb: ok. manage-projects now works right via the image on review-dev23:44
mordredok. maybe if I were smarter I'd test more than one thing at a time23:45
openstackgerritGhanshyam Mann proposed openstack/project-config master: Replace python-charm-jobs to py3 job  https://review.opendev.org/71479623:50
clarkband I'm through the meetpad stack. I think it is ready to go but will keep the spec open until thursday as mentioned in the meeting23:51
openstackgerritMonty Taylor proposed opendev/jeepyb master: Update StringIO to use six  https://review.opendev.org/71479723:52
mordredclarkb: sigh ^^23:52
mordredshould have checked update-bug and update-blueprint and not just manage-projects23:53
clarkbyay testing23:53
*** mlavalle has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!