| ianw | yeah, what i was hoping was that the -src jobs could be validating the dib bindep.txt | 00:00 |
|---|---|---|
| ianw | but it just doesn't seem practical | 00:00 |
| fungi | doesn't running those jobs on changes to the bindep.txt in the devstack plug-in solve that anyway? | 00:00 |
| corvus | #status log restarted all of zuul with commit sha 67ef71d2a2d6b5b06e2355eefff00ae3df24bbf7 | 00:01 |
| mordred | woot | 00:01 |
| corvus | (well, it's still restarting, but ...) | 00:01 |
| mordred | corvus: so now landing https://review.openstack.org/#/c/630744/ should be safe | 00:01 |
| openstackstatus | corvus: finished logging | 00:02 |
| corvus | mordred: yep | 00:02 |
| mordred | corvus: at which point I can rework the not-yet-fully-working gerrit image build to use required-projects, which will be neat | 00:02 |
| clarkb | corvus: Re-registering as job is within its limits 9.45 <= 20.0, 62.4% <= 5.0, running, 34.7% <= 5.0, 3 <= 4 | 00:02 |
| clarkb | corvus: something seems off with the way we log <= 5.0, 62.4% is less than 5.0 what? | 00:03 |
| corvus | clarkb: i agree :) | 00:03 |
| clarkb | but other than that I agree I don't find any logs indicating that swap is directly the cause | 00:03 |
| clarkb | (or any other issue that is) | 00:03 |
| *** yamamoto has quit IRC | 00:03 | |
| ianw | fungi: not really, i don't think we can skew when you look at the whole suite of tests -- if we're using dib's bindep in all it's functional tests, but the nodepool plugins bindep in it's test, taken as a whole both have to pass to get a zuul +1 | 00:04 |
| corvus | clarkb: yeah. executor process swapping is one of the things that would cause a job to take a while to start, but not the only one. | 00:04 |
| *** gfidente|afk has quit IRC | 00:04 | |
| *** yamamoto has joined #openstack-infra | 00:05 | |
| *** markvoelker has quit IRC | 00:06 | |
| fungi | ianw: unless i'm misunderstanding, that sounds like what we'd want anyway | 00:07 |
| *** mriedem_away has quit IRC | 00:07 | |
| *** yamamoto has quit IRC | 00:07 | |
| corvus | there seems to be a problem with zuul | 00:08 |
| clarkb | puppet just ran on review.o.o using the previous puppet-gerrit state. I expect it will run with the new state in about 45 minutes | 00:08 |
| fungi | ianw: or maybe what you're really looking for is a dib devstack plugin which the nodepool devstack plugin uses? | 00:08 |
| clarkb | corvus: let me know if I can help | 00:08 |
| corvus | i don't know the problem yet, other than the status page isn't working | 00:08 |
| fungi | i'm also around, in case i can be of assistance | 00:08 |
| clarkb | corvus: do we need to add the zk config to our zuul web? | 00:08 |
| corvus | should be there | 00:09 |
| clarkb | I seem to recall we write it everywhere but I can double check | 00:09 |
| corvus | oh, zuul-web is not running | 00:09 |
| corvus | 2019-01-14 23:57:11,832 DEBUG zuul.Web: Configured logging: 3.4.1.dev19 | 00:09 |
| corvus | it attempted to start | 00:10 |
| corvus | but did not log why it died | 00:10 |
| clarkb | systemd seems to show it restarted it a few times | 00:10 |
| corvus | i'll try running it in fg | 00:10 |
| *** jbadiapa has quit IRC | 00:10 | |
| *** yamamoto has joined #openstack-infra | 00:11 | |
| ianw | it also sounds similar holding onto the logging handlers across daemonize to https://review.openstack.org/#/c/547889/ which i have never got back to and ported as requested | 00:11 |
| corvus | okay that's weird, it's running now | 00:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Add repos from gerrit.googlesource.com to zuul https://review.openstack.org/630744 | 00:11 |
| corvus | i don't know what happened earlier; i didn't do anything other than restart it again | 00:12 |
| corvus | re-enqueuing now | 00:12 |
| clarkb | corvus: perhaps the old web server was still bound to its port? | 00:12 |
| corvus | clarkb: i'm pretty sure i checked that it was stopped, but perhaps i was mistaken | 00:13 |
| *** yamamoto has quit IRC | 00:13 | |
| *** yamamoto has joined #openstack-infra | 00:14 | |
| corvus | hypothesis: an unhandled (unlogged) error in sql migrations | 00:14 |
| corvus | 2019-01-14 23:57:11,832 DEBUG zuul.Web: Configured logging: 3.4.1.dev19 | 00:15 |
| corvus | 2019-01-14 23:57:11,947 DEBUG zuul.SQLConnection: Current migration revision: ea2bae776723 | 00:15 |
| corvus | 2019-01-15 00:10:23,406 DEBUG zuul.Web: Configured logging: 3.4.1.dev19 | 00:15 |
| corvus | 2019-01-15 00:10:23,543 DEBUG zuul.SQLConnection: Current migration revision: 649ce63b5fe5 | 00:15 |
| clarkb | oh interesting | 00:15 |
| corvus | the restart that bombed had a different sql migration sha than the following which succeeded -- but both were running the same zuul version. so probably the scheduler was in the process of migrating | 00:16 |
| corvus | and maybe web started the migration and then failed once the scheduler finished | 00:16 |
| corvus | (or even just attempting to start the migration failed) | 00:16 |
| *** yamamoto has quit IRC | 00:16 | |
| corvus | ianw: and yeah, that logging change would have helped :) | 00:18 |
| corvus | i'll push up a targeted fix | 00:18 |
| *** wolverineav has quit IRC | 00:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Test a child job depending on a parent that did not run https://review.openstack.org/629885 | 00:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 00:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 00:19 |
| mordred | corvus: ^^ this time with required-projects | 00:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Log connection exceptions when starting web https://review.openstack.org/630796 | 00:20 |
| corvus | clarkb, ianw, fungi: ^ i think that would have caught our mystery error | 00:20 |
| mordred | ++ | 00:21 |
| corvus | mordred: well need to wait for ansible to puppet before that'll really work | 00:21 |
| mordred | corvus: oh yeah. good point | 00:21 |
| clarkb | we can all wait for puppet together | 00:22 |
| fungi | eep | 00:23 |
| clarkb | unrelated but I smell popcorn (told its actually rolls in the oven) and hear The Princess Bridge downstairs | 00:23 |
| clarkb | I might have to do my waiting down there :) | 00:24 |
| mordred | clarkb: is the Princess Bridge something I don't know about or a typing error? | 00:24 |
| clarkb | its a typing error :) | 00:24 |
| clarkb | *Bride | 00:24 |
| mordred | kk. it totally could have been the other thing too | 00:24 |
| clarkb | I just ssh'd into bridge.o.o to watch ansible | 00:24 |
| fungi | i was about to go add the princess bridge to my queue | 00:24 |
| mordred | fungi: ++ | 00:24 |
| mordred | I figured it would be a portland retelling of the princess bride | 00:25 |
| clarkb | my typing drivers have a bad habit of caching commonly used words and overusing them | 00:25 |
| fungi | and also make actual popcorn which isn't rolls | 00:25 |
| corvus | i thought it was the one where buttercup goes to terabithia | 00:25 |
| mordred | corvus: I kind of want to make a series of film adaptations of books but with characters from the princess bridge subbed in | 00:26 |
| mordred | corvus: the dread pirate roberts and the rats of nimh | 00:26 |
| corvus | mordred: if you want to do *that*, you'd have to write "the princess bridge" first. :) | 00:26 |
| mordred | yeah | 00:27 |
| mordred | that's too much work | 00:27 |
| clarkb | Trek in the Park was in Cathedral park which is so named as its under the St. Johns Bridge and it looks like a cathedral if you got rid of all the walls | 00:27 |
| *** yamamoto has joined #openstack-infra | 00:27 | |
| clarkb | now I'm trying to find a picture that shows the bridge | 00:27 |
| corvus | clarkb: https://en.wikipedia.org/wiki/Cathedral_Park,_Portland,_Oregon#/media/File:Cathedral_Park_St_Johns_Bridge_-_Portland_Oregon.jpg | 00:28 |
| corvus | the flare cinches it | 00:29 |
| clarkb | now imagine a stage production with tribbles | 00:29 |
| clarkb | off on the left I think | 00:29 |
| clarkb | also if anyone is wondering that is the best bridge in portland | 00:30 |
| mordred | clarkb: it is clearly the best bridge in portland | 00:31 |
| mordred | corvus: I feel like you just admitted jj into the canon | 00:31 |
| *** yamamoto has quit IRC | 00:31 | |
| *** tosky has quit IRC | 00:32 | |
| corvus | mordred: i said that before i knew we were talking about star trek; i thought we were talking about stained glass windows | 00:33 |
| mordred | AH | 00:33 |
| mordred | corvus: ok. I understand your statement better | 00:33 |
| mordred | corvus: I thought you were saying "that there is a flare clinches that it really is star trek" - and I was confused | 00:33 |
| corvus | mordred: i forgive you | 00:34 |
| clarkb | discovery is getting a spin off, also to be locked away on cbs all access | 00:34 |
| corvus | i feel like that service is ironically named | 00:34 |
| mordred | yeah | 00:34 |
| clarkb | because if your first attempt isn't any good then just do it again worse | 00:34 |
| clarkb | (I mean it could be better, that is to be seen but I likely won't watch it because ya "all" isn't much here) | 00:34 |
| mordred | I haven't watched any of discovery - but that's mostly because the previews make it look too much like an action flick | 00:35 |
| mordred | I also might be old and set in my ways | 00:35 |
| clarkb | mordred: I couldn't finish season 1 | 00:35 |
| clarkb | the expanse was coming out at about the same time and it was far better | 00:35 |
| mordred | ok. good to know I'm not just projecting unnecessary grump | 00:35 |
| *** wolverineav has joined #openstack-infra | 00:35 | |
| clarkb | so watched that iwth my time instead | 00:35 |
| mordred | I have not seen that - I shoudl add to the watchlist? | 00:36 |
| clarkb | -infra does pick your favorite scifi | 00:36 |
| clarkb | mordred: you should | 00:36 |
| clarkb | its a bit bsg like | 00:36 |
| corvus | oh cool; i'm in the middle of a free-prime trial; i'll add it :) | 00:37 |
| fungi | i've had trouble getting into the expanse too. i made it about 6 episodes in... less bsg and more b5 to me (especially with the slow start to the plot) | 00:38 |
| clarkb | fungi: the expanse or discovery? | 00:38 |
| clarkb | (both were slow to me) | 00:38 |
| fungi | the expanse | 00:38 |
| clarkb | If you can get to the end of the first season of the expanse it sort of all starts to clikc together | 00:39 |
| fungi | i haven't even tried discovery yet. seen and own copies of every episode of every other st series so i should probably give it a shot | 00:39 |
| clarkb | but ya they put a lot of effort into world building early and it does start slow | 00:39 |
| corvus | clarkb, fungi: then it's exactly like b5 :) | 00:39 |
| fungi | indeed! | 00:39 |
| fungi | i mean, not picking on b5, i loved it. i just know a lot of people who never made it past season 1 | 00:39 |
| corvus | i loved it too. s1 hurts to watch. | 00:40 |
| fungi | i tried to get christine to watch it and she made it ~1.5 episodes before refusing to watch any more | 00:40 |
| fungi | starting her with season 1 was probably a mistake, in hindsight | 00:40 |
| corvus | i'll just be over here in the cone of silence | 00:41 |
| fungi | if only it had been a get smart reference :/ | 00:41 |
| fungi | just needed a shoe phone to round out the bit | 00:41 |
| clarkb | stranger things (and if you liked that then dark) are other recent shows really worth a watch. Moving away from scifi/fantasy Fargo is quite good. I'm hopeful for this new season of true detective too | 00:42 |
| fungi | stranger things is indeed marvellous. looking forward to another season. i haven't tried dark yet but it's on my list to check out. whether i can convince christine to watch something with me usually influences the chance i'll get around to it | 00:43 |
| clarkb | be warned that dark is quite dark | 00:44 |
| *** zaneb has joined #openstack-infra | 00:46 | |
| corvus | says right there on the tin i guess | 00:46 |
| corvus | clarkb, mordred, fungi: can you take a look at https://review.openstack.org/630782 ? | 00:46 |
| fungi | yup | 00:47 |
| corvus | i expect it to pass check; the real action is in promote, so we'll have to land it to see. | 00:47 |
| clarkb | does < work like that in ansible? | 00:48 |
| *** wolverineav has quit IRC | 00:48 | |
| clarkb | thats neat if so | 00:48 |
| corvus | clarkb: it's a text comparison, but that works for iso8601 | 00:48 |
| clarkb | corvus: was it intentional to remove the no_log from the get dockerhub token task? | 00:49 |
| corvus | clarkb: nope! | 00:49 |
| mordred | oh! | 00:50 |
| *** wolverineav has joined #openstack-infra | 00:50 | |
| fungi | that seems dangerous, yes | 00:50 |
| clarkb | also it looks like you wanted to consolidate token creation but its still in two places | 00:50 |
| corvus | also i don't need the delete there | 00:50 |
| mordred | clarkb: I see the no_log ... what am I missing? | 00:50 |
| clarkb | https://review.openstack.org/#/c/630782/2/playbooks/zuul/build-image/promote.yaml,unified has it too | 00:50 |
| corvus | clarkb: you'd think... but no, i need the JWT token for deleting tags, and the bearer token for pushing manifests | 00:50 |
| corvus | !@#$@$#$ | 00:51 |
| openstack | corvus: Error: "@#$@$#$" is not a valid command. | 00:51 |
| clarkb | ha | 00:51 |
| mordred | oh - I see it | 00:51 |
| clarkb | corvus: got it | 00:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Delete change tags from docker image repos https://review.openstack.org/630782 | 00:51 |
| corvus | i really tried doing both with the other and failed | 00:51 |
| corvus | but maybe there's some magic combination of hostnames i'm not using or i dunno | 00:51 |
| corvus | (note how many different hostnames are involved in this) | 00:52 |
| clarkb | puppet has run on review.o.o which should've used the new puppet-gerrit and I don't see anything unexpected so I think that is good and ready for testing tomorrow | 00:52 |
| mordred | corvus: otoh - doing rest from ansible is actually really pleasant! | 00:52 |
| corvus | auth.docker.io, registry.hub.docker.com, hub.docker.com | 00:52 |
| corvus | mordred: yes, i thoroughly enjoy it | 00:52 |
| corvus | mordred: it's worked really well with docker (as much as it can), gitea, and even gerrit (though it's extra work because of the ']}>' garbage) | 00:53 |
| mordred | corvus: I so don't understand the ]}> garbage in gerrit | 00:54 |
| clarkb | and confirmed that git log in /etc/puppet/modules/gerrit on review01 shows the latest commits | 00:54 |
| mordred | woot | 00:54 |
| *** wolverineav has quit IRC | 00:55 | |
| clarkb | as a heads up our build a wall contractors just dropped off portapotty so I think that may mean construction fun tomorrow. I should be able to run meeting and everything just fine though | 00:55 |
| clarkb | its all outdoor work and I'm not terribly worrid about anything out there | 00:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 00:57 |
| mordred | corvus: that's going to work this time. I swear | 00:58 |
| *** yamamoto has joined #openstack-infra | 01:15 | |
| *** eernst has joined #openstack-infra | 01:17 | |
| *** whoami-rajat has joined #openstack-infra | 01:20 | |
| *** yamamoto has quit IRC | 01:21 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update kuryr-kubernetes irc meeting time https://review.openstack.org/630689 | 01:21 |
| *** sthussey has quit IRC | 01:22 | |
| *** rkukura has quit IRC | 01:23 | |
| *** markvoelker has joined #openstack-infra | 01:35 | |
| *** wolverineav has joined #openstack-infra | 01:35 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Work in Progress https://review.openstack.org/629688 | 01:37 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 corrections https://review.openstack.org/630809 | 01:37 |
| *** wolverin_ has joined #openstack-infra | 01:38 | |
| *** wolverineav has quit IRC | 01:39 | |
| *** markvoelker has quit IRC | 01:40 | |
| *** wolverin_ has quit IRC | 01:43 | |
| *** smarcet has joined #openstack-infra | 01:52 | |
| *** hongbin has joined #openstack-infra | 01:57 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 01:58 |
| *** hongbin has quit IRC | 01:59 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 corrections https://review.openstack.org/630819 | 02:02 |
| *** eernst has quit IRC | 02:02 | |
| *** hongbin has joined #openstack-infra | 02:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Migrate OpenStackID dev server to php7 https://review.openstack.org/625640 | 02:11 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Migrate OpenStackID dev server to php7 https://review.openstack.org/625640 | 02:12 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver WIP https://review.openstack.org/629688 | 02:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix add_host typo https://review.openstack.org/630866 | 02:22 |
| openstackgerrit | Merged openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 02:30 |
| *** yamamoto has joined #openstack-infra | 02:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix missing safe_args for add_host https://review.openstack.org/630866 | 02:33 |
| *** yamamoto has quit IRC | 02:37 | |
| *** apetrich has quit IRC | 03:16 | |
| *** wolverineav has joined #openstack-infra | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:30 | |
| *** wolverineav has quit IRC | 03:32 | |
| *** ykarel|away has joined #openstack-infra | 03:42 | |
| *** diablo_rojo has quit IRC | 03:45 | |
| *** udesale has joined #openstack-infra | 04:05 | |
| *** udesale has quit IRC | 04:15 | |
| *** udesale has joined #openstack-infra | 04:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Delete change tags from docker image repos https://review.openstack.org/630782 | 04:22 |
| *** hongbin has quit IRC | 04:25 | |
| *** udesale has quit IRC | 04:26 | |
| *** yamamoto has quit IRC | 04:34 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 https://review.openstack.org/630875 | 04:35 |
| *** smarcet has quit IRC | 04:36 | |
| *** rh-jelabarre has quit IRC | 04:47 | |
| *** wolverineav has joined #openstack-infra | 04:51 | |
| *** wolverineav has quit IRC | 05:02 | |
| *** wolverineav has joined #openstack-infra | 05:03 | |
| *** roman_g has quit IRC | 05:16 | |
| *** kjackal has joined #openstack-infra | 05:20 | |
| *** kjackal has quit IRC | 05:38 | |
| *** chkumar|out is now known as chandankumar | 05:38 | |
| *** ykarel|away is now known as ykarel | 05:38 | |
| *** yamamoto has joined #openstack-infra | 05:42 | |
| *** Bhujay has joined #openstack-infra | 05:43 | |
| *** ykarel has quit IRC | 05:43 | |
| *** Bhujay has quit IRC | 05:44 | |
| *** Bhujay has joined #openstack-infra | 05:45 | |
| *** Bhujay has quit IRC | 05:46 | |
| *** Bhujay has joined #openstack-infra | 05:46 | |
| *** markvoelker has joined #openstack-infra | 05:48 | |
| *** yamamoto has quit IRC | 05:49 | |
| *** markvoelker has quit IRC | 05:59 | |
| *** ykarel has joined #openstack-infra | 06:00 | |
| *** udesale has joined #openstack-infra | 06:05 | |
| *** udesale has quit IRC | 06:12 | |
| *** udesale has joined #openstack-infra | 06:13 | |
| *** kjackal has joined #openstack-infra | 06:20 | |
| *** yboaron has joined #openstack-infra | 06:21 | |
| *** zhangfei has joined #openstack-infra | 06:24 | |
| *** wolverineav has quit IRC | 06:28 | |
| *** wolverineav has joined #openstack-infra | 06:29 | |
| *** kjackal has quit IRC | 06:32 | |
| *** wolverineav has quit IRC | 06:33 | |
| *** udesale has quit IRC | 06:33 | |
| *** udesale has joined #openstack-infra | 06:33 | |
| *** gfidente has joined #openstack-infra | 06:38 | |
| *** diablo_rojo has joined #openstack-infra | 06:46 | |
| *** rcernin has quit IRC | 06:59 | |
| *** Bhujay has quit IRC | 07:01 | |
| *** AJaeger has quit IRC | 07:03 | |
| *** roman_g has joined #openstack-infra | 07:04 | |
| *** AJaeger has joined #openstack-infra | 07:06 | |
| *** yamamoto has joined #openstack-infra | 07:08 | |
| *** Bhujay has joined #openstack-infra | 07:10 | |
| *** Bhujay has quit IRC | 07:11 | |
| *** Bhujay has joined #openstack-infra | 07:11 | |
| *** yamamoto has quit IRC | 07:13 | |
| *** jbadiapa has joined #openstack-infra | 07:15 | |
| *** pgaxatte has joined #openstack-infra | 07:29 | |
| *** e0ne has joined #openstack-infra | 07:30 | |
| *** kopecmartin|off is now known as kopecmartin | 07:35 | |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 07:35 |
| *** dpawlik has joined #openstack-infra | 07:36 | |
| spsurya | can anyone please review my affiliation patch https://review.openstack.org/#/c/629150/ clarkb fungi | 07:40 |
| spsurya | Thanks | 07:40 |
| *** zhangfei has quit IRC | 07:43 | |
| *** wolverineav has joined #openstack-infra | 07:43 | |
| *** rascasoft has joined #openstack-infra | 07:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] use bindep.txt with devstack https://review.openstack.org/630890 | 07:45 |
| *** wolverineav has quit IRC | 07:47 | |
| *** apetrich has joined #openstack-infra | 07:48 | |
| *** ccamacho has joined #openstack-infra | 07:56 | |
| *** rpittau has joined #openstack-infra | 08:03 | |
| *** e0ne has quit IRC | 08:04 | |
| *** ginopc has joined #openstack-infra | 08:09 | |
| *** gfidente has quit IRC | 08:12 | |
| *** panda|off is now known as panda | 08:19 | |
| *** ccamacho has quit IRC | 08:25 | |
| *** jpena|off is now known as jpena | 08:33 | |
| *** kjackal has joined #openstack-infra | 08:35 | |
| *** jpich has joined #openstack-infra | 08:38 | |
| *** yamamoto has joined #openstack-infra | 08:39 | |
| *** gfidente has joined #openstack-infra | 08:39 | |
| openstackgerrit | Merged openstack-infra/zuul master: Log connection exceptions when starting web https://review.openstack.org/630796 | 08:42 |
| *** tosky has joined #openstack-infra | 08:44 | |
| *** yamamoto has quit IRC | 08:44 | |
| *** ykarel is now known as ykarel|lunch | 08:55 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** xek has joined #openstack-infra | 09:00 | |
| *** ccamacho has joined #openstack-infra | 09:08 | |
| *** diablo_rojo has quit IRC | 09:26 | |
| *** pcaruana has joined #openstack-infra | 09:30 | |
| *** kjackal has quit IRC | 09:32 | |
| *** derekh has joined #openstack-infra | 09:33 | |
| *** wolverineav has joined #openstack-infra | 09:36 | |
| *** e0ne has joined #openstack-infra | 09:38 | |
| *** jungleboyj has quit IRC | 09:39 | |
| *** wolverineav has quit IRC | 09:41 | |
| *** jungleboyj has joined #openstack-infra | 09:42 | |
| *** evrardjp has joined #openstack-infra | 09:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 09:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 09:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add verify-status plugin to gerrit build https://review.openstack.org/630526 | 09:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add delete-project and importer plugins https://review.openstack.org/630527 | 09:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove trailing slash from googlesource url https://review.openstack.org/630919 | 09:51 |
| *** yboaron has quit IRC | 09:55 | |
| *** yboaron_ has joined #openstack-infra | 09:55 | |
| *** kjackal has joined #openstack-infra | 09:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use urljoin for git driver getGitUrl https://review.openstack.org/630920 | 10:03 |
| *** kjackal has quit IRC | 10:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use urljoin for git driver getGitUrl https://review.openstack.org/630920 | 10:05 |
| *** jaosorior has joined #openstack-infra | 10:09 | |
| *** yamamoto has joined #openstack-infra | 10:09 | |
| *** yamamoto has quit IRC | 10:12 | |
| *** yamamoto has joined #openstack-infra | 10:12 | |
| *** smarcet has joined #openstack-infra | 10:19 | |
| *** kjackal has joined #openstack-infra | 10:23 | |
| *** udesale has quit IRC | 10:25 | |
| *** smarcet has quit IRC | 10:30 | |
| *** chandankumar has quit IRC | 10:31 | |
| *** Qianbiao has joined #openstack-infra | 10:32 | |
| Qianbiao | Hello | 10:32 |
| Qianbiao | I am building a 3rd party CI | 10:33 |
| Qianbiao | Zuul is configed to watch openstack/sandbox-ci project. | 10:34 |
| *** arxcruz|ruck has quit IRC | 10:34 | |
| Qianbiao | But zuul got an error when receive a new patch | 10:34 |
| Qianbiao | GitCommandError: Cmd('git') failed due to: exit code(128) | 10:34 |
| Qianbiao | cmdline: git fetch --tags -v origin | 10:34 |
| Qianbiao | stderr: 'fatal: Could not read from remote repository. | 10:34 |
| Qianbiao | anyone could help me with this | 10:34 |
| *** udesale has joined #openstack-infra | 10:35 | |
| *** chandan_kumar has joined #openstack-infra | 10:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: properly format error exception https://review.openstack.org/630928 | 10:38 |
| *** lpetrut has joined #openstack-infra | 10:39 | |
| *** jtomasek has quit IRC | 10:40 | |
| *** jtomasek has joined #openstack-infra | 10:40 | |
| *** ykarel|lunch is now known as ykarel | 10:42 | |
| *** dkehn has quit IRC | 10:45 | |
| *** electrofelix has joined #openstack-infra | 10:48 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove trailing slash from googlesource url https://review.openstack.org/630919 | 10:52 |
| *** kjackal has quit IRC | 10:53 | |
| *** apetrich has quit IRC | 10:54 | |
| *** smarcet has joined #openstack-infra | 11:00 | |
| *** udesale has quit IRC | 11:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-virtual-baremetal to openstack https://review.openstack.org/620613 | 11:02 |
| *** arxcruz has joined #openstack-infra | 11:05 | |
| *** apetrich has joined #openstack-infra | 11:07 | |
| *** roman_g has quit IRC | 11:12 | |
| *** openstackgerrit has quit IRC | 11:22 | |
| *** openstackgerrit has joined #openstack-infra | 11:23 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added application credential to cloud assets config to get more granular control over scopes https://review.openstack.org/630937 | 11:23 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added application credential to cloud assets config to get more granular control over scopes https://review.openstack.org/630937 | 11:24 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Sahara plugins: enable translations https://review.openstack.org/630938 | 11:26 |
| *** udesale has joined #openstack-infra | 11:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: runner: add execute sub-command https://review.openstack.org/630944 | 11:39 |
| *** apetrich has quit IRC | 11:44 | |
| *** martinkennelly has joined #openstack-infra | 11:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use os.path.join for git driver getGitUrl https://review.openstack.org/630920 | 11:52 |
| *** kjackal has joined #openstack-infra | 11:54 | |
| *** rpittau is now known as rpittau|lunch | 11:59 | |
| *** apetrich has joined #openstack-infra | 12:01 | |
| *** kjackal has quit IRC | 12:02 | |
| *** rpittau has joined #openstack-infra | 12:02 | |
| *** jpena is now known as jpena|lunch | 12:03 | |
| *** kjackal has joined #openstack-infra | 12:03 | |
| *** rpittau is now known as rpittau|lunch | 12:05 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 12:06 |
| *** raissa has joined #openstack-infra | 12:10 | |
| *** rpittau|lunch has quit IRC | 12:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: runner: add execute sub-command https://review.openstack.org/630944 | 12:14 |
| *** rpittau|lunch has joined #openstack-infra | 12:15 | |
| *** smarcet has quit IRC | 12:20 | |
| *** jcoufal has joined #openstack-infra | 12:27 | |
| *** rpittau|lunch has quit IRC | 12:29 | |
| *** rpittau|lunch has joined #openstack-infra | 12:29 | |
| *** rpittau|lunch has quit IRC | 12:39 | |
| *** zul has joined #openstack-infra | 12:44 | |
| *** panda is now known as panda|lunch | 12:47 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul-jobs master: WIP: Default private_ipv4 to use public_ipv4 address when null https://review.openstack.org/623294 | 12:50 |
| *** gyee has joined #openstack-infra | 12:53 | |
| *** chandan_kumar has quit IRC | 12:58 | |
| *** chandan_kumar has joined #openstack-infra | 12:59 | |
| *** jpena|lunch is now known as jpena | 12:59 | |
| *** smarcet has joined #openstack-infra | 12:59 | |
| *** kjackal has quit IRC | 13:02 | |
| *** rpittau has joined #openstack-infra | 13:03 | |
| *** wolverineav has joined #openstack-infra | 13:12 | |
| *** rh-jelabarre has joined #openstack-infra | 13:15 | |
| *** wolverineav has quit IRC | 13:16 | |
| *** rlandy has joined #openstack-infra | 13:19 | |
| *** arxcruz is now known as arxcruz|ruck | 13:21 | |
| *** eharney has joined #openstack-infra | 13:24 | |
| *** smarcet has quit IRC | 13:25 | |
| *** dave-mccowan has joined #openstack-infra | 13:28 | |
| *** trown|outtypewww is now known as trown | 13:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 13:29 |
| *** gfidente has quit IRC | 13:30 | |
| *** boden has joined #openstack-infra | 13:30 | |
| *** sthussey has joined #openstack-infra | 13:32 | |
| *** gyee has quit IRC | 13:34 | |
| *** auristor has quit IRC | 13:34 | |
| *** kgiusti has joined #openstack-infra | 13:36 | |
| *** udesale has quit IRC | 13:36 | |
| *** udesale has joined #openstack-infra | 13:36 | |
| elod | hi, I'm looking at some failing stable periodic jobs and there is an interesting thing: they fail at "Run tox without tests" and it seems that wrong version of neutron is used however the good version / branch is cloned to workspace (e.g. on stable/rocky the neutron version is from master) | 13:37 |
| *** auristor has joined #openstack-infra | 13:37 | |
| elod | e.g.: http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/neutron-fwaas/stable/rocky/openstack-tox-py27/c916191/job-output.txt.gz | 13:37 |
| elod | anyone any idea how could this happen? | 13:38 |
| *** boden has quit IRC | 13:39 | |
| elod | (3 stable jobs fail on this on stable/rocky: neutron-fwaas, networking-odl, neutron-vpnaas) | 13:39 |
| *** boden has joined #openstack-infra | 13:39 | |
| *** Qianbiao has quit IRC | 13:41 | |
| *** kjackal has joined #openstack-infra | 13:42 | |
| *** panda|lunch is now known as panda | 13:43 | |
| mordred | elod: it looks like zuul is set to checkout stable/rocky of neutron: http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/neutron-fwaas/stable/rocky/openstack-tox-py27/c916191/job-output.txt.gz#_2019-01-15_06_14_10_607838 | 13:43 |
| mordred | still looking though | 13:43 |
| elod | thanks! it seems that somehow pip installs neutron from pypi, but why doesn't it use the cache? | 13:46 |
| mordred | elod: yes - this is how it works to make depends-on and whatnot work right - first we install requirements using pip from requirements.txt - then inspect the virtualenv, see what packages are there and what packages are provided by the git repos that have been put on the host - then we reinstall anything that matches from git | 13:48 |
| fungi | does the job definition need vars: tox_install_siblings: true | 13:49 |
| fungi | i was trying to trace through the inheritance path and not finding it | 13:49 |
| mordred | hrm. that _should_ be on by default, no? | 13:49 |
| mordred | oh - no, duh. it would need that to install siblings for these jobs, eah | 13:50 |
| fungi | https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n364 doesn't seem to set it, parented to https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n343 which also doesn't | 13:51 |
| ttx | Looking at what it would take to define a specific Zuul tenant for kata-containers... IIUC beyond specifying the tenant in project-config's main.yaml, I'd need a kata-specific repo to hold the Zuul configuration for things like pipelines ? | 13:51 |
| fungi | ttx: yes, that way their pipeline configurations can ultimately have different reviewers than openstack's | 13:52 |
| mordred | yeah. we'll also probably want to talk about how to manage base jobs in such a scenario - because I actually don't think we'd want separate base jobs | 13:52 |
| fungi | is that how, e.g., softwarefactory is doing it? | 13:53 |
| mordred | or, rather, separate definitions is fine - but most of what's in the base jobs are deployment specific not tenant specific - so we woudln't want to manage the base jobs twice | 13:53 |
| ttx | So we need to create a new repository first... We'd want it in Gerrit since it's a trusted one... suggestions on name/location ? openstack-infra/kata-zuul-config ? | 13:54 |
| mordred | which means maybe we want an opendev-base-jobs repo or something that can have the base jobs and be attached to each tenant | 13:54 |
| ttx | mordred: yes it feels like you'd ultimately need to separate templates/jobs definition from pure configuration (pipelines, which jobs for each project etc) | 13:55 |
| mordred | yeah. | 13:56 |
| ttx | and use different repos for each so that templates/jobs definitions can be loaded by multiple tenants | 13:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix missing safe_args for add_host https://review.openstack.org/630866 | 13:56 |
| mordred | ttx: I think that sounds like a fine name - when more of infra-root are awake we can chat about the approach for base jobs | 13:56 |
| ttx | ok I'll propose something up | 13:58 |
| pabelanger | fungi: no, sf.io each tenant has their own base job. I do want to fix that to use a single base, and talking to tobiash it should be possible. | 13:58 |
| pabelanger | but, doing base-minimal does make things easier to manage | 13:58 |
| fungi | mordred: elod: i guess tox_install_siblings is true by default in the tox role, so that's probably not the issue: https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox/defaults/main.yaml#n6 | 13:59 |
| elod | mordred: i tried to compare the things with networking-bgpvpn stable/rocky job: the installed neutron version is from stable/rocky on its job | 14:01 |
| elod | http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/networking-bgpvpn/stable/rocky/openstack-tox-py27/1d24b01/job-output.txt.gz | 14:01 |
| ssbarnea|rover | fungi: do we know if a build was a recheck? mainly I am curious if we can measure the percentage of rechecks, maybe with logstash, e-r,...? | 14:01 |
| fungi | elod: in that case we probably want to compare the tox-specific logs from each job to see what pip is being asked to install at each stage | 14:02 |
| *** kjackal has quit IRC | 14:02 | |
| fungi | ssbarnea|rover: we could probably scrape gerrit comments with a change query to count them | 14:02 |
| fungi | wouldn't be 100% accurate, but likely close (sometimes people add recheck comments when the change is already being tested, or when it's untestable) | 14:03 |
| ssbarnea|rover | fungi: it could be something quite valuable if we can measure this overtime. | 14:03 |
| elod | fungi, mordred: one difference maybe what i see now is that bgpvpn uses ".tox/py27/bin/pip install" and neutron-fwaas uses ".tox/py27/bin/python -m pip install"... but i don't know whether it's relevant | 14:03 |
| elod | and i don't find where this difference comes from | 14:04 |
| ssbarnea|rover | fungi: is true, but overall it would be a relatively decent way to measure the sanity of infrastructure. i assume that most rechecks happen when users think that another run will work. | 14:05 |
| ssbarnea|rover | if that is true or not is less important, but knowing that a single change needs ~x retries on average is valuable, especially as a trend. | 14:06 |
| mordred | elod: it shouldn't matter ... | 14:06 |
| *** efried1 has joined #openstack-infra | 14:06 | |
| *** apetrich has quit IRC | 14:07 | |
| mordred | elod, fungi: my gut tells me this is going to be a sequencing thing - and that somewhere in teh sequences something is getting installed in an order that is unhappy | 14:07 |
| *** roman_g has joined #openstack-infra | 14:10 | |
| *** efried has quit IRC | 14:10 | |
| *** efried1 is now known as efried | 14:10 | |
| *** mriedem has joined #openstack-infra | 14:10 | |
| *** apetrich has joined #openstack-infra | 14:12 | |
| mordred | fungi, pabelanger: we need to restart the mergers and executors to pick up the config fix ... we don't seem to have a playbook for that though - is there any already-existing thing we can use for that? | 14:13 |
| pabelanger | mordred: http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/zuul_restart.yaml with using --limit would be my suggestion | 14:14 |
| mordred | pabelanger: ah - good point | 14:15 |
| *** psachin has joined #openstack-infra | 14:15 | |
| mordred | pabelanger: --limit='zm*:ze*' should work as a limit clause, right? | 14:15 |
| pabelanger | yah | 14:16 |
| pabelanger | or limit zuul-executor:zuul-merger, if that is the group | 14:16 |
| *** gfidente has joined #openstack-infra | 14:16 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/infra-manual master: Improve the details about settings i18n for new repous https://review.openstack.org/630981 | 14:16 |
| mordred | ah right. groups. yay | 14:16 |
| mordred | pabelanger, fungi: there's nothing in post or release - any issues with me restarting the mergers and executors? | 14:17 |
| mordred | with: root@bridge:/opt/system-config# ansible-playbook --limit 'zuul-executor:zuul-merger' playbooks/zuul_restart.yaml | 14:18 |
| mordred | oh - I guess also with -f 20 | 14:19 |
| mordred | root@bridge:/opt/system-config# ansible-playbook -f 20 --limit 'zuul-executor:zuul-merger' playbooks/zuul_restart.yaml | 14:19 |
| pabelanger | agree, release / tag pipelines look clear | 14:19 |
| pabelanger | +1 for command too | 14:19 |
| * mordred waits just a few more to give fungi a chance to object | 14:19 | |
| mordred | ok. seeing no dissention - I'm going to restart things | 14:20 |
| fungi | sorry, had stepped away for a sec. catching up | 14:20 |
| * mordred waits | 14:20 | |
| fungi | yeah, no objection to a executor/merger restart | 14:22 |
| mordred | fungi: (the motivation is that we landed a config change to add git driver repos yesterday and the baseurl had a trailing slash - which caused zuul mergers to hang trying to clone it indefinitely) | 14:22 |
| mordred | sweet | 14:22 |
| * mordred runs command | 14:22 | |
| fungi | my "limited" grasp of ansible suggests that --limit='zm*:ze*' should be fine | 14:22 |
| elod | mordred, fungi : besides that maybe a sequencing problem, i see one more diff now: on the failing jobs' requirements.txt the req is neutron>=13.0.0.0b2 , however on e.g. bgpvpn it is 13.0.0 ... could it be that pip handles something wrongly? | 14:23 |
| fungi | ssbarnea|rover: well, "how many rechecks before a patchset passes all its jobs" is likely to be much different from "how many job runs are triggered by rechecks" | 14:23 |
| mordred | elod: >= 13.0.0.0b2 will tell pip to install any pre-release version of neutron found, while 13.0.0 will ignore pre-releases | 14:23 |
| fungi | ssbarnea|rover: people have a tendency to repeatedly recheck their changes and only look at the logs to discover they've got an error in their code after repeated rechecks fail the same way | 14:24 |
| mordred | elod: maybe it's time to change fwaas to >= 13.0.0 ? | 14:24 |
| elod | mordred: yes, i was about to try that, thanks :) | 14:24 |
| mordred | yeah - that'll be fwaas picking up 14.0.0b1 | 14:25 |
| mordred | elod: I think fixing that will fix uyou | 14:25 |
| elod | mordred: according what you sad i think so. nevertheless, i think the problem will come up when there will 14.0.0 version of neutron that is not a pre-release, maybe? | 14:26 |
| mordred | elod: yes - possibly - we might need to think about making sure requirement ranges don't conflict | 14:27 |
| elod | mordred: ok | 14:29 |
| elod | mordred, fungi , thanks for the help | 14:29 |
| mordred | elod: sure thing! | 14:30 |
| *** markmcd has quit IRC | 14:30 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Define a specific Zuul tenant for Kata containers https://review.openstack.org/630994 | 14:31 |
| ttx | mordred, fungi: here is my try at adding the repo ^^ might be wrong as there is a non-trivial amount of chicken and egg involved (adding a repository requires adding your repo to a Zuul tenant) | 14:31 |
| mordred | fungi: wow. the executors are still trying to stop | 14:31 |
| ttx | at the very minimum that gives us a place to discuss that change :) | 14:32 |
| *** markmcd has joined #openstack-infra | 14:35 | |
| mordred | #status log restarted zuul executors and mergers to pick up git connection config fix | 14:39 |
| openstackstatus | mordred: finished logging | 14:39 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 14:40 |
| mordred | poo. I think we need a scheduler restart too :( | 14:43 |
| openstackgerrit | Graham Whaley proposed openstack-infra/project-config master: kata: add dco-license jobs to kata projects https://review.openstack.org/630998 | 14:44 |
| fungi | we can do those too. activity levels aren't super high right now anyway | 14:44 |
| mordred | fungi: actually - lemme see something... | 14:45 |
| *** rossella_s has quit IRC | 14:46 | |
| *** kjackal has joined #openstack-infra | 14:46 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Sahara plugins: enable translations https://review.openstack.org/630938 | 14:51 |
| *** rossella_s has joined #openstack-infra | 14:59 | |
| *** Bhujay has quit IRC | 15:02 | |
| *** ykarel has quit IRC | 15:04 | |
| *** kjackal has quit IRC | 15:06 | |
| *** quiquell is now known as quiquell|off | 15:07 | |
| ssbarnea|rover | fungi: mordred : do we have a policy/guideline regarding using or not-using github mirrors? | 15:09 |
| mordred | ssbarnea|rover: what do you mean? like - about someone wanting to opt-out of them? | 15:10 |
| fungi | yeah, i can't tell if you're referring to our process of mirroring content to github, or if you're referring to mirroring content from github into our nodepool regions | 15:11 |
| mordred | yeah. me either | 15:12 |
| fungi | (or something else altogether) | 15:13 |
| mordred | fungi: I think ssbarnea|rover doesn't like us anymore :) | 15:16 |
| mordred | fungi: ok. we don't need to restart the schedule | 15:16 |
| mordred | fungi: ok. we don't need to restart the scheduler | 15:16 |
| fungi | can i at least restart my schedule? i don't like the way my day is looking | 15:17 |
| fungi | good news about zuul though, thanks! | 15:17 |
| *** yboaron_ has quit IRC | 15:18 | |
| ssbarnea|rover | fungi: mordred : i asked because i got some comments about not using github as not being reliable and I didn't really consider this a true. but if we would have a guideline about the subject it would much easier to follow it. | 15:18 |
| mordred | ssbarnea|rover: ah - so you mean in jobs | 15:18 |
| mordred | ssbarnea|rover: so - generally speaking, if a repo exists in our git farm (like openstack repos) you should NEVER use github for it | 15:18 |
| fungi | github is unreliable | 15:19 |
| ssbarnea|rover | ok thanks. | 15:19 |
| mordred | ssbarnea|rover: if it only exists on github, like it's an external dependency that gets pulled from source because it doesn't have another system of being gotten | 15:19 |
| mordred | then there's not much of a choice | 15:19 |
| fungi | historically, statistics have backed that up for us | 15:19 |
| mordred | yah. very much so | 15:19 |
| fungi | it's not the day-long 100% github outages which really bite us so much as the background % failure rate from random machines in their server farm not getting commits copied around fast enough or having other random problems | 15:21 |
| mordred | ssbarnea|rover: of course, if a repo exists in our git farm, jobs should be configured to have zuul clone them anyway - so github really shouldn't be coming up for those repos anyway | 15:21 |
| clarkb | ssbarnea|rover: re rechecks there was a researcher looking into that according to a thread on the infra mailing list. You may want to follow up with them to see if they have any tooling they can share. That said I disagree that we can assume infra problems if a change is rechecked. People dont diagnose failures beyond it failed so Im trying again and to infer more is inaccurate | 15:21 |
| mordred | corvus: we have hit an actual issue using zuul to clone the gerrit repo for us using required-projects | 15:22 |
| clarkb | This is one of the reasons elastic recheck exists | 15:22 |
| mordred | corvus: because zuul doesn't do submodule update --init on the repo, and because there is no working origin on the build node ... | 15:22 |
| clarkb | we found the self identified recheck reasons to be highly inaccurate | 15:22 |
| mordred | corvus: the job cannot itself do the submodule update --init | 15:22 |
| mordred | corvus: http://logs.openstack.org/68/630468/20/check/system-config-build-image-gerrit/1cd5f0b/job-output.txt.gz#_2019-01-15_15_16_50_418841 | 15:23 |
| *** ykarel has joined #openstack-infra | 15:23 | |
| *** efried has quit IRC | 15:24 | |
| *** efried has joined #openstack-infra | 15:25 | |
| *** kjackal has joined #openstack-infra | 15:26 | |
| clarkb | mordred: the push from executor to test node wont push submodules either right? | 15:26 |
| clarkb | is that a pull only action? | 15:26 |
| mordred | clarkb: I don't know the answer to that | 15:28 |
| mordred | clarkb: I think it's complicated enough that for now we should go back to having the job do the clone | 15:28 |
| mordred | clarkb: git config push.recurseSubmodules on-demand | 15:31 |
| mordred | clarkb: will configure git to push submodules content when it pushes other things | 15:32 |
| clarkb | oh neat, so if the merger/executor configured submodules then the later push would work | 15:33 |
| clarkb | assuming we set that flag | 15:33 |
| fungi | clarkb: ssbarnea|rover: yeah, we used to require commenters to include a "reason" in every recheck comment and the infra/qa teams tried to mine that for useful information, but ultimately the "reasons" given were next to useless. and when we started requiring recheck comments to include a bug number, teams would open a "recheck bug" and refer to it in almost every recheck to save time, or people | 15:34 |
| fungi | would copy/paste other bug numbers they saw someone use elsewhere, or would at times even just make up bug numbers at random | 15:34 |
| mordred | clarkb: yah - in theory. although I think we'd also have to make mirror-workspace-git-repos know how to remove the remote settings from the submodules - and also have configure-workspace-git do a --recursive when it clones from the cache | 15:36 |
| mordred | clarkb: there's going to be a pile of fiddly things here | 15:36 |
| mordred | oh - I suppose the other thing we coudl do ... | 15:36 |
| mordred | it put ALL of the plugins repos that are in gerrit's submodules stack into the list | 15:36 |
| *** efried has quit IRC | 15:36 | |
| clarkb | mordred: everyone knows you buy into that (fiddly things that don't work) when you use submodules right? | 15:36 |
| clarkb | I guess gerrit didn't get the memo :) | 15:36 |
| *** efried has joined #openstack-infra | 15:37 | |
| fungi | ssbarnea|rover: many developers just seem to want to avoid looking at failure logs, so if there's a slim chance the job failed for reasons unrelated to their change they'd rather just keep rechecking it until they get a passing grade (and we've seen changes with literally dozens of rechecks where the failure was actually due to a bug in the change, or changes introducing nondeterministic failures | 15:37 |
| fungi | where the author and reviewers rechecked the change repeatedly until they lucked out and got that bug to merge) | 15:37 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: kata: add dco-license jobs to kata projects https://review.openstack.org/630998 | 15:37 |
| mordred | ignore the submodules in the gerrit repo completely and have the job overwrite the dirs in the gerrit plugins dir with the content from the zuul repos | 15:37 |
| mordred | clarkb, corvus: ^^ I think if we did that (which is basically what we're doing for the things that don't have git submodules in the gerrit tree) - it would be easier in the short term - and probably more resilient in the long term, since they use gerrit submodule tracking anyway | 15:38 |
| mordred | so we know the stable-2.15 branch of each plugin repo is going to be what would be in the submodules of the gerrit repo | 15:38 |
| fungi | mordred: clarkb: submodule init has had some nasty security vulnerabilities in the past. if we decide to do that, would zuul perform those in the context of the bwrap sandbox? | 15:39 |
| clarkb | fungi: I don't think the mergers currently run in that context | 15:39 |
| *** efried has quit IRC | 15:40 | |
| fungi | thinking of situations like https://public-inbox.org/git/xmqqy3bcuy3l.fsf@gitster-ct.c.googlers.com/T/#u | 15:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add in-tree gerrit submodules to googlesource https://review.openstack.org/631007 | 15:41 |
| mordred | fungi: yah - I think I'd prefer to just do this ^^ | 15:41 |
| *** kjackal has quit IRC | 15:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 15:47 |
| mordred | and do that ^^ to go with it | 15:47 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support cracklib in pam for Gentoo's musl profile https://review.openstack.org/631009 | 15:51 |
| *** efried has joined #openstack-infra | 15:53 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 15:57 |
| openstackgerrit | Hervé Beraud proposed openstack-dev/pbr master: Resolve ``ValueError`` when mapping value contains a literal ``=``. https://review.openstack.org/625372 | 15:58 |
| fungi | heading out for a lunch appointment, but should be back by 18:00z at the latest | 16:00 |
| clarkb | mriedem: https://review.openstack.org/#/c/487664/ will only affect legacy jobs and I wanted to double check that the tests you want that data from are legacy jobs. Do you have more context for where we need to collect that log info (eg what jobs)? | 16:03 |
| *** e0ne has quit IRC | 16:11 | |
| mriedem | when i wrote that we only had legacy jobs | 16:14 |
| *** eharney has quit IRC | 16:14 | |
| *** ccamacho has quit IRC | 16:16 | |
| mriedem | clarkb: grenade is still legacy fwiw http://logs.openstack.org/64/487664/4/check/grenade-py3/cf44ea0/logs/rabbitmq/cleanup-host-report.txt.txt.gz | 16:16 |
| clarkb | mriedem: ya frickler was going to work on changing that iirc. So maybe we want that change in d-g and in the devstack zuul ansible roles? | 16:16 |
| corvus | ttx, pabelanger, mordred: regarding the tenant for kata containers.... i thought we were waiting until we moved zuul.openstack.org to opendev, because otherwise we have no url to host the multi-tenant dashboard.... | 16:21 |
| corvus | at least... that's what people said to me when i started talking about a zuul tenant. | 16:21 |
| mordred | corvus: ah yes - I knew there was something else we were blocking on | 16:21 |
| AJaeger | clarkb: could you review https://review.openstack.org/630994 as PTL , please? | 16:22 |
| AJaeger | clarkb: oh, I see corvus mention above - corvus, so want to WIP 630994? | 16:22 |
| *** ykarel is now known as ykarel|away | 16:23 | |
| corvus | AJaeger: well, we at least need to confirm we have consensus. if, instead, we just wanted to bust open zuul.openstack.org as a multi-tenant host for now (https://zuul.openstack.org/t/kata-containers, https://zuul.openstack.org/t/openstack, https://zuul.openstack.org/t/zuul), i'd be ok with that, temporarily, if others are. it just seems a little rude to openstack. | 16:25 |
| corvus | what would be *great* if someone new could jump in and help with the work to move zuul to opendev. | 16:26 |
| mordred | corvus++ | 16:26 |
| corvus | it's on my list, but i'm spending 90% of my time working on moving gerrit to opendev, and it's lined up behind that. | 16:26 |
| clarkb | mordred: AJaeger is http://logs.openstack.org/0d/0d75b259c065bb3d8e3acc45ce85ec767ac29db1/post/gerrit-package-buck/f2bb979/ara-report/ failing due to a bug in the new log collection work? | 16:28 |
| clarkb | that failure means i don't have a published war to test on review-dev01 | 16:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Add in-tree gerrit submodules to googlesource https://review.openstack.org/631007 | 16:28 |
| mordred | clarkb: yes. hrm. | 16:29 |
| clarkb | corvus: considering it is relatively straightforward to add a vhost to zuul.openstack.org for zuul.opendev.org that is probably preferable. One gotcha is ssl certs, but maybe we just buy that one for now | 16:30 |
| mordred | clarkb: well - maybe - but maybe not | 16:30 |
| clarkb | mordred: re artifacts or zuul.opendev.org? | 16:31 |
| mordred | clarkb: artifacts | 16:31 |
| *** eernst has joined #openstack-infra | 16:31 | |
| mordred | clarkb: the move artifacts dir logic is only supposed to run if zuul.change is defined, and I don't see it having been run in that ara either | 16:32 |
| corvus | clarkb: i left a comment on 630994 you can respond to | 16:32 |
| mordred | clarkb: so I think there is a bug somewhere, but I don't think it's with that role | 16:32 |
| clarkb | mordred: note the host is tarballs.o.o I think that is the bug | 16:32 |
| mordred | clarkb: no - it's not finding the local dir | 16:33 |
| mordred | "/var/lib/zuul/builds/f2bb9792b21148eba7e9ecc166b28a01/work/artifacts" failed: No such file or directory (2) | 16:33 |
| clarkb | mordred: ya but it shouldn't run those tasks against tarballs.o.o | 16:34 |
| clarkb | hrm actually wait I may have confused myself | 16:34 |
| clarkb | this is the actual copy artifacts to tarballs step, not the make stuff happy on zuul.o.o executor | 16:35 |
| *** dklyle has joined #openstack-infra | 16:35 | |
| clarkb | and /var/lib/zuul/builds doesn't exist there because it needs to be /srv/tarballs/zuul or whatever | 16:35 |
| clarkb | mordred: maybe the new stuff is conflicting on the var name for the dest with the existing roles that publish stuff? | 16:35 |
| mordred | clarkb: remote: https://review.openstack.org/631017 Update the host spec for post jobs | 16:38 |
| mordred | clarkb: the problem is taht the post playbook didn't actually copy the artifacts back to the executor | 16:38 |
| clarkb | ya so src didn't exist | 16:38 |
| mordred | yah | 16:39 |
| clarkb | and it didn't copy because the node name is wrong? | 16:39 |
| mordred | yah | 16:39 |
| mordred | it didn't run the tasks at all | 16:39 |
| mordred | because there were no hosts matching | 16:39 |
| clarkb | mordred: can you push to openstack/2.13.12 ? | 16:39 |
| mordred | if you look in the ara, you'll see 0 tasks for post-publish | 16:39 |
| mordred | clarkb: oh - the branch is openstack/2.13.12? fun | 16:39 |
| mordred | yeah - one sec | 16:39 |
| clarkb | ya I would've had to force update 2.13 and I'm not sure that is what we want. But I think once we get to docker builds most of that mess goes away | 16:40 |
| clarkb | (at one time I had an idea that we would be able to do 2.13 longterm with updates but the zuul job maintenance makes that weird (as its more commits) | 16:40 |
| mordred | clarkb: done | 16:40 |
| mordred | remote: https://review.openstack.org/631019 Update the host spec for post jobs | 16:40 |
| corvus | clarkb, mordred, fungi: good news on the image front: https://hub.docker.com/r/opendevorg/gitea/tags looks like all the new delete stuff worked (there's one change tag still there, it wasn't old enough to be deleted at the time, it'll be deleted next run). all the other old stuff is gone, as well as the newest change tag used for the promote. | 16:41 |
| mordred | \o/ | 16:41 |
| clarkb | corvus: yay | 16:41 |
| corvus | mordred: are you pretty sure your build-arg thing is working now? | 16:42 |
| mordred | corvus: uhm. I mean - it worked locally | 16:43 |
| corvus | mordred: (the change itself isn't self-tested due to files matchers, and the followup is hung up on the additional gerrit repos) | 16:43 |
| corvus | mordred: so you've got the ansible/yaml/line-folding syntax, etc worked out? | 16:43 |
| openstackgerrit | Merged openstack-infra/zuul master: Use os.path.join for git driver getGitUrl https://review.openstack.org/630920 | 16:43 |
| *** hamzy_ has quit IRC | 16:43 | |
| mordred | corvus: oh - I mean - I think so - but given how fiddly that has been and I've got some jetlag - I think I'd rather see the gerrit patch work and do the right thing | 16:44 |
| corvus | mordred: okay i'm going to update your patch to force an image rebuild, since i'm now blocked on that in order to move the roles | 16:44 |
| mordred | corvus: we should be able to recheck once zuul reconfigures with the repo adds once they land, yeah? | 16:44 |
| mordred | ah - gotcha | 16:44 |
| mordred | yeah - please do - I'm 99% sure - but let's make 100% sure - jinja+yaml-folding == what | 16:45 |
| Shrews | corvus: mordred: I'm more than happy to help pitch in on the zuul move to opendev. I just don't know where to pitch in | 16:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 16:45 |
| corvus | Shrews: i think we're at step 0: "make a plan" :) | 16:46 |
| Shrews | corvus: oh, i thought you and mordred were already somewhere between step 100 and 117 :/ | 16:46 |
| clarkb | Shrews: I did some step -1 hand wavy planning in response to corvus' comment on that change | 16:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Allow known_hosts to be run in untrusted context https://review.openstack.org/631020 | 16:47 |
| clarkb | I think the simplest path forward that gets us the name sanity without redeploying everything is to add a vhost to zuul.openstack.org that listens to zuul.opendev.org | 16:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 16:48 |
| clarkb | and it won't rewrite with an assumed tenant like zuul.openstack.org's tenant | 16:48 |
| ttx | corvus: ah sorry I misread comments that this was ready to be supported. I proposed a temporary solution in comments at https://review.openstack.org/#/c/630994/ | 16:48 |
| *** smarcet has joined #openstack-infra | 16:48 | |
| clarkb | your logs will still be hosted at logs.openstack.org but other than that it should be pretty transparent | 16:48 |
| *** wolverineav has joined #openstack-infra | 16:49 | |
| *** rpittau has quit IRC | 16:49 | |
| corvus | i think clarkb's idea will probably work and probably won't significantly hinder the "real" move to opendev, so if we want to go with that, i have no objection. | 16:50 |
| ttx | clarkb: would that vhost solution still be "rude" as defined above? | 16:50 |
| ttx | corvus: we still need to refactor the base-jobs, no? | 16:51 |
| clarkb | ttx: no zuul.openstack.org will continue to exist for openstack under that plan so nothing changes for openstack | 16:51 |
| corvus | ttx: i don't think so -- by rude i meant changing the meaning of 'zuul.openstack.org' with no notice :) | 16:51 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 16:52 |
| *** pgaxatte has quit IRC | 16:52 | |
| *** wolverineav has quit IRC | 16:53 | |
| mordred | ttx: and no, we won't need to refactor base jobs - they existing ones will work fine | 16:53 |
| corvus | mordred: ? | 16:54 |
| ttx | mordred: ok, I'm a bit confused. | 16:54 |
| corvus | i think we should refactor the base job | 16:54 |
| corvus | currently they are in project-config, which has a bunch of openstack-specific jobs too | 16:55 |
| ttx | and has config things we can't import | 16:55 |
| corvus | we should pull them out into their own 'opendev' repo | 16:55 |
| mordred | oh - totally - I mean it's not a prereq for doing these kata jobs | 16:55 |
| *** yamamoto has quit IRC | 16:55 | |
| corvus | i think it is a pre-req before we add our second tenant | 16:55 |
| mordred | yes. it is a prereq for that | 16:55 |
| corvus | this job should not exist in the kata or zuul tenants: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n217 | 16:56 |
| mordred | I was trying to answer ttx's question as to whether refactoring the base jobs still needed to be done with clarkb's zuul.opendev.org suggestion | 16:56 |
| corvus | i think it does | 16:56 |
| corvus | clarkb's zuul.opendev.org suggestion is a way to provide a second tenant | 16:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 16:56 |
| mordred | corvus: gotcha. I was thinking it was in conjunction with ttx's 'just make a couple of kata-specific pipelines in the existing tenant for now' idea | 16:57 |
| * mordred is caught up | 16:57 | |
| corvus | mordred: ah, i think that's an exclusive alternative. agreed that if that happened refactoring is not needed. | 16:57 |
| mordred | corvus: yah. and I agree that with a second tenant refactoring is needed :) | 16:58 |
| ttx | corvus: would you be ok with temporarily defining kata-specific pipelines in the openstack tenant while we work on multi-tenant support ? That way they can start using them and we'll just move them over to the new tenant when that is ready | 16:58 |
| ttx | (currently they are reusing the openstack third-party-testing pipeline but they will need slightly different triggers | 16:58 |
| smarcet | fungi: afternoon when u have some chance could you take a look at https://review.openstack.org/#/c/625640/, its failing on infra-puppet-apply-4-ubuntu-xenial but from log i cant figure what its wrong, its complaning about a permission denied for a rm command but i dont know why :) | 17:00 |
| pabelanger | re: opendev zuul, I'd be happy to do some of that work in the evening. need to read up on the spec again | 17:00 |
| clarkb | ttx: any idea what type of trigger changes they will need? | 17:00 |
| ttx | clarkb: they need a label GitHub trigger for the WIP check | 17:00 |
| ttx | clarkb: and bonus points to have more kata-specific names and messages | 17:01 |
| ttx | which will smooth the adoption curve | 17:01 |
| clarkb | smarcet: http://logs.openstack.org/40/625640/5/check/infra-puppet-apply-4-ubuntu-xenial/771ccee/applytest/puppetapplytest44.final.out.FAILED that shows the actual error. The rm is just noise I think | 17:01 |
| pabelanger | ttx: what is a WIP check? how is that different then check pipeline today? I guess way to limit specific jobs running | 17:02 |
| smarcet | clarkb: oh ok thx u for pointing that out | 17:02 |
| ttx | pabelanger: the WIP check would block the PR from merging if it has a do-not-merge label on | 17:02 |
| bnemec | Can someone add me to the core list for OVB: https://review.openstack.org/#/admin/groups/1993,members ? | 17:03 |
| bnemec | ^cc jaosorior | 17:03 |
| jaosorior | bnemec: I don't have permissions for that./ | 17:03 |
| *** ginopc has quit IRC | 17:03 | |
| corvus | ttx: i have two concerns with that: i don't want to place an undue burden or expectation on the folks who have volunteered to help maintain openstack's system. and two: no one has re-integrated the kata mailing list yet, so this will add to the backlog of "kata exceptional situations that need to be normalized" | 17:03 |
| bnemec | jaosorior: Yeah, I know. That was more of an FYI as the owning PTL. | 17:03 |
| bnemec | Not sure if you need to sign off on it. | 17:04 |
| corvus | ttx: i think the new tenant for kata will be a good experience for everyone. | 17:04 |
| *** udesale has quit IRC | 17:04 | |
| clarkb | bnemec: usually we add the PTL then they can add whoever they like | 17:04 |
| clarkb | bnemec: I don't see a change in governance to make it officially tripleo though? | 17:04 |
| jaosorior | clarkb: there is nobody added to that list, it seems. | 17:04 |
| bnemec | Oh, hmm. | 17:04 |
| bnemec | ssbarnea|rover: ^ | 17:04 |
| clarkb | jaosorior: yes we have to seed the initial user, I'll add you momentarily | 17:05 |
| bnemec | I think there was supposed to be a governance change. | 17:05 |
| clarkb | jaosorior: done, you can add bnemec or anyone else now | 17:05 |
| clarkb | bnemec: I may also be blind | 17:05 |
| jaosorior | thanks clarkb | 17:05 |
| bnemec | clarkb: I don't recall seeing the governance change either, so I suspect you're right. | 17:06 |
| AJaeger | bnemec, jaosorior : can either of you push up a governance change, please? | 17:06 |
| AJaeger | clarkb: argh, missed it this time ;( | 17:06 |
| clarkb | AJaeger: ya me too | 17:06 |
| clarkb | thats ok I expect bnemec will get it sorted shortly | 17:07 |
| * bnemec may have set expectations too high ;-) | 17:07 | |
| *** roman_g has quit IRC | 17:07 | |
| ttx | pabelanger: to solve your race on import, should I split the change with the gerrit/ changes first, then the zuul/ part ? Or is that more subtle? | 17:09 |
| *** jpich has quit IRC | 17:09 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 17:12 |
| *** eharney has joined #openstack-infra | 17:13 | |
| *** e0ne has joined #openstack-infra | 17:13 | |
| *** ekultails has joined #openstack-infra | 17:13 | |
| pabelanger | ttx: yah, I believe landing the repos first, then zuul changes is the order | 17:13 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 17:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 17:22 |
| *** derekh has quit IRC | 17:22 | |
| mordred | corvus: ^^ rebased on your update to the change - the repos are in the main.yaml now, so hopefully this time it passes | 17:23 |
| mordred | corvus: build-arg passed, so it at least works with no build-args specified :) | 17:23 |
| corvus | mordred: oh, gah | 17:25 |
| corvus | mordred: you never fixed those 2 changes did you? :) | 17:25 |
| clarkb | mordred: fwiw gerrit publishing still failing, but progress, http://logs.openstack.org/72/72550fdd77a16c275710b9b0eb5cf03d3f04c2c0/post/gerrit-package-buck/e3086de/ara-report/ | 17:25 |
| corvus | mordred: the build-arg change is still split across both | 17:25 |
| corvus | mordred: i'm going to fix it good. | 17:26 |
| mordred | corvus: crappit. sorry | 17:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 17:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 17:28 |
| *** dtantsur is now known as dtantsur|afk | 17:28 | |
| *** e0ne has quit IRC | 17:28 | |
| corvus | mordred, clarkb, pabelanger, ttx: what should we call the new base job repo? opendev/base-jobs? opendev/site-base-jobs? something else? | 17:28 |
| mordred | corvus: I like opendev/base-jobs | 17:29 |
| clarkb | corvus: I agree with mordred opendev/base-jobs sgtm | 17:30 |
| clarkb | I see the issue in buck publishing now to make sure my fix is correct | 17:30 |
| mordred | clarkb: I believe in you | 17:30 |
| pabelanger | opendev/base-jobs wfm | 17:30 |
| pabelanger | corvus: that will be a config-project right? | 17:31 |
| corvus | pabelanger: yes, in every tenant | 17:31 |
| pabelanger | kk | 17:31 |
| corvus | we could also do something like "opendev/zuul-tenant-config" | 17:31 |
| pabelanger | I actually like site-base-jobs more in that case, if all tenants will use it. zuul-tenant-config also works | 17:32 |
| pabelanger | site-variables, site-base-jobs, etc | 17:32 |
| ttx | base-jobs ++ | 17:32 |
| ttx | ah, though pabelanger makes good point | 17:33 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Define a specific Zuul tenant for Kata containers https://review.openstack.org/630994 | 17:33 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Add Kata tenant Zuul configuration repository https://review.openstack.org/631032 | 17:33 |
| ttx | just split into two patchsets ^ | 17:33 |
| ttx | will WIP the second one waiting for base-jobs | 17:34 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 https://review.openstack.org/630875 | 17:35 |
| *** dtantsur|afk has quit IRC | 17:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add opendev/base-jobs repo https://review.openstack.org/631033 | 17:36 |
| *** ykarel has joined #openstack-infra | 17:37 | |
| clarkb | mordred: AJaeger https://git.openstack.org/cgit/openstack-infra/gerrit/tree/playbooks/gerrit-buck/post-publish.yaml?h=openstack/2.13.12 the issue is buck_publish_path is undefined. I want that path to be roughly artifacts/ci/gerrit so that when we copy artifacts to the tarballs server they end up in the /ci/gerrit/ path there | 17:37 |
| *** ykarel|away has quit IRC | 17:38 | |
| clarkb | mordred: AJaeger however, because we are synchronizing files in subdirs I am not sure if those synchronize flags will result in a gerrit/some/path/foo.war subpath being created | 17:38 |
| clarkb | do you know? | 17:38 |
| * clarkb works up a local test | 17:39 | |
| mordred | clarkb: well ... for publish-artifacts base job, I believe we'll be publishing to a subdir of project-name on tarballs (on purpose) | 17:39 |
| mordred | clarkb: so it'll be /gerrit - unless you use a different base job defined in project-config | 17:39 |
| corvus | considering that we're okay with the wars going to the logserver in check, i think we can discard the use of /ci now | 17:40 |
| clarkb | mordred: it uses zuul-jobs/roles/publish-artifacts-to-fileserver/ | 17:40 |
| mordred | corvus: yah | 17:40 |
| *** jamesmcarthur has joined #openstack-infra | 17:40 | |
| clarkb | corvus: thats fair. I am just trying to make sense of this mess of flags | 17:40 |
| clarkb | like does --include preserve the path or only the file? | 17:40 |
| clarkb | (I'm working on a test locally) | 17:40 |
| mordred | clarkb: yuah - publish-openstack-artifacts shoudl publish whatever is in work/artifacts on the executor to tarballs.o.o/{project_short_name} | 17:41 |
| corvus | ttx: should we go ahead and use a different prefix (rather than openstack-infra/) for kata-zuul-config? | 17:41 |
| corvus | clarkb: yep. just making sure we agree on the desired end state :) | 17:41 |
| mordred | clarkb: I' just remove buck_publish_path | 17:41 |
| clarkb | mordred: ok, then I still need to sort out if I get the unwated buck output path | 17:42 |
| ttx | corvus: tempting. kata/zuul-config ? | 17:42 |
| mordred | clarkb: and while you're at it - just collapse that synchronize command to not be like the legacy synchronizes | 17:42 |
| ttx | or even kata/project-config | 17:42 |
| clarkb | mordred: I don't know what that means | 17:42 |
| * ttx needs to run... put suggestions in that review | 17:42 | |
| corvus | ttx: wfm, though should it be kata-containers to normalize with their github org? or do you want it to intentionally be different. | 17:42 |
| mordred | clarkb: make {{ ansible_user_dir }}/workspace/gerrit/buck-out/gen/gerrit-*.war the src argument | 17:43 |
| mordred | remove buck_publish_path from the dest - and remove all of the rsync_options | 17:43 |
| mordred | in fact, you could probably switch to using the copy module | 17:43 |
| mordred | but synchronize will work fine too | 17:43 |
| ttx | corvus: you could (in theory) have "kata foobar" as another project sharing some branding/ governance | 17:43 |
| AJaeger | clarkb: sorry, can't help right now | 17:44 |
| clarkb | mordred: can I drop all of the include and exclude flags/ | 17:44 |
| mordred | yeah | 17:44 |
| ttx | but yeah kata-containers/project-config is probably better for alignment with openstack repo name ? | 17:44 |
| clarkb | fwiw I can't get those include flags to reproduce locally | 17:45 |
| *** yamamoto has joined #openstack-infra | 17:45 | |
| clarkb | keeps saying it is skipping my source dir | 17:45 |
| mordred | clarkb: something liek this: http://paste.openstack.org/show/742672/ | 17:45 |
| corvus | clarkb: that looks like an auto-generate synchronize from the zuulv3 migration | 17:45 |
| corvus | (as the author) i think they are incomprehensible and should be replaced rather than modified :) | 17:46 |
| clarkb | corvus: ok I agree they are incomprehensible as testing shows they produce noops :) | 17:46 |
| clarkb | I'll update as mordred suggests | 17:46 |
| corvus | ++ mordred's thing | 17:46 |
| clarkb | corvus: do you know what that was trying to do ( now I'm somewhat curious) | 17:47 |
| corvus | (the complexity comes because we're convincing ansible to make rsync behave like jenkins with its ant expression parser) | 17:47 |
| clarkb | will the * work in the src as in mordred's example? | 17:48 |
| *** yamamoto has quit IRC | 17:48 | |
| corvus | clarkb: i'm hoping that ^ is sufficient to slake your curiosity; if you want more, i can go dig up the comments that explain it | 17:48 |
| *** yamamoto has joined #openstack-infra | 17:48 | |
| mordred | clarkb: you could leave out the star and just do the directory | 17:48 |
| *** yamamoto has quit IRC | 17:48 | |
| clarkb | mordred: assuming there isn't anything else in there we don't want. Do you happen to have a gerrit build dir handy still? you could check | 17:49 |
| corvus | should be easy to test the * | 17:49 |
| mordred | clarkb: src: '{{ ansible_user_dir }}/workspace/gerrit/buck-out/gen/' | 17:49 |
| clarkb | corvus: ya testing the * locally now | 17:49 |
| mordred | but also - yeah - should be easy to test the * | 17:49 |
| *** auristor has quit IRC | 17:49 | |
| clarkb | rsync proper won't expand the * itself | 17:50 |
| clarkb | so ya I don't think it will work | 17:50 |
| clarkb | I'll do the dir suggestion | 17:50 |
| mordred | cool. and if we get some garbage - oh well, we get some garbage | 17:50 |
| clarkb | mordred: ^ and if you happen to have one of those builds avaialbe you can double check if there are other artifacts? | 17:50 |
| mordred | we won't be running this too many times anyway | 17:50 |
| mordred | clarkb: I do not have any buck builds locally | 17:51 |
| clarkb | and you deleted your held node? | 17:51 |
| mordred | clarkb: my held node was a docker/bazel node | 17:52 |
| mordred | so completely different/unuseful | 17:52 |
| mordred | although - come to think of it - no, I have not deleted it yet | 17:52 |
| *** trown is now known as trown|lunch | 17:52 | |
| *** yamamoto has joined #openstack-infra | 17:53 | |
| clarkb | mordred: corvus remote: https://review.openstack.org/631037 Fix tarballs.o.o publishing | 17:53 |
| *** wolverineav has joined #openstack-infra | 17:54 | |
| mordred | +A | 17:54 |
| clarkb | thanks! | 17:55 |
| fungi | okay, back from some amazing fish curry, will catch up real quick and see what i can help with | 17:55 |
| *** auristor has joined #openstack-infra | 17:56 | |
| clarkb | fungi: probably reviewing the discusion around zuul.opendev.org and base job reorg is good to catch up on | 17:57 |
| *** wolverineav has quit IRC | 17:57 | |
| *** wolverineav has joined #openstack-infra | 17:57 | |
| *** mdrabe has joined #openstack-infra | 17:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: DNM Testing fetch-output and merge-output-to-logs https://review.openstack.org/628668 | 17:59 |
| *** jpena is now known as jpena|off | 17:59 | |
| mordred | clarkb, corvus: ^^ that should test out the merge-output-to-logs in base-test - and if it works I think we can go head and add to base and then we can start migrating things to take advantage of it | 18:00 |
| mordred | fungi: mm. I want amazing fish curry | 18:00 |
| efried | Howdy folks. I need to get my stackalytics affiliation updated. I submitted https://review.openstack.org/#/c/628013/ but it hasn't gotten attention. Anyone else able to push that through? | 18:00 |
| *** gfidente is now known as gfidente|afk | 18:00 | |
| *** auristor has quit IRC | 18:01 | |
| *** pcaruana has quit IRC | 18:01 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 18:01 |
| *** yamamoto has quit IRC | 18:01 | |
| *** psachin has quit IRC | 18:03 | |
| mordred | efried: normally ilya gets around to them | 18:03 |
| *** martinkennelly has quit IRC | 18:03 | |
| mordred | efried: also - congrats on the new job! | 18:03 |
| efried | mordred: Okay. It's been sitting for a week, so wasn't sure if Ilya was no longer around or on vacation or... | 18:03 |
| efried | thanks! | 18:03 |
| *** auristor has joined #openstack-infra | 18:06 | |
| *** studarus has joined #openstack-infra | 18:06 | |
| *** yamamoto has joined #openstack-infra | 18:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Fix read the docs build for sqlalchemy-migrate https://review.openstack.org/631040 | 18:08 |
| mriedem | mordred: ^ | 18:08 |
| mriedem | https://readthedocs.org/dashboard/sqlalchemy-migrate/integrations/61274/ | 18:08 |
| mordred | mriedem: looks great | 18:09 |
| clarkb | note rtd still broken anyway | 18:09 |
| clarkb | beause their api doesn't let us trigger the builds currently (there is an open bug and they intend to fix it) | 18:10 |
| mriedem | sqla-migrate docs haven't built in over a year and a half so i'm guessing no one cares that much about this one | 18:11 |
| fungi | clarkb: yep, zuul.opendev.org seems like a good step, opendev/base-jobs and kata-containers/project-config also make sense | 18:12 |
| *** diablo_rojo has joined #openstack-infra | 18:12 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 18:13 |
| corvus | clarkb: have a link to the rtd bug? | 18:13 |
| fungi | corvus: it's still https://github.com/rtfd/readthedocs.org/issues/4986 afaik | 18:14 |
| clarkb | ya that is it | 18:15 |
| *** jamesmcarthur has quit IRC | 18:18 | |
| *** panda is now known as panda|off | 18:19 | |
| *** yamamoto has quit IRC | 18:19 | |
| *** jamesmcarthur has joined #openstack-infra | 18:19 | |
| *** jamesmcarthur has quit IRC | 18:20 | |
| *** jamesmcarthur has joined #openstack-infra | 18:21 | |
| *** efried has quit IRC | 18:21 | |
| mordred | clarkb, corvus: http://logs.openstack.org/68/628668/6/check/tox-py35/afa99fc/ -- base-test job passed! | 18:22 |
| *** yamamoto has joined #openstack-infra | 18:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add fetch-output and merge-output-to-logs to base https://review.openstack.org/629585 | 18:23 |
| mordred | fungi, clarkb, corvus: ^^ that should now be ready to go | 18:24 |
| *** ykarel has quit IRC | 18:25 | |
| *** efried has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix read the docs build for sqlalchemy-migrate https://review.openstack.org/631040 | 18:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update base-test jobs to publish artifacts too https://review.openstack.org/631045 | 18:32 |
| *** yamamoto has quit IRC | 18:32 | |
| *** yamamoto has joined #openstack-infra | 18:33 | |
| *** wolverineav has quit IRC | 18:34 | |
| *** wolverineav has joined #openstack-infra | 18:35 | |
| *** yamamoto has quit IRC | 18:36 | |
| *** smarcet has quit IRC | 18:37 | |
| *** yamamoto has joined #openstack-infra | 18:37 | |
| *** wolverineav has quit IRC | 18:40 | |
| *** wolverineav has joined #openstack-infra | 18:43 | |
| *** smarcet has joined #openstack-infra | 18:43 | |
| *** hamzy_ has joined #openstack-infra | 18:44 | |
| *** smarcet has quit IRC | 18:44 | |
| *** yamamoto has quit IRC | 18:45 | |
| *** yamamoto has joined #openstack-infra | 18:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add commit-message-length-validator gerrit plugin https://review.openstack.org/631047 | 18:47 |
| *** wolverineav has quit IRC | 18:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 18:48 |
| mordred | clarkb, corvus, fungi, pabelanger: https://review.openstack.org/631047 if you don't mind ... you'd think I'd be able to collect a simple list, but you'd be wrong | 18:49 |
| clarkb | mordred: http://tarballs.openstack.org/gerrit/ so it worked but it copied a lot of stuff :/ | 18:49 |
| mordred | clarkb: jeez | 18:50 |
| clarkb | we could do an ls first. We might also be able to do an include ? | 18:50 |
| fungi | yeah, i guess that's the whole build tree for the warfile? | 18:51 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul-jobs master: install-nodejs: add support for RPM-based OSes https://review.openstack.org/631049 | 18:52 |
| *** kopecmartin is now known as kopecmartin|off | 18:52 | |
| clarkb | fungi: ya | 18:53 |
| clarkb | I can clean it up, I just want to sort out a fix first | 18:53 |
| *** eharney has quit IRC | 18:54 | |
| *** yamamoto has quit IRC | 18:59 | |
| *** hamzy_ has quit IRC | 18:59 | |
| *** hamzy_ has joined #openstack-infra | 19:00 | |
| *** jamesmcarthur has quit IRC | 19:00 | |
| *** jamesmcarthur has joined #openstack-infra | 19:01 | |
| clarkb | mordred: fungi corvus remote: https://review.openstack.org/631052 Only publish the gerrit war I managed to test that locally | 19:02 |
| *** wolverineav has joined #openstack-infra | 19:02 | |
| clarkb | with rsync | 19:02 |
| fungi | excellent | 19:02 |
| clarkb | we should clean out the dir above before merging that to confirm | 19:02 |
| * clarkb switches to meeting now | 19:03 | |
| mordred | clarkb: ++ | 19:03 |
| *** yamamoto has joined #openstack-infra | 19:05 | |
| *** e0ne has joined #openstack-infra | 19:07 | |
| *** yamamoto has quit IRC | 19:09 | |
| *** raissa has quit IRC | 19:09 | |
| *** eharney has joined #openstack-infra | 19:09 | |
| *** hamzy_ has quit IRC | 19:13 | |
| *** rlandy is now known as rlandy|brb | 19:14 | |
| *** yamamoto has joined #openstack-infra | 19:16 | |
| *** trown|lunch is now known as trown | 19:19 | |
| AJaeger | mordred: https://review.openstack.org/#/c/511860/3 has dependency on an abandoned change | 19:21 |
| mordred | AJaeger: sounds like me | 19:21 |
| AJaeger | mordred: and how does it relate to https://review.openstack.org/#/c/631045/ ? | 19:22 |
| mordred | AJaeger: ah! the first one can be abandoned (done now) | 19:22 |
| mordred | AJaeger: 631045 is the new hotness | 19:23 |
| AJaeger | mordred: great! | 19:23 |
| clarkb | http://tarballs.openstack.org/gerrit/ is cleaned up, https://review.openstack.org/631052 can be approved if it looks correct to everyone | 19:25 |
| *** hamzy_ has joined #openstack-infra | 19:31 | |
| *** rlandy|brb is now known as rlandy | 19:33 | |
| *** lpetrut has quit IRC | 19:38 | |
| *** smarcet has joined #openstack-infra | 19:41 | |
| *** yamamoto has quit IRC | 19:42 | |
| *** yamamoto has joined #openstack-infra | 19:42 | |
| fungi | mordred: where was base-test exercised to demonstrate the new roles? i see base-test includes them, just was hoping for a comment pointing to an exercise run | 19:45 |
| mordred | fungi: https://review.openstack.org/#/c/628668/ | 19:46 |
| fungi | thanks! | 19:46 |
| mordred | fungi: I will add comment too | 19:46 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 19:46 |
| *** electrofelix has quit IRC | 19:46 | |
| *** ekultails has quit IRC | 19:47 | |
| anteaya | the uk has voted down may's proposal | 19:48 |
| anteaya | if anyone cares | 19:48 |
| *** wolverineav has quit IRC | 19:48 | |
| *** wolverineav has joined #openstack-infra | 19:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add commit-message-length-validator gerrit plugin https://review.openstack.org/631047 | 19:49 |
| anteaya | sorry the uk parliament, not the uk as a country | 19:50 |
| *** wolverineav has quit IRC | 19:51 | |
| *** wolverineav has joined #openstack-infra | 19:51 | |
| *** wolverineav has quit IRC | 19:52 | |
| *** wolverineav has joined #openstack-infra | 19:52 | |
| fungi | wake me back up when scotland votes to secede ;) | 19:53 |
| d0ugal | +1 | 19:54 |
| d0ugal | How do we make that happen | 19:54 |
| anteaya | shouldn't take long now | 19:54 |
| fungi | yeah, was going to say, i think the question is merely how to make it happen *sooner* | 19:54 |
| d0ugal | vvnfcinbfdtltuujnhvfkctkeftggctfcgcivirkkhkt | 19:55 |
| d0ugal | umm, oops | 19:55 |
| anteaya | was that your vote? | 19:55 |
| anteaya | or a cat on the keyboard? | 19:55 |
| d0ugal | yubi key is also unhappy with the situation | 19:55 |
| d0ugal | anyway, /me goes back to waiting for mentions of scotland | 19:56 |
| anteaya | :) | 19:56 |
| *** wolverineav has quit IRC | 19:58 | |
| *** whoami-rajat has quit IRC | 20:00 | |
| mordred | d0ugal: I don't think scotland could secede from a country whose leader says sentences like "A decision that will define our decision for decades to come" | 20:04 |
| *** ekultails has joined #openstack-infra | 20:06 | |
| fungi | even i can't parse that, and i have a very flexible language parser | 20:06 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 20:06 |
| fungi | clarkb: on 631052, did you try just putting that file pattern in the source path string? | 20:09 |
| fungi | i think that would eliminate the need for all the --include and --exclude options | 20:09 |
| clarkb | fungi: I tried a * in the source of an rsync and it failed beacuse it looks for a literal * | 20:11 |
| clarkb | it may work locally if you don't quote the * because it will expand | 20:11 |
| clarkb | but in that case its a remote path | 20:11 |
| fungi | ahh, so rsync supports globbing in --include/--exclude but not in pathnames? | 20:11 |
| fungi | indeed... rsync: link_stat "/home/fungi/foo/baz*.bin" failed: No such file or directory (2) | 20:12 |
| *** hamzy_ has quit IRC | 20:12 | |
| fungi | that's what i get if i tell my shell not to expand it | 20:13 |
| *** kjackal has joined #openstack-infra | 20:15 | |
| fungi | and rsync has funky rules for its filter patterns to deal with things like variable-level directory matching | 20:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 20:19 |
| *** studarus has quit IRC | 20:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add fetch-output and merge-output-to-logs to base https://review.openstack.org/629585 | 20:21 |
| *** rfolco has quit IRC | 20:28 | |
| *** yamamoto has quit IRC | 20:28 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 20:32 |
| *** jamesmcarthur has quit IRC | 20:35 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 20:36 |
| *** jamesmcarthur has joined #openstack-infra | 20:37 | |
| *** wolverineav has joined #openstack-infra | 20:39 | |
| fungi | clarkb: so digging into the new storyboard-dev server, `pip freeze` raises "locale.Error: unsupported locale setting" even though the output of the `locale` command matches the old server where pip's fine | 20:42 |
| *** wolverineav has quit IRC | 20:43 | |
| ianw | fungi: pip version? 18 had some big differences in output (c.f. cmurphy's fixes) | 20:43 |
| fungi | interestingly, i think it's older | 20:44 |
| fungi | we have pip 9.0.1 on storyboard-dev.openstack.org and pip 8.1.1 on the new storyboard-dev01.opendev.org | 20:45 |
| fungi | the former is ubuntu trusty and the latter is xenial | 20:45 |
| *** jamesmcarthur has quit IRC | 20:45 | |
| *** jamesmcarthur has joined #openstack-infra | 20:46 | |
| fungi | yeah, pip is being provided by python-pip 8.1.1-2ubuntu0.4 on the new server | 20:46 |
| clarkb | fungi: I had that on an install athome re ently and it was becahsemy local locale was passed through ssh but not valid on remote | 20:46 |
| clarkb | do you use a weird local locale? | 20:47 |
| clarkb | (also Im trying to do lunch now so may not be super responsive) | 20:47 |
| fungi | clarkb: LC_ALL and LANGUAGE are unset, LC_CTYPE, LC_COLLATE and LC_MEASUREMENT are C.UTF-8, LC_TIME is en_DK.UTF-8 and all the rest are en_US.UTF-8 | 20:48 |
| clarkb | is en_DK.UTF-8 valid on that server? | 20:49 |
| clarkb | may not be buily? | 20:49 |
| *** yamamoto has joined #openstack-infra | 20:49 | |
| fungi | possible that's it, i suppose. might have to do with the version of python 2.7 since the exception is being raised in the locale stdlib module | 20:50 |
| fungi | yeah, if i export LC_TIME=C.UTF-8 it works | 20:51 |
| fungi | good call | 20:51 |
| fungi | oof, lots of differences in python modules between the two, for sure | 20:52 |
| fungi | consistent version of the oauth module though, so that's probably not it | 20:53 |
| efried | Here's a thought: /msg openstack flush to eavesdrop now please | 20:54 |
| efried | ...so when I want to link a conversation I've just had, I can force it to be there rather than waiting for the bot to catch up. | 20:54 |
| fungi | yeah, it is slightly laggy | 20:54 |
| *** wolverineav has joined #openstack-infra | 20:55 | |
| efried | I'm sure it batches every so-many minutes, which is totally reasonable - wouldn't want to be realtime, that would be a bandwidth/processing hog. | 20:55 |
| efried | but a manual trigger would be useful for the reason cited | 20:55 |
| corvus | yeah. the realtime log is available here: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-01-15.log but is not linkable; only the batch-updated html is. | 20:59 |
| corvus | (or, well, as real-time as irc gets :) | 20:59 |
| *** kjackal has quit IRC | 20:59 | |
| *** wolverineav has quit IRC | 21:00 | |
| *** wolverineav has joined #openstack-infra | 21:00 | |
| efried | corvus: Is that realtime? Wouldn't the nova equivalent be http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-01-15.log.html ? | 21:00 |
| corvus | efried: drop the '.html' at the end for the real-time log | 21:00 |
| efried | cause that guy took a 15-minute break just now. | 21:01 |
| corvus | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-01-15.log | 21:01 |
| efried | ohhh | 21:01 |
| efried | gotcha. | 21:01 |
| efried | is this under the purview of statusbot? looking for the source to see how hard this would be... | 21:01 |
| corvus | efried: it's meetbot | 21:01 |
| efried | okay. | 21:01 |
| corvus | efried: http://git.openstack.org/cgit/openstack-infra/meetbot/ | 21:01 |
| corvus | we have a local fork for... some really good reasons i'm sure. | 21:02 |
| *** yamamoto has quit IRC | 21:02 | |
| corvus | efried: i don't see any immediate blockers for your idea | 21:02 |
| fungi | i think at the time we forked, it didn't have a clear upstream any longer | 21:03 |
| corvus | efried: other than you'll probably have to remind us to go look at the proposed change :) | 21:03 |
| efried | But of corvus | 21:03 |
| fungi | clarkb: ohh... the oauth package is 1.0.1 on both servers, but oauthlib is 1.1.1 on the old one and 3.0.0 on the new one | 21:06 |
| clarkb | that seems suspicious | 21:08 |
| corvus | mordred: what's the gerrit image status? | 21:08 |
| corvus | mordred: the new repos change landed, right? but it looks like it's outdated in gerrit | 21:09 |
| corvus | mordred: should i just rebase? | 21:09 |
| corvus | mordred: did you re-upload the old version of the change? | 21:10 |
| fungi | clarkb: also the exception we're getting is raised in oauthlib, so that might be it. i'm going to see if it's downgradeable | 21:10 |
| corvus | mordred: yes you did. you undid my fixes. :( | 21:10 |
| mordred | corvus: blerg. I can fix | 21:11 |
| mordred | corvus: too much juggling | 21:11 |
| corvus | friendly reminder that 'c' in gertty, or 'git review -d' are really handy :) | 21:12 |
| corvus | i basically just hit 'c' before i start doing anything these days; it's the only way i can work with so many things going on at once | 21:12 |
| fungi | clarkb: success! for some reason we're broken with oauthlib 3.0.0 but working with 1.1.1 | 21:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 21:13 |
| mordred | corvus: that should be better | 21:13 |
| corvus | mordred: looks great, thanks! | 21:14 |
| *** eharney has quit IRC | 21:15 | |
| *** yamamoto has joined #openstack-infra | 21:17 | |
| *** whoami-rajat has joined #openstack-infra | 21:18 | |
| *** yamamoto has quit IRC | 21:20 | |
| *** hamzy_ has joined #openstack-infra | 21:21 | |
| *** xek has quit IRC | 21:24 | |
| *** kgiusti has quit IRC | 21:27 | |
| *** hamzy_ has quit IRC | 21:29 | |
| *** jamesmcarthur has quit IRC | 21:31 | |
| *** jamesmcarthur has joined #openstack-infra | 21:33 | |
| *** roman_g has joined #openstack-infra | 21:36 | |
| *** jamesmcarthur has quit IRC | 21:38 | |
| *** jamesmcarthur has joined #openstack-infra | 21:40 | |
| *** trown is now known as trown|outtypewww | 21:41 | |
| efried | corvus: You sure it's meetbot that does regular channel logging? | 21:41 |
| corvus | efried: positive | 21:42 |
| corvus | efried: though... | 21:42 |
| *** jamesmcarthur has quit IRC | 21:42 | |
| *** ekultails has quit IRC | 21:42 | |
| *** jamesmcarthur has joined #openstack-infra | 21:43 | |
| corvus | efried: maybe meetbot is actually just the plugin for meetings, and some other part of supybot actually does the logging | 21:43 |
| corvus | so, er, that's the program/process doing it; i may have pointed to the wrong bit of code | 21:43 |
| efried | corvus: having stared at meetbot for the past half hour or so, it's looking to me like the code in meetbot is only operating on meetings | 21:44 |
| efried | corvus: Interestingly, in my poking, I did discover that - during a meeting - there's an existing #save command that flushes the log under eavesdrop/meetings/* (but not under eavesdrop/irclogs/*) | 21:45 |
| efried | But it doesn't work in regular channel flow | 21:45 |
| corvus | efried: oh, wow, i don't think supybot/meetbot does the html conversion | 21:46 |
| corvus | efried: https://git.openstack.org/cgit/openstack-infra/puppet-meetbot/tree/manifests/site.pp#n105 | 21:46 |
| corvus | efried: i'm sorry about leading you astray there. | 21:47 |
| efried | this is so not a big deal, but obviously I'm having trouble letting go of it >:| | 21:47 |
| corvus | yeah. i like the idea, though implementation now seems much more difficult. | 21:48 |
| *** ekultails has joined #openstack-infra | 21:48 | |
| efried | corvus: when I did the #save in my test meeting, it definitely updated the html log | 21:48 |
| corvus | efried: the html meeting log though, not the html channel log | 21:49 |
| efried | yes, just so. | 21:49 |
| corvus | i think we can see the 15m cpu spikes on eavesdrop from the html conversion: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=176&rra_id=all | 21:50 |
| efried | wait, so there's a thing that isn't meetbot that logs raw text in realtime, and then a thing that isn't either of those that flushes the raw logs to HTML periodically? | 21:50 |
| corvus | efried: the supybot process runs on the eavesdrop vm and writes the channel .log files in realtime. it has a plugin (meetbot) which handles meetings and writes meeting logs (text and html). there is a cron job on the eavesdrop vm which converts the channel .log files written by supybot to .log.html files every 15m. | 21:52 |
| *** yamamoto has joined #openstack-infra | 21:52 | |
| *** yamamoto has quit IRC | 21:52 | |
| efried | wow, okay. | 21:52 |
| corvus | confusing all of this is that, collectively, we've spent the past 8 years calling this entire system 'meetbot'. :) | 21:53 |
| corvus | which i'm going to cite as the reason why i apparently forgot how it all worked :) | 21:53 |
| efried | frankly I'm shocked that you can keep as much in your brain as you do | 21:53 |
| efried | I'm going to leave this alone now. Really. I swear. | 21:54 |
| efried | I'll count it as a win that I now know that the HTML logs are flushed on the quarter hour, which I didn't know before, just thought it took "a while to catch up". And also that I now know about #save for meetbot. | 21:55 |
| corvus | efried: :) i'm planning on replacing supybot with limnoria; when i do, we should look into whether there's an html generation plugin, and if so, maybe we can use that and implement your idea. | 21:55 |
| efried | love it, lmk if I can help with that. Thanks for the guidance corvus | 21:55 |
| *** wolverineav has quit IRC | 21:58 | |
| *** wolverineav has joined #openstack-infra | 21:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add docker image promotion roles https://review.openstack.org/631078 | 22:03 |
| *** yamamoto has joined #openstack-infra | 22:05 | |
| corvus | mordred: Successfully tagged opendevorg/gerrit:change_630468 http://logs.openstack.org/68/630468/25/check/system-config-build-image-gerrit/6d20db6/job-output.txt.gz#_2019-01-15_21_30_47_766273 | 22:05 |
| corvus | clarkb, fungi: https://review.openstack.org/630468 is ready to land | 22:06 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Pin oauthlib<3 for now https://review.openstack.org/631079 | 22:06 |
| fungi | clarkb: ^ | 22:06 |
| *** imacdonn has quit IRC | 22:06 | |
| *** imacdonn has joined #openstack-infra | 22:07 | |
| *** owalsh_ has joined #openstack-infra | 22:07 | |
| *** jamesmcarthur has quit IRC | 22:09 | |
| *** owalsh has quit IRC | 22:10 | |
| *** owalsh_ is now known as owalsh | 22:13 | |
| *** yamamoto has quit IRC | 22:14 | |
| *** jamesmcarthur has joined #openstack-infra | 22:15 | |
| *** boden has quit IRC | 22:19 | |
| clarkb | ok lunch took longer than expected but back now | 22:20 |
| clarkb | http://tarballs.openstack.org/gerrit/ looks like that worked. I'll get a change up to use the new war then review the chagnes above | 22:20 |
| *** jcoufal has quit IRC | 22:22 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Use correct 2.13.12 war on review-dev https://review.openstack.org/631082 | 22:22 |
| *** yamamoto has joined #openstack-infra | 22:23 | |
| *** gfidente|afk has quit IRC | 22:24 | |
| fungi | mordred: corvus: i've approved 630468, though now that's working i'm wondering if we've adequately explained why we need it. is this so that we can backport patches? or to include plugins not included in the official gerrit images? or some other reason? | 22:24 |
| *** jamesmcarthur has quit IRC | 22:24 | |
| clarkb | fungi: mordred and corvus should confirm, but I believe it is in part to set configuration things that upstream images don't expose via its images? | 22:25 |
| *** jamesmcarthur has joined #openstack-infra | 22:25 | |
| fungi | ohh, so configuration isn't externally suppliable with the official gerrit images? | 22:25 |
| corvus | i think we could do that by basing on the upstream image; i believe this will let us backport or make our own patches | 22:25 |
| clarkb | some of it is but no all of it iirc | 22:25 |
| clarkb | corvus: ah | 22:26 |
| corvus | oh, and to build the plugins we want into it | 22:26 |
| corvus | that may be the most important thing :) | 22:26 |
| fungi | got it, so plugins can't be bolted on with... sidecars? is that what they're called? | 22:27 |
| clarkb | a sidecar is a container with a running process itself, not something you load into another process | 22:27 |
| corvus | i reckon they maybe probably could be, or possibly built and added into a custom image based on upstream. | 22:27 |
| fungi | cool, just curious to what extent we're duplicating work also being done by the gerrit maintainers | 22:28 |
| corvus | though i think as long as you're building plugins you're 95% of the way to building gerrit anyway | 22:28 |
| corvus | fungi: the dockerfile is mostly a copy of upstreams | 22:28 |
| fungi | and these are opendev-specific container images, so, like, zuul integration tests or quickstart walkthrough wouldn't use them | 22:28 |
| *** jamesmcarthur has quit IRC | 22:30 | |
| dmsimard | corvus: out of curiosity, could the docker_image ansible module have been used instead of the command tasks ? | 22:30 |
| dmsimard | in 631078 | 22:30 |
| dmsimard | I wonder myself when is the right time or opportunity to use Ansible modules for certain things :) | 22:31 |
| corvus | dmsimard: i did not know about it :) | 22:32 |
| *** jamesmcarthur has joined #openstack-infra | 22:32 | |
| dmsimard | that's an interesting conclusion as well | 22:32 |
| fungi | certainly a valid reason | 22:32 |
| dmsimard | there's so many modules now o_O | 22:32 |
| clarkb | fwiw I like the command tasks because its an easy I want to run this locally conversion without needing to set up an ansible inventory and such first | 22:33 |
| fungi | as they say, ignorance is nine tenths of the law, or something like that | 22:33 |
| dmsimard | clarkb: I don't think either are wrong | 22:33 |
| clarkb | dmsimard: ya the big thing you miss out on is the variable magic usually | 22:33 |
| dmsimard | it's like using a role on galaxy or rolling out your own | 22:33 |
| clarkb | so if checking an rc is sufficient its fine | 22:34 |
| *** lpetrut has joined #openstack-infra | 22:34 | |
| clarkb | but if you need to parse output then it can get ugly fast | 22:34 |
| dmsimard | ++ | 22:34 |
| fungi | command tasks are a little more self-explanatory sometimes too. it gets tedious tracking down every single module and figuring out what command each is running | 22:34 |
| *** jamesmcarthur has quit IRC | 22:34 | |
| fungi | though at least if you have a working example, you can look at the json output | 22:35 |
| dmsimard | but then where does it end ? I mean, you could use jq | 22:35 |
| dmsimard | haha | 22:35 |
| fungi | infra-root: i'm going ahead and updating dns for storyboard-dev now (ttl is already 5 minutes) to the new server, since the openid breakage has been identified as a new behavior with oauthlib 3.0.0 released a week ago (it seems to have broken some auth-related sb unit tests too) | 22:37 |
| clarkb | fungi: its all confirmed working after downgrade? | 22:38 |
| clarkb | also yay | 22:38 |
| clarkb | I'm going to reboot now because I'm seeing really weird network latency nd I have pending kernel updates anyway | 22:38 |
| *** lpetrut has quit IRC | 22:38 | |
| clarkb | back in a few | 22:38 |
| *** jamesmcarthur has joined #openstack-infra | 22:38 | |
| fungi | clarkb: yeah, actually downgrading to the last release <3 (2.1.0 from may) also works | 22:39 |
| *** yamamoto has quit IRC | 22:40 | |
| *** ekultails has quit IRC | 22:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Remove all docker image building jobs https://review.openstack.org/631084 | 22:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add docker image building jobs https://review.openstack.org/631085 | 22:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add jinja-init image jobs https://review.openstack.org/631086 | 22:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea-init image jobs https://review.openstack.org/631087 | 22:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea image jobs https://review.openstack.org/631088 | 22:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gerrit image jobs https://review.openstack.org/631089 | 22:40 |
| *** jamesmcarthur has quit IRC | 22:41 | |
| corvus | clarkb, fungi, mordred, dmsimard: ^ that's our jobs rebased on the new roles; i deleted everything and re-added them from scratch stepwise so we can show these changes to folks as exemplars of how to use the roles | 22:41 |
| *** yamamoto has joined #openstack-infra | 22:41 | |
| *** jamesmcarthur has joined #openstack-infra | 22:43 | |
| clarkb | ++ I'll take a look. Turns out it wasn't local kernel having problems but my AP | 22:44 |
| *** yamamoto has quit IRC | 22:44 | |
| clarkb | I have working networking again though | 22:44 |
| *** rcernin has joined #openstack-infra | 22:46 | |
| fungi | efried: corvus: possibly another option after the limnoria conversion is that we could switch to something which renders text logs to html on the fly, eliminating the need for any periodic cron job | 22:47 |
| efried | ++ | 22:47 |
| fungi | so not as a bot plugin, just as a ssi/cgi/wsgi/whatever's popular these days | 22:48 |
| corvus | oh interesting idea | 22:48 |
| fungi | i have a very ancient one i wrote (in php!!?!) for some irc channels on an irc server i've been running for decades | 22:49 |
| fungi | which basically just htmlizes eggdrop logs as they're accessed | 22:49 |
| fungi | so something along the lines of what we do with osloganalyze | 22:49 |
| fungi | obviously wouldn't want to reuse that, but these days there are plenty of those log prettifiers floating around to choose from should we want | 22:50 |
| clarkb | I dunno fungi php tools sound like fun :P | 22:50 |
| fungi | not this one. it tries to safely feed search terms into agrep to do keyword searches in logfiles | 22:51 |
| fungi | some serious ugliness | 22:51 |
| *** trident has joined #openstack-infra | 22:52 | |
| *** bgmccollum has quit IRC | 22:53 | |
| dmsimard | I've used errbot on several projects to some levels of success | 23:00 |
| *** jtomasek has quit IRC | 23:01 | |
| clarkb | corvus: that stack is far more readable. Question on https://review.openstack.org/#/c/631089/1 I think the upload job might be slightly wrong? | 23:01 |
| corvus | dmsimard: it's nice, but we evaluated it and rejected it for a number of good reasons. see http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html for the plan. | 23:01 |
| dmsimard | RDO has an errbot instance that receives sensu alerts and forwards them to IRC :p | 23:01 |
| dmsimard | corvus: thanks, I'll catch up | 23:01 |
| clarkb | corvus: question is in gerrit | 23:01 |
| dmsimard | ah, yes, I remember now. meetbot. | 23:02 |
| dmsimard | I remember searching for that. :( | 23:02 |
| corvus | clarkb: oh good point. we should probably just copy required-projects | 23:03 |
| corvus | clarkb: if we parented on build, we'd need to move the upload playbook and credential to each of the individual image-upload jobs | 23:06 |
| *** rascasoft has quit IRC | 23:06 | |
| *** jcoufal has joined #openstack-infra | 23:06 | |
| clarkb | ah right | 23:06 |
| corvus | clarkb: (either that, or use multiple inheritance) | 23:06 |
| *** efried1 has joined #openstack-infra | 23:06 | |
| *** efried has quit IRC | 23:06 | |
| *** efried1 is now known as efried | 23:06 | |
| corvus | the gerrit job is the one job where the config is longer by not doing that :) | 23:06 |
| corvus | also, apparently i forgot to delete the gerrit repos playbook in my first change, but i don't think that's important or affects the instructional value of the changes, so maybe we'll just let that slide. | 23:07 |
| clarkb | I think i'm ok just using an anchor for required projects | 23:07 |
| clarkb | corvus: ya I think the first couple end up being most applicable as examples anyway | 23:08 |
| clarkb | since they are simple and focus more on the docker not the gerrit is weird | 23:08 |
| *** yamamoto has joined #openstack-infra | 23:08 | |
| *** yamamoto has quit IRC | 23:08 | |
| corvus | yeah. the last one has tags and build args though, so it's something. plus it shows the flexibility of using custom jobs here. | 23:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gerrit image jobs https://review.openstack.org/631089 | 23:09 |
| corvus | clarkb: fixed. i only added it to the upload job; i don't think promote needs it. | 23:09 |
| clarkb | corvus: ya promote shouldn't | 23:09 |
| *** jamesmcarthur has quit IRC | 23:13 | |
| *** jamesmcarthur has joined #openstack-infra | 23:14 | |
| clarkb | infra-root can I get reviews on https://review.openstack.org/#/c/631082/ I'm able to watch that go in today. I think my rough plan is get that in. See how puppet applies it. Then if that looks good downgrade to version currently on prod server and have puppet redo upgrade to this version (so that it mimics prod upgrade) | 23:20 |
| fungi | oh, and i said i was going to update storyboard-dev dns but then got sidetracked and haven't yet. doing now | 23:21 |
| *** ijw has joined #openstack-infra | 23:21 | |
| *** efried has quit IRC | 23:22 | |
| *** efried has joined #openstack-infra | 23:22 | |
| *** e0ne has quit IRC | 23:24 | |
| *** rlandy has quit IRC | 23:33 | |
| *** jamesmcarthur has quit IRC | 23:33 | |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| *** rascasoft has joined #openstack-infra | 23:34 | |
| fungi | #status log storyboard-dev.openstack.org dns updated to cname to storyboard-dev01.opendev.org | 23:39 |
| openstackstatus | fungi: finished logging | 23:39 |
| *** rascasoft has quit IRC | 23:39 | |
| *** jamesmcarthur has quit IRC | 23:40 | |
| corvus | clarkb: i believe the voip.ms account balance is negative | 23:41 |
| corvus | should i drop some coins in, or...? | 23:41 |
| clarkb | corvus: I did bring it up with scott and he did say that he would look at setting an account up, but I haven't heard if that has happened | 23:42 |
| clarkb | let me ping scott to see if that has happened | 23:42 |
| clarkb | corvus: if we can wait a few days scott and I will be in the same physical location starting mondayish | 23:43 |
| clarkb | corvus: and I can put it high on the list to walk through together | 23:44 |
| clarkb | (I've also pinged to see if maybe its already done) | 23:44 |
| corvus | clarkb: no skin off my back, but we might lose the number | 23:44 |
| corvus | i'm not sure we're tied to it anyway, and that might be the easiest way to make the switch. | 23:44 |
| clarkb | we'd probably replace the number with a new account anyway? I am also not personally attached to it | 23:44 |
| clarkb | ya | 23:44 |
| corvus | just that's the only downside i can think of | 23:44 |
| *** rossella_s has quit IRC | 23:45 | |
| *** jcoufal has quit IRC | 23:45 | |
| *** rossella_s has joined #openstack-infra | 23:46 | |
| *** whoami-rajat has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Pin oauthlib<3 for now https://review.openstack.org/631079 | 23:49 |
| *** yamamoto has joined #openstack-infra | 23:51 | |
| *** wolverineav has quit IRC | 23:56 | |
| *** yamamoto has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
