| *** rlandy|rover|biab is now known as rlandy|rover | 00:31 | |
| *** rlandy|rover is now known as rlandy|out | 00:45 | |
| *** ysandeep|out is now known as ysandeep | 01:47 | |
| *** ysandeep is now known as ysandeep|afk | 03:15 | |
| *** akahat|ruck is now known as akahat | 03:46 | |
| *** ysandeep|afk is now known as ysandeep | 03:55 | |
| *** ysandeep is now known as ysandeep|afk | 04:32 | |
| *** chandankumar is now known as chkumar|rover | 04:32 | |
| *** ysandeep|afk is now known as ysandeep | 05:07 | |
| marios | o/ | 05:26 |
|---|---|---|
| chkumar|rover | Good morning marios sshnaidm #oooq o/ | 05:51 |
| marios | \o | 05:52 |
| marios | sshnaidm: is around chkumar|rover ? | 05:52 |
| marios | :) | 05:52 |
| chkumar|rover | marios: he is our mate, I miss him a lot :-) | 05:53 |
| marios | :) | 05:53 |
| Tengu | ysandeep: heya! weird, I don't see the console rule in the standalone job... ? | 06:03 |
| ysandeep | Tengu, which job logs you are checking? | 06:05 |
| Tengu | ysandeep: https://review.rdoproject.org/r/c/testproject/+/31954, https://logserver.rdoproject.org/54/31954/98/check/periodic-tripleo-ci-centos-9-scenario003-standalone-master/f330fe7/logs/undercloud/var/log/extra/nftables.txt.gz | 06:05 |
| Tengu | ysandeep: though the rule IS present in the parameter_defaults: https://logserver.rdoproject.org/54/31954/98/check/periodic-tripleo-ci-centos-9-scenario003-standalone-master/f330fe7/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz | 06:05 |
| Tengu | ysandeep: I was about to check if there isn't some additional include that may override. | 06:06 |
| ysandeep | I know what is happening, that's why i ran again sc03 | 06:06 |
| ysandeep | yes we have additional override in sc03 file | 06:06 |
| Tengu | bingo. | 06:06 |
| Tengu | it's overridden. | 06:06 |
| Tengu | scenario003-standalone.yaml | 06:06 |
| ysandeep | yes, I plan to drop the override from sc03, they were added just to test ExtraFirewallRules works well | 06:07 |
| Tengu | fun. I actually see the rule only once in the ruleset. | 06:08 |
| ysandeep | https://opendev.org/openstack/tripleo-heat-templates/commit/69fe39c8e402875fd1a6bd55c136f4dd2a5d7bce | 06:08 |
| ysandeep | https://opendev.org/openstack/tripleo-heat-templates/commit/dbe38cac185ef2b51cdd283531bce393e9ce8e6c | 06:09 |
| Tengu | ysandeep: is there a log with the iptables engine for that very same job? | 06:09 |
| Tengu | I want to compare something | 06:09 |
| ysandeep | Tengu, yes, grabing | 06:09 |
| Tengu | thanks. | 06:09 |
| Tengu | I think I understand with Emilien's patch, but I want to be 100% sure. | 06:09 |
| ysandeep | Tengu, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a748a15/ | 06:10 |
| Tengu | ++ | 06:10 |
| Tengu | ah. yeah. no comment in there. | 06:10 |
| Tengu | but it seems to match my thoughts. | 06:10 |
| Tengu | ysandeep: what if I add the ExtraFirewallRules inside StandaloneParameters ? | 06:11 |
| Tengu | that should make it work all the time.. ? | 06:11 |
| Tengu | though it may override some other things... humf. | 06:11 |
| Tengu | lemme do that test. | 06:11 |
| ysandeep | Tengu, we pass sc03 yaml after standalone_parameters.yaml, so sc03 will win | 06:12 |
| ysandeep | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a748a15/ | 06:12 |
| ysandeep | sry: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-master/a748a15/logs/undercloud/home/zuul/tripleo_deploy.sh.txt.gz | 06:12 |
| Tengu | ysandeep: afaik it should merge in this case. | 06:13 |
| ysandeep | if they merges - we are good | 06:13 |
| Tengu | let's just test. | 06:13 |
| Tengu | it's running | 06:13 |
| marios | pojadhav: o/ hey fyi you need to rebase that one https://review.opendev.org/c/openstack/tripleo-ci/+/856061/1#message-23bdfeefe5d963c189eee63780017b29d0308d5f | 06:15 |
| ysandeep | Tengu, ack | 06:16 |
| ysandeep | marios, there is a comment on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45202/1#message-9693b75bb34413799c7ddb68b8adb11deca9b445 , do you want to address that/I don't mind mering as it is as we have current-tripleo-rdo everywhere and they can be removed together | 06:31 |
| *** amoralej|off is now known as amoralej | 06:34 | |
| chkumar|rover | marios: ysandeep https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/429398 to clear cix for rhos-17 rhel-9 | 06:36 |
| ysandeep | chkumar|rover, akahat we only need new tag for 17.1/9 and not for 17.1/8 and 17.0/9? | 06:37 |
| marios | looking chkumar|rover | 06:58 |
| chkumar|rover | ysandeep: marios let me check with fmount | 07:01 |
| marios | chkumar|rover: ysandeep: merged it as the bz only mentioned 9/17.1 jobs we can have followup patch | 07:01 |
| *** frenzyfriday is now known as frenzyfriday|ruck | 07:01 | |
| ysandeep | okay | 07:02 |
| frenzyfriday|ruck | chkumar|rover, 0/ Good morning | 07:03 |
| marios | ysandeep: chkumar|rover: don't know about 8 but the duplicate BZ has some 17.0 logs... though are we monitoring those any more at least we don't report them | 07:03 |
| marios | (duplicate bz https://bugzilla.redhat.com/show_bug.cgi?id=2123335 linked from the original one) | 07:04 |
| pojadhav | marios, ack | 07:18 |
| Tengu | ysandeep: seems to work! | 07:24 |
| Tengu | https://logserver.rdoproject.org/54/31954/98/check/periodic-tripleo-ci-centos-9-scenario003-standalone-master/dd1f01f/logs/undercloud/var/log/extra/dropped-packets.txt.gz | 07:24 |
| Tengu | ysandeep: https://logserver.rdoproject.org/54/31954/98/check/periodic-tripleo-ci-centos-9-scenario003-standalone-master/dd1f01f/logs/undercloud/var/log/extra/nftables.txt.gz yesss | 07:25 |
| Tengu | so the param is merged. | 07:25 |
| Tengu | peeerfect. | 07:25 |
| Tengu | we can go with that if you're OK. | 07:25 |
| jm1 | happy friday #oooq :) | 07:25 |
| *** jpena|off is now known as jpena | 07:26 | |
| Tengu | hey jm1 | 07:26 |
| ysandeep | Tengu, nice, unrelated Q - When we pass inside StandaloneParameter, do they override the global one... Not seeing 301 and 302 rule from here: https://opendev.org/openstack/tripleo-heat-templates/commit/69fe39c8e402875fd1a6bd55c136f4dd2a5d7bce | 07:28 |
| jm1 | chkumar|rover: today is the most difficult day to miss or meet sshnaidm: (1) he prefers kubernetes nowadays (😋), (2) its friday aaaaand (3) he is on PTO 🤯 | 07:28 |
| Tengu | ysandeep: it seems to override, yep | 07:28 |
| Tengu | that's what I get from Emilien's message | 07:28 |
| Tengu | ysandeep: especially https://opendev.org/openstack/tripleo-heat-templates/commit/dbe38cac185ef2b51cdd283531bce393e9ce8e6c#diff-a6f2029e3378b44da1fc1f6a972ba30d27685c10 - the last change in that file. | 07:30 |
| Tengu | if we wanted to keep both, we'd should have added, not replaced, I think. | 07:30 |
| ysandeep | ack | 07:31 |
| ysandeep | jm1, good morning o/ | 07:33 |
| jm1 | Tengu, ysandeep: reporting for duty 💂 | 07:34 |
| ysandeep | jm1: :) thank you sir 💂 | 07:36 |
| Tengu | ysandeep: that said.... imho we should merge the 2 listings... I'll propose a patch in order to ensure we aren't overriding things. | 08:01 |
| Tengu | getting a bunch of "default" extra rules in the ExtraFirewallRule, then adding some specific, per-role via the [Role]Params: {ExtraFirewallRules: []} kind of makes sense imho | 08:02 |
| Tengu | but not now, have to run for some errand. | 08:02 |
| ysandeep | yes, that will be good improvement | 08:03 |
| dpawlik | arxcruz: merging lukas change, thanks | 08:29 |
| dpawlik | arxcruz: please also take a look on https://review.opendev.org/c/openstack/ci-log-processing/+/859028 | 08:29 |
| akahat | ysandeep, we seen that only in 17.1/9, thank you for +w marios | 08:36 |
| ysandeep | akahat, ack | 08:44 |
| *** ysandeep is now known as ysandeep|lunch | 08:44 | |
| arxcruz | dpawlik great, what's the next step to have the dashboard? | 08:50 |
| soniya29 | chkumar|rover, frenzyfriday|ruck, bhagyashris, rhel8 featureset001-internal-rhos-17.1, do we have any bug reported about it? | 08:52 |
| marios | ysandeep|lunch: thanks for checking i missed your message earlier - so we aren't consuming the current-tripleo-rdo anywhere but probably this is a bigger discussion that we should have and we can followup later, will comment on the patch | 08:54 |
| bhagyashris | soniya29, yes | 08:54 |
| marios | akahat: thanks | 08:54 |
| soniya29 | bhagyashris, can you pass the link? | 08:55 |
| marios | oooci o/ easy vote needed please https://review.opendev.org/c/openstack/tripleo-ci/+/858092 when you have time | 09:10 |
| dpawlik | arxcruz: so added permission for the logstash role | 09:11 |
| dpawlik | arxcruz: deployed new image | 09:11 |
| dpawlik | arxcruz: so soon will be available | 09:12 |
| dpawlik | just need to create a new index pattern | 09:12 |
| arxcruz | dpawlik++ | 09:16 |
| dpawlik | ]: INFO:root:Working on /mnt/logscraper/openstack/5cc6e4e83ced4b088a523ebc16fee504/testrepository.subunit | 09:29 |
| dpawlik | ]: CRITICAL:root:An error occurred on sending message to Opensearch 'utf-8' codec can't decode byte 0xb3 in position 0: invalid start byte | 09:29 |
| dpawlik | arxcruz ^^ | 09:29 |
| dpawlik | can lukas make a patch, or I should? | 09:30 |
| arxcruz | dpawlik pinging him | 09:30 |
| dpawlik | the header seems to be not ok in the subunit file | 09:31 |
| *** ysandeep|lunch is now known as ysandeep | 09:50 | |
| *** rlandy|out is now known as rlandy | 10:38 | |
| rlandy | chkumar|rover: frenzyfriday|ruck; hey - need help with anything? | 10:39 |
| chkumar|rover | rlandy: currently nope | 10:39 |
| chkumar|rover | rlandy: updated your config constriant patch | 10:39 |
| rlandy | chkumar|rover: k - looking | 10:40 |
| frenzyfriday|ruck | rlandy, ovb jobs are failing on wallabys and master. I am running testprojs with your fix. Some failures look different but they are still keystone timeout related. Rest is on the hackmd. I havent finished with the components yet | 10:40 |
| rlandy | frenzyfriday|ruck: chkumar|rover: ok - I'll continue looking at two things - the keystone failures and the kvm nodeset | 10:42 |
| chkumar|rover | need to check 16.2 line | 10:42 |
| chkumar|rover | rlandy: cool! | 10:42 |
| rlandy | chkumar|rover: frenzyfriday|ruck: want to try https://review.rdoproject.org/r/c/config/+/45232? I can merge | 10:42 |
| rlandy | and we can see if it helps | 10:42 |
| frenzyfriday|ruck | rlandy, trying that for wallaby here: https://review.rdoproject.org/r/c/testproject/+/44965/6/.zuul.yaml | 10:43 |
| rlandy | k - let me know | 10:44 |
| rlandy | chkumar|rover: 16.2 should promote | 10:44 |
| frenzyfriday|ruck | sorry wrong link, I'll update the patch | 10:44 |
| rlandy | it's just to see if the components then retrigger on jenkins | 10:44 |
| rlandy | yep 16.2 juts promoted | 10:45 |
| Tengu | ysandeep: I'm back - I'll push the patch about the ExtraFirewallRules, but it will once more require a lot of testing imho. | 10:45 |
| Tengu | probably with some custom scenarios in order to ensure things are properly done. | 10:45 |
| Tengu | though... wait. | 10:46 |
| Tengu | https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/tripleo-firewall/tripleo-firewall-baremetal-ansible.yaml#L44-L45 it should merge actually. | 10:47 |
| rlandy | frenzyfriday|ruck: idk this helps: https://review.rdoproject.org/r/c/testproject/+/44965 - left lucca notes on card - will follow up with him | 10:47 |
| Tengu | did we just uncover a bug? :D | 10:47 |
| chkumar|rover | rlandy: frenzyfriday|ruck I am looking into this https://bugs.launchpad.net/tripleo/+bug/1990480 | 10:47 |
| chkumar|rover | skipping the test for now | 10:48 |
| rlandy | chkumar|rover: ack - skip for now | 10:48 |
| rlandy | frenzyfriday|ruck; chkumar|rover: lucca is asking us to hold a node for investigation | 10:49 |
| rlandy | we will need to restart the job and then stop the ovb clean up | 10:49 |
| rlandy | monday night be better | 10:49 |
| frenzyfriday|ruck | rlandy, ack, I am holding a node for https://review.rdoproject.org/r/c/testproject/+/44965 | 10:49 |
| frenzyfriday|ruck | oh, ok, yep, I'll do that on monday | 10:49 |
| rlandy | frenzyfriday|ruck: ok - if you have the node, it's fine | 10:50 |
| rlandy | but the script will remove the stack | 10:50 |
| frenzyfriday|ruck | yeah, I havent held the node yet. I'll disable the cleanup script and hold the node on monday | 10:50 |
| rlandy | frenzyfriday|ruck: ok - I won't be here so you'll need to help him out | 10:51 |
| rlandy | you can reach him on #rhos-dev | 10:51 |
| rlandy | frenzyfriday|ruck: you'll also need to include var to stop stack deletion in the job itself | 10:52 |
| frenzyfriday|ruck | rlandy, yep | 10:52 |
| Tengu | ysandeep: I'll run some tests here to try to understand the whole thing. but apparently, there's a bug. | 10:53 |
| rlandy | frenzyfriday|ruck: chkumar|rover: k - card updated | 10:53 |
| frenzyfriday|ruck | rlandy, this is the cix right? https://trello.com/c/I7Rep3ZC/2725-cixlp1990415tripleociproa-ovb-jobs-are-failing-deployment-failure-running-exec-keystonebootstrap-lost-connection-to-mysql-server | 10:53 |
| rlandy | frenzyfriday|ruck: https://trello.com/c/fMby586x/2679-cixlp1987092tripleociproa-pacemaker-performance-causes-intermittent-galera-issues-in-loaded-ci-env | 10:54 |
| frenzyfriday|ruck | ack, I'll follow up on monday | 10:54 |
| rlandy | the other one may be a duplicate | 10:54 |
| ysandeep | Tengu, ack o/ I was going through https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/role_specific_parameters.html , and looks like we override in other cases, but if we can create something where we can override or append that will be great. | 10:58 |
| Tengu | ysandeep: maybe replacing the map_replace by a map_merge or something... | 10:59 |
| Tengu | the comment isn't really clear, to be fair. | 10:59 |
| Tengu | " # Merging role-specific parameters (RoleParameters) with the default parameters. | 10:59 |
| Tengu | " | 10:59 |
| Tengu | for me, merging means... well... merging. but according to the doc you linked, it's more "overriding" | 11:00 |
| Tengu | ysandeep: in order to get an actual merge, it would require something like <RoleName>ExtraFirewallRules afaik | 11:01 |
| Tengu | and changing the behaviour of the <RoleName>Parameters would create an inconsistency. | 11:01 |
| Tengu | better keeping it like that. | 11:01 |
| ysandeep | true, with some customer reusing older template on newer version - changing behavior might cause some issue | 11:04 |
| ysandeep | thanks for the <RoleName>ExtraFirewallRules hint | 11:04 |
| chkumar|rover | rlandy: frenzyfriday|ruck https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859074 | 11:05 |
| Tengu | ysandeep: not sure we actually would need the <RoleName>ExtraFirewallRules thingy. "it's working fine like that", and I never heard anyone needing this level of configuration for the firewall. | 11:06 |
| bhagyashris | rlandy, hi facing this isue on rhos17.1 on rhel8 ovb jobs https://trello.com/c/O4fPuAsk/2638-cixbz2110535osp171rhel8error-code-8-invalid-argument-the-machine-pc-q35-rhel900-is-not-supported-by-emulator | 11:06 |
| Tengu | so I propose to just keep things as they are, in order to not make the firewall management over-complicated. | 11:06 |
| rlandy | chkumar|rover: w+'ed | 11:12 |
| chkumar|rover | thanks! | 11:12 |
| rlandy | bhagyashris: lol - finally got there | 11:12 |
| rlandy | ok - let me dig up my patch after review time | 11:12 |
| bhagyashris | rlandy, hahaha i see the bug is closed | 11:12 |
| bhagyashris | ack | 11:12 |
| rlandy | bhagyashris: yeah - because we solved it on multinode | 11:12 |
| rlandy | and then found it on ovb | 11:13 |
| bhagyashris | ahh ok | 11:13 |
| rlandy | there are other options in the card | 11:13 |
| rlandy | will have to refresh myself on that | 11:13 |
| pojadhav | folks, review time | 11:16 |
| marios | review call now if someone wants to join us we are starting in a sec | 11:16 |
| Tengu | folks, care to add this to your review queue? https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858868 marios, rlandy, chkumar|rover maybe? :) | 11:22 |
| Tengu | uho. and... well. we're ready. | 11:22 |
| Tengu | rlandy, marios fyi, we're ready to switch the firewall engine! (even without the patch above -^^) | 11:23 |
| rlandy | jm1: can you review https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45206 | 11:23 |
| rlandy | Tengu: adding to review hackmd | 11:23 |
| marios | Tengu: ack we will add to reviews ^^ | 11:23 |
| Tengu | rlandy, marios do you want me to come once more to the community call on Tuesday, or.. ? | 11:23 |
| Tengu | marios, rlandy thanks :) | 11:23 |
| rlandy | Tengu: you are welcome to - I will be on PTO - pls coordinate with pojadhav | 11:24 |
| Tengu | rlandy: well deserved PTO :) | 11:25 |
| rlandy | Tengu: religious holiday ... how much that is PTO is up for debate :) | 11:25 |
| Tengu | ah. well. | 11:25 |
| * Tengu avoids this kind of debate | 11:25 | |
| rlandy | smart move - | 11:28 |
| Tengu | :) | 11:28 |
| Tengu | rlandy, pojadhav added a topic to next week agenda (created the entry as well in the hackmd) | 11:30 |
| rlandy | Tengu: thanks | 11:51 |
| rlandy | marios: w+'ed https://review.rdoproject.org/r/c/rdo-jobs/+/45036 | 11:51 |
| rlandy | you can fix anything in a a follow up but that's a start | 11:51 |
| marios | great thankx | 11:55 |
| marios | zuul -2 as we need the tripleo-ci one to merge first so will need recheck later | 11:55 |
| * bhagyashris stepping out for 1 hr | 12:04 | |
| bhagyashris | rlandy, me stepping out for hr will ping you once back | 12:04 |
| rlandy | bhagyashris: np - on 1-1s now | 12:06 |
| *** frenzyfriday|ruck is now known as frenzyfriday|ruck|food | 12:26 | |
| *** amoralej is now known as amoralej|lunch | 12:30 | |
| soniya | rlandy, are we moving 1:1? | 12:32 |
| soniya | pinged you in pvt | 12:32 |
| rlandy | soniya: I can 2pm | 12:32 |
| rlandy | meet at | 12:32 |
| soniya | rlandy, okay no problem..let me move the meeting to 2 pm utc | 12:33 |
| soniya | rlandy, it seems only you can move the meeting on calendar | 12:34 |
| rlandy | soniya: moved | 12:34 |
| soniya | rlandy, thanks :) | 12:34 |
| pojadhav | Tengu, thanks for update !! | 12:49 |
| chkumar|rover | marios: rlandy config patch https://review.rdoproject.org/r/c/config/+/45232 good to go | 12:53 |
| rlandy | chkumar|rover: k - let's try this | 12:57 |
| chkumar|rover | tested on both cs8 and cs9 | 12:57 |
| chkumar|rover | both works | 12:57 |
| marios | chkumar|rover: k looks like it merged was too late for revote | 12:59 |
| chkumar|rover | marios: rlandy thank you :-) | 13:00 |
| Tengu | pojadhav: np - trying to keep CI in the loop with this kind of changes :) | 13:00 |
| pojadhav | Tengu, its a great thing.. thank you for your efforts.. :) | 13:01 |
| pojadhav | rlandy, https://review.opendev.org/c/openstack/tripleo-ci/+/856051 | 13:05 |
| *** frenzyfriday|ruck|food is now known as frenzyfriday|ruck | 13:11 | |
| *** amoralej|lunch is now known as amoralej | 13:18 | |
| * bhagyashris back | 13:19 | |
| bhagyashris | rlandy, i am back should we meet ? | 13:21 |
| chkumar|rover | so rhos-16.2 rhel-8 promoted . | 13:26 |
| rlandy | bhagyashris: got meetings for another hour | 13:27 |
| rlandy | can meet after that | 13:27 |
| bhagyashris | ok | 13:30 |
| *** pojadhav is now known as pojadhav|afk | 13:31 | |
| amoralej | rlandy, https://review.opendev.org/c/openstack/tripleo-quickstart/+/859090 is adding to oooq config for master the temporary repo with a-o-c 2.0.0 preview and last release of openstacksdk | 13:47 |
| amoralej | what's the best way of testing all oooq configs with it? | 13:47 |
| amoralej | review to testproject or some other upstream repo? | 13:47 |
| amoralej | jm1, ^ | 13:48 |
| jm1 | rcastillo: ^ | 13:48 |
| jm1 | amoralej: great, thank you so much! i think rcastillo knows most about how to marry a-c-o and tripleo jobs | 13:49 |
| jm1 | ..besides rlandy, of course | 13:49 |
| amoralej | good, nice! | 13:49 |
| rlandy | amoralej: nice - will review | 13:54 |
| rlandy | rcastillo has a patch with ovb and multinode jobs | 13:55 |
| rlandy | a testproject | 13:55 |
| rlandy | so if we add a standalone to that | 13:55 |
| rlandy | and then set the vars | 13:55 |
| rlandy | we can test that out | 13:55 |
| rlandy | jm1: amoralej: ^^ | 13:55 |
| rlandy | let's give that a shot when rcastillo is in | 13:55 |
| amoralej | so adding a depends-on to that this review should be all we need | 13:56 |
| rlandy | yeah | 13:56 |
| amoralej | yep, let's wait for rcastillo | 13:56 |
| * rlandy check url | 13:56 | |
| amoralej | thanks rlandy | 13:56 |
| rlandy | yeah .. https://trunk.rdoproject.org/centos9-master/aoc-temp/ | 13:57 |
| rlandy | marios: ^^ will need that for zed files | 13:57 |
| rlandy | when we qualify it | 13:57 |
| amoralej | i've also created for zed | 13:57 |
| amoralej | https://trunk.rdoproject.org/centos9-zed/aoc-temp/ | 13:57 |
| amoralej | but i guess it's better to start with master ... | 13:57 |
| marios | o/ | 13:58 |
| rlandy | amoralej: jm1: in fact - jobs will run on this change | 13:58 |
| amoralej | if we want to do this permanently we create a parameter to enable/disable this repo | 13:58 |
| rlandy | so I see all these repos are in fact enabled | 13:58 |
| amoralej | rlandy, yes, but not all jobs, right? | 13:58 |
| amoralej | yes, that will be the first check | 13:58 |
| rlandy | https://zuul.opendev.org/t/openstack/status#859090 | 13:59 |
| rlandy | is a list of what runs now | 13:59 |
| amoralej | i'm monitoring | 13:59 |
| rlandy | and it will kick a few ovb jobs as well | 13:59 |
| amoralej | nice | 13:59 |
| rlandy | we just need to check that these jobs in fact do pick up and USE the repo | 13:59 |
| rlandy | and the right version of collections | 13:59 |
| rlandy | and they are not defaulting to the old ones | 14:00 |
| amoralej | a-o-c is installed in host and containers? | 14:00 |
| amoralej | i added priority=1, we'll see if that's enough | 14:00 |
| rlandy | so check jobs use provider | 14:00 |
| rlandy | so containers will be rebuilt | 14:01 |
| chkumar|rover | anyone wants to join happy hour call? | 14:01 |
| rlandy | got 1-1 now | 14:01 |
| chkumar|rover | ok | 14:01 |
| *** dasm|off is now known as dasm | 14:02 | |
| dasm | o/ | 14:02 |
| *** ysandeep is now known as ysandeep|out | 14:02 | |
| jm1 | dasm: o/ | 14:05 |
| marios | rlandy: chkumar|rover: please when you have time check https://bugs.launchpad.net/tripleo/+bug/1990012/comments/2 and https://review.rdoproject.org/r/c/config/+/45259 (for mixed OS cix card) | 14:08 |
| marios | pojadhav|afk: ^^ | 14:08 |
| marios | sorry pojadhav|afk was meant for frenzyfriday|ruck ^^^ | 14:08 |
| * marios fetch coffee | 14:09 | |
| jm1 | marios: dont wake up pojadhav|afk, it is nearly 8pm for her :D | 14:10 |
| frenzyfriday|ruck | marios, yep we can merge it and test. Sorry, where is job.mixed_os_stable_version defined? | 14:12 |
| chkumar|rover | frenzyfriday|ruck: https://review.rdoproject.org/codesearch/?q=mixed_os_stable_version&i=nope&files=&repos= | 14:13 |
| frenzyfriday|ruck | oh cool! thanks | 14:14 |
| marios | frenzyfriday|ruck: only comes from the mixed os jobs https://codesearch.rdoproject.org/codesearch/?q=mixed_os_stable_version | 14:14 |
| marios | thanks chkumar|rover missed it | 14:14 |
| marios | we don't set it anywhere else so 'normal' case should be unaffected | 14:14 |
| marios | :) jm1 | 14:14 |
| chkumar|rover | rlandy: please +w https://review.rdoproject.org/r/c/config/+/45259 | 14:18 |
| rlandy | will do after 1-1 | 14:19 |
| marios | thanks chkumar|rover rlandy frenzyfriday|ruck | 14:20 |
| *** pojadhav|afk is now known as pojadhav | 14:23 | |
| pojadhav | marios, np :) | 14:23 |
| pojadhav | jm1, :D | 14:23 |
| chkumar|rover | rlandy: please +w https://review.opendev.org/c/openstack/tripleo-ci/+/859071 and https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/859072 | 14:29 |
| rlandy | off meetings - looking | 14:33 |
| chkumar|rover | rlandy: please keep an eye on rhos-17.1 rhel-9 | 14:33 |
| chkumar|rover | waiting on fs01 to finish | 14:33 |
| chkumar|rover | all passed | 14:33 |
| rlandy | chkumar|rover; ack - will do | 14:33 |
| rlandy | chkumar|rover; w+'ed both patches | 14:34 |
| chkumar|rover | rlandy: thanks | 14:34 |
| rlandy | and marios patch above | 14:34 |
| chkumar|rover | rlandy: once https://review.opendev.org/q/topic:tripleo-ansible-ee merges, we can close https://trello.com/c/jODWDYTd/2717-cixbz2128230osp171ospcomposeopenstack-containersdockerfile-arg-command-before-from-line-breaks-downstream-container-builds this | 14:34 |
| rlandy | chkumar|rover: if mixed jobs stars reporting correctly, pls add back to criteria | 14:34 |
| rlandy | ack | 14:35 |
| chkumar|rover | rlandy: sure will do that on monday! | 14:35 |
| chkumar|rover | happy weekend people, see ya on monday! | 14:36 |
| marios | thanks rlandy - will run a test now to make sure we're OK with config/+/45259 | 14:39 |
| rlandy | k | 14:39 |
| marios | rlandy: there https://review.rdoproject.org/r/c/testproject/+/44234/11/.zuul.yaml (added a standalone to sanity check the 'normal' case too) | 14:42 |
| dasm | jm1: any idea what "toolbox" is and what is its ip? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/25873 | 14:54 |
| dasm | it's changed, so telegraf is failing | 14:55 |
| dasm | i'm not sure if we even need that anymore | 14:55 |
| jm1 | dasm: toolbox is in infra-setup. somebody did a fresh install on vexxhost earlier this year (or end of last year?). not sure who it was but we have a jira card to get toolbox synced to our repo again | 15:05 |
| jm1 | dasm: maybe bhagyashris knows more? | 15:05 |
| dasm | mhm | 15:06 |
| dasm | next step: prevent from doing that manually | 15:07 |
| jm1 | dasm: i know, not a satisfying answer :( | 15:07 |
| jm1 | dasm: yes, please. restore what we had when sagi was still here ;) | 15:07 |
| jm1 | dasm: but infra-setup is a good start. it should give you an idea what it was meant to do. you could log in to vexxhost to find its real ip and then try to ssh into it. if ansible is running there, then you should be able to login | 15:08 |
| jm1 | dasm: i know something was running there, but i never had time to take a look | 15:09 |
| rlandy | chkumar|rover: frenzyfriday|ruck; 17.1 on rhel-9 all passed | 15:10 |
| rlandy | should promote | 15:10 |
| * jm1 bbl | 15:15 | |
| sshnaidm | dasm, jm1 toolbox is here: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/toolbox it's for running some cron jobs on ovb and publish metrics about ovb clouds. | 15:20 |
| dasm | sshnaidm: ack. ip address has changed and telegraf is erroring on metrics collection. i'm trying to find what the ip of toolbox is right now. | 15:21 |
| dasm | i recently started looking into different pieces of our infra to understand how it's designed | 15:22 |
| marios | rlandy: tests are still running but not sure it fixed the issue - but at least it didn't break the non mixed-os jobs so no need to revert the test (it is always executing the 'old' task and never my new one). i'll have to revisit on monday | 15:23 |
| marios | dasm: i think ysandeep|out worked on the toolbox most recently .. can you find info on bw? | 15:24 |
| marios | o/ sshnaidm :) | 15:24 |
| sshnaidm | marios, \o :) | 15:24 |
| dasm | marios: i see some info. i'll give it a try | 15:24 |
| dasm | i started with vexxhost but i couldn't find the right domain | 15:24 |
| dasm | there are so many of them | 15:24 |
| sshnaidm | dasm, it was "infra-tripleo" once | 15:26 |
| dasm | ok, that sounds promising | 15:26 |
| rlandy | there is an internal toolbox and an upstream one | 15:27 |
| rlandy | dasm: ^^ | 15:27 |
| rlandy | which one are you looking for? | 15:27 |
| dasm | rlandy: currentlu upstream. but i'm gonna need downstream as well | 15:28 |
| rlandy | dasm: sending you | 15:28 |
| dasm | ack | 15:28 |
| *** amoralej is now known as amoralej|off | 15:37 | |
| marios | o/ oooci have a good weekend | 15:45 |
| *** marios is now known as marios|out | 15:46 | |
| dasm | marios|out: o/ | 15:56 |
| *** jpena is now known as jpena|off | 16:22 | |
| rlandy | rcastillo: hey - you around? | 16:23 |
| rlandy | https://zuul.opendev.org/t/openstack/status#859090 | 16:24 |
| rlandy | rcastillo: ^^ pls see | 16:24 |
| rlandy | amoralej|off's patch plus testing | 16:24 |
| rlandy | we will need your OVB jobs on this as well | 16:24 |
| frenzyfriday|ruck | rlandy, do you know if we already saw this last week: Failed to download packages: httpd-tools-2.4.53-7.el9.x86_64 ? I think I saw it somewhere but it resolved | 16:24 |
| rlandy | frenzyfriday|ruck: you see that commonly? | 16:25 |
| rlandy | or once off? | 16:25 |
| rlandy | there was a mirrors fix | 16:25 |
| frenzyfriday|ruck | one off, but for 2 jobs I ran in the same testproj. rerunning to be sure | 16:25 |
| *** rlandy is now known as rlandy|brb | 16:57 | |
| frenzyfriday|ruck | rlandy, I am leaving for the day in some time. The ovb jobs are failing on wallaby and master. I'll hold the node and follow up on Monday. On the component there are some inconsistent failures. I have testprojects running (link on hackmd) | 16:58 |
| *** rlandy|brb is now known as rlandy | 17:20 | |
| rlandy | frenzyfriday|ruck: thanks | 17:21 |
| dasm | jm1: fyi, toolbox nginx instance died some time ago, hence telegraf spilling errors | 17:32 |
| dasm | we're gonna need to do something with that | 17:32 |
| dasm | at least some kind of simple monitoring | 17:32 |
| dasm | it seems to be (was?) under heavy automated scanning. | 17:35 |
| rlandy | lunch - brb | 17:41 |
| rcastillo | oops, sorry | 17:52 |
| rcastillo | so amoralej's patch works well. Standalone is failing because quickstart.sh is still installing the older collections | 18:01 |
| rcastillo | because it installs from galaxy | 18:02 |
| rlandy | rcastillo: ^^ fixable? | 18:09 |
| rcastillo | we can add something like a conditional in quickstart maybe? It's trickier since this is before repo-setup | 18:10 |
| jm1 | dasm: what kind of heavy automated scanning? periodic unsuccessful ssh login attempts? | 18:17 |
| dasm | jm1: nope. just attempts to exploit. | 18:21 |
| jm1 | dasm: how did you find that out? | 18:25 |
| dasm | check logs | 18:26 |
| dasm | nginx is spilling them out | 18:26 |
| dasm | it hasn't been updated quite some time. it needs to be brought up to date | 18:26 |
| jm1 | dasm: the host itself seems to be up to date and working as expected, e.g. ansible is running and pulling stuff from the repo | 18:28 |
| dasm | it needs some TLC | 18:29 |
| jm1 | dasm: TLC? | 18:29 |
| dasm | Tender Loving Care | 18:30 |
| dasm | it's working, but it's been neglected a bit | 18:30 |
| jm1 | dasm: what needs love is nginx container, in particular this has to be replaced with a more robust solution https://github.com/rdo-infra/ci-config/blob/43d194a98c0aafe40559706d85a538bdf60e9dff/ci-scripts/infra-setup/roles/toolbox/tasks/main.yml#L39 | 18:31 |
| jm1 | dasm: the host is (dnf) updated from ansible every 5 minutes | 18:31 |
| dasm | indeed | 18:31 |
| jm1 | dasm: there is a nice ansible collection for managing podman, developed by someone you might know, sshnaidm ;) | 18:32 |
| dasm | indeed | 18:32 |
| jm1 | dasm: :D | 18:32 |
| jm1 | dasm: for a quick solution you could simply stop and rm the podman container and image. ansible will recreate it after 5 minutes | 18:33 |
| jm1 | dasm: ah you know that | 18:34 |
| jm1 | dasm: i should better get off, its late 🙈 | 18:34 |
| dasm | jm1: friday night :) | 18:34 |
| rcastillo | have a good weekend jm1 | 18:36 |
| dasm | o/ jm1 | 18:36 |
| jm1 | dasm: you will take care of toolbox? | 18:36 |
| dasm | yup | 18:36 |
| jm1 | dasm: thanks :) | 18:36 |
| dasm | one way or another | 18:36 |
| dasm | stonith is also a solution ;) | 18:36 |
| jm1 | dasm: it looks better than i expected, so please dont stonith 😬 | 18:37 |
| dasm | nah. no one even noticed it not working ;) | 18:38 |
| jm1 | dasm: it is still doing stuff, only nginx wasnt working | 18:39 |
| jm1 | dasm: or was it? | 18:39 |
| dasm | nginx was dead. | 18:39 |
| jm1 | dasm: nginx is running | 18:40 |
| dasm | because i started it :P | 18:40 |
| jm1 | dasm: ah :D | 18:40 |
| jm1 | dasm: how about the two cron jobs? | 18:40 |
| dasm | i haven't checked anything yet. just finding my way through all the stuff | 18:40 |
| dasm | jm1: aren't you suppose to clock out already? :) | 18:41 |
| jm1 | dasm: hehe ok. you handle toolbox, i handle sleep | 18:42 |
| dasm | sounds good | 18:42 |
| dasm | ttyl | 18:42 |
| jm1 | dasm: :P | 18:42 |
| jm1 | dasm, rcastillo, #oooq: have a nice weekend :) | 18:42 |
| dasm | o/ jm1 | 18:42 |
| rcastillo | o/ | 18:42 |
| dasm | jm1[m]: fyi, it wasn't *that* nice with nginx container. Like I said, it needs TLC. At the moment we have latest, greatest version running. | 19:12 |
| * dasm => offline | 21:10 | |
| dasm | see you! | 21:10 |
| *** dasm is now known as dasm|off | 21:10 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!