Thursday, 2022-08-18

opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use group var to detect instance ha  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85356800:13
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use ansible group var to detect instance ha  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85356800:18
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: ovn: Clean up unused items from hieradata  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85357000:42
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: ovn: Clean up unused items from hieradata  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85357000:47
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Nova: Use ansible host var to detect canonical FQDN  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85357101:07
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Nova: Use ansible host var to detect canonical FQDN  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85357101:09
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Allow setting ROOT_LABEL from environment  https://review.opendev.org/c/openstack/diskimage-builder/+/85357301:19
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Allow setting ROOT_LABEL from environment  https://review.opendev.org/c/openstack/diskimage-builder/+/85357301:23
opendevreviewIan Wienand proposed openstack/diskimage-builder master: [dnm] trigger builds with different root labels  https://review.opendev.org/c/openstack/diskimage-builder/+/85357501:23
opendevreviewIan Wienand proposed openstack/diskimage-builder master: [dnm] trigger builds with different root labels  https://review.opendev.org/c/openstack/diskimage-builder/+/85357501:28
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use ansible group var to detect instance ha  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85356802:25
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use ansible group var to detect instance ha  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85356802:25
opendevreviewMerged openstack/tripleo-heat-templates stable/train: [train-only] Don't update hash when using paunch  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85237402:45
opendevreviewMerged openstack/tripleo-quickstart master: Bump ceph repos to Quincy  https://review.opendev.org/c/openstack/tripleo-quickstart/+/85273102:45
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Remove leftover of certmonger management by puppet  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85358303:50
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Remove leftover of certmonger management by puppet  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85358303:54
*** ysandeep|out is now known as ysandeep05:29
*** ysandeep is now known as ysandeep|ruck06:01
opendevreviewyatin proposed openstack/tripleo-upgrade master: Honor undercloud_custom_env_files for oooq deployments  https://review.opendev.org/c/openstack/tripleo-upgrade/+/85358906:03
opendevreviewyatin proposed openstack/tripleo-quickstart master: Make undercloud upgrade jobs to deploy ml2 ovs  https://review.opendev.org/c/openstack/tripleo-quickstart/+/85350306:05
*** amoralej|off is now known as amoralej06:16
opendevreviewCedric Jeanneret proposed openstack/tripleo-heat-templates stable/wallaby: Ensure mode is correct for the tmp directory  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85344306:39
opendevreviewCedric Jeanneret proposed openstack/tripleo-ansible stable/wallaby: tripleo_auditd: Clear unused default of tripleo_auditd_rules  https://review.opendev.org/c/openstack/tripleo-ansible/+/85329706:40
opendevreviewMarios Andreou proposed openstack/tripleo-quickstart master: Revert "Bump ceph repos to Quincy"  https://review.opendev.org/c/openstack/tripleo-quickstart/+/85344406:44
mariosfpantano: ysandeep|ruck: ^^ 06:44
fpantanomarios: o/06:45
fpantanomarios: the testproject should be https://review.rdoproject.org/r/c/testproject/+/3672706:45
marioso/ fpantano we are discussing in oooq we were looking for that ^^^ 06:45
fpantanobut I'm not sure how dviroel|afk triggers it 06:45
mariosyeah fpantano ysandeep|ruck found it but it is abandoned06:46
mariosand looks like the tripleo-ci-promote-ceph-daemon-tag-to-current-ceph-master was not run 06:46
fpantanoand I'm not sure why it's abandoned06:46
mariosso this is why we think we have to post the revert 06:46
mariosotherwise the gate is broken all day for scen1/406:46
fpantanomarios: should we first try to run testproject?06:46
mariosfpantano: sure... the revert needs to run zuul check first anyway so may as well get that started ... and ysandeep|ruck is looking into the testproject too 06:47
mariosfpantano: but mainly was hoping you would have more context 06:47
marioslike why was it abandoned06:47
fpantanomarios: yeah, testproject should be run right before that change merges06:47
opendevreviewBrendan Shephard proposed openstack/tripleo-common master: Add multi-rhel support for Nova services  https://review.opendev.org/c/openstack/tripleo-common/+/85321106:47
fpantanono, I'm not aware about the reason it's abandoned, but I remember it was used to to test quincy06:48
mariosfpantano: ysandeep|ruck: don't think we can restore dviroel|afk testproject but ysandeep|ruck will you post a new one?06:48
mariosysandeep|ruck: after that runs, if we see improvement then fine abandon https://review.opendev.org/c/openstack/tripleo-quickstart/+/853444 06:48
mariosotherwise we can go with ^^ and send note to mailing list06:49
mariossounds ok fpantano ysandeep|ruck ? 06:49
ysandeep|ruckmarios, sounds good06:49
mariosysandeep|ruck: need me to post it (you're on your own now so let me know if need that or something else)06:49
fpantano+1 on my side06:50
ysandeep|ruckmarios, thanks ! I will take care of it.06:50
mariosthank you ysandeep|ruck 06:50
fpantanomarios: ysandeep|ruck we can also merge https://review.rdoproject.org/r/c/rdo-jobs/+/4451806:50
fpantanowhich is ready06:50
fpantanoit will block any new pacific promotion in master and looks for quincy regex 06:50
fpantanoin the stable container images06:50
marioslets not confuse things first lets deal with broken gate06:51
fpantanoack06:51
mariosfpantano: or we need that for the gate? looking 06:51
fpantanoyeah, as long as it's manually triggered it's fine06:51
fpantanowe can merge it later06:51
mariosfpantano: oh i see... so it would run the quincy promotion 06:52
mariosfpantano: instead of doing the testproject? 06:52
fpantanoyeah06:52
mariosfpantano: yeah i'd rather stick with testproject for now make sure its all good with quincy then we can switch that promotion pipeline?06:52
fpantanomarios: see https://paste.opendev.org/06:52
mariosfpantano: missing paste? 06:53
fpantano//paste.opendev.org/show/b5BKmWsQwRGsY7X60Gt8/ 06:53
fpantanohttps://paste.opendev.org/show/b5BKmWsQwRGsY7X60Gt8/06:53
marioslooking06:53
* fpantano needs some coffee06:53
mariosack fpantano lets run the testproject get gate fixed then we can merge that for tomorrow's run 06:54
fpantanoyeah06:55
ysandeep|ruckI think dviroel|afk abandoned this because he wanted the tq repo patch to first go in.06:57
ysandeep|ruckhttps://review.rdoproject.org/r/c/testproject/+/36727/57/.zuul.yaml06:57
ysandeep|rucklet me promote the latest container with depends on  https://review.rdoproject.org/r/c/rdo-jobs/+/44518 , and that should do the job.06:58
mariosack ysandeep|ruck as discussed it seems the tqe merged ~2 hours later but... he still why abandon? oh perhaps it was running already and too early 06:58
mariosbut should have left some note/email tho06:58
opendevreviewCedric Jeanneret proposed openstack/tripleo-ansible master: Ensure nftables is installed on the nodes  https://review.opendev.org/c/openstack/tripleo-ansible/+/85325206:58
ysandeep|ruckmarios, yeah he ran that too early06:58
mariosso he had to abandon it cos he thought it would break the gate by promoting to early before tqe06:58
opendevreviewCedric Jeanneret proposed openstack/tripleo-ansible master: We don't need the final drop rule anymore  https://review.opendev.org/c/openstack/tripleo-ansible/+/85328206:58
ysandeep|ruck^^ agree 06:58
fpantanoyeah, and see https://review.rdoproject.org/r/c/testproject/+/36727/56..57/.zuul.yaml06:59
fpantanohe updated the patch right before abandoning it06:59
ysandeep|ruckfpantano, yeah so we need a testproject with ps 57 now.. let me post that06:59
fpantanobecause the last update is the part which triggers the promotion playbook06:59
fpantanoysandeep|ruck: right06:59
mariosso lets run the promotion ysandeep|ruck and hopefully will unblock us. otherwise revert is ready if needed but would rather not go that way it will take most of the day to merge :/07:00
*** jgilaber_ is now known as jgilaber07:01
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Apache: Use only required modules  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85348007:02
ysandeep|ruckmarios, fpantano testproject is up: https://review.rdoproject.org/r/c/testproject/+/4456207:03
mariosthanks ysandeep|ruck 07:03
fpantanoack thanks, let's see how it goes07:03
fpantanoon the first log I see -> https://paste.opendev.org/show/bfD614gfcAnSNJYZEJbK/07:10
fpantanomarios: ysandeep|ruck and a new container has been pushed https://quay.rdoproject.org/repository/tripleomastercentos9/daemon?tab=tags07:12
fpantanoa minute ago ^07:12
ysandeep|rucklets pull that container can inspect that version is quincy to be double sure07:13
fpantanoysandeep|ruck: yeah you can double check w/ podman run --rm --entrypoint=ceph quay.rdoproject.org/repository/tripleomastercentos9/daemon:current-ceph --version07:15
ysandeep|ruckdownloading image here.. with podman run --rm --entrypoint=ceph quay.rdoproject.org/tripleomastercentos9/daemon:current-ceph --version 07:18
opendevreviewFrancesco Pantano proposed openstack/tripleo-ansible master: Skip CephIngress daemon deploy  https://review.opendev.org/c/openstack/tripleo-ansible/+/85253207:18
fpantanolet's see now sc001 and sc004 ^07:18
fpantanoalso marios ysandeep|ruck  the location containing the last master discovered tag has been updated: https://images.rdoproject.org/ceph/daemon/master/current-ceph07:21
fpantanoso ... we should have quincy in master now07:21
ysandeep|ruckwohoo.. ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)07:22
ysandeep|ruck^^ inside container..07:22
ysandeep|rucksc01/04 should pass now - i have also trigger a testproject to confirm07:22
fpantano++07:22
fpantanoack07:22
fpantanothanks07:22
fpantanomarios: ^^07:23
fpantanowe have Quincy \o/07:23
mariosgreat thanks fpantano ysandeep|ruck abandoning the revert then 07:24
fpantano++07:24
opendevreviewManojkatari proposed openstack/tripleo-heat-templates master: Disable haproxy for etcd based on CinderEtcdLocalConnect  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85345907:24
opendevreviewManojkatari proposed openstack/tripleo-heat-templates master: Disable haproxy for etcd based on CinderEtcdLocalConnect  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85345907:29
ade_leeysandeep|ruck, Tengu , tkajinam chandankumar any idea what might be going on with this error -- 07:34
ade_leetemplate error while templating string: cannot import name 'environmentfilter' from 'jinja2.filters' (/usr/lib/python3.9/site-packages/ansible/_vendor/jinja2/filters.py)\n  line 0. String: {{ 'A' if record_value | ansible.utils.ipv4 else 'AAAA' }}"07:34
ysandeep|ruckhttps://github.com/readthedocs/readthedocs.org/issues/9038  ->> This may be due to the latest release of Jinja2. Requiring Jinja2<3.1 fixes my issue. 07:36
ade_leeyeah - I just googled that .. maybe something got updated recently?07:36
ysandeep|ruckade_lee, You are seeing this error in CI? 07:37
*** jpena|off is now known as jpena07:38
ade_leeysandeep|ruck, yeah https://zuul.opendev.org/t/openstack/build/5272aaca59a14b758dd5a2c190ccc21807:38
ade_leefor some reason not coming up for me 07:39
Tenguhmmm. weird. I thought the UC would expose its APIs with tls when we're in a TLS-e env...07:40
ade_leeysandeep|ruck, https://zuul.opendev.org/t/openstack/build/5272aaca59a14b758dd5a2c190ccc218/log/logs/undercloud/home/zuul/ansible.log#101907:43
ysandeep|ruckhah, check jobs which are currently consuming older x/tripleo-ipa bits are passing: https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-standalone-on-multinode-ipa&skip=0 , so might be coming after the recent changes.07:46
ade_leeysandeep|ruck, what version of jinja are they using?07:49
ysandeep|ruckIn CI we have Jinja2 3.0.1 https://5e3861b11d69bac387ee-0e48560d50c149c15f98c62304a66a58.ssl.cf5.rackcdn.com/853478/6/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/5272aac/logs/undercloud/var/log/extra/pip.txt07:53
ysandeep|ruckIt comes from rpm actually not pip: python3-jinja2.noarch 3.0.1-2.el9s  @delorean-master-testing  07:54
ade_leeweird - I see the same08:00
*** ysandeep|ruck is now known as ysandeep|lunch08:34
opendevreviewMartin Magr proposed openstack/tripleo-heat-templates master: Make sure sensubility has proper permission  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85360408:44
opendevreviewCedric Jeanneret proposed openstack/tripleo-ansible stable/train: New role, replacing puppet-auditd  https://review.opendev.org/c/openstack/tripleo-ansible/+/85360709:13
opendevreviewBrendan Shephard proposed openstack/tripleo-common master: Add multi-rhel support for Nova services  https://review.opendev.org/c/openstack/tripleo-common/+/85321109:20
opendevreviewyatin proposed openstack/tripleo-heat-templates master: Switch default ML2 plugin in Undercloud to OVN  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85015809:23
opendevreviewArchana Singh proposed openstack/tripleo-upgrade stable/train: Ensure fencing is correctly handled during update  https://review.opendev.org/c/openstack/tripleo-upgrade/+/85096709:36
opendevreviewKe Niu proposed openstack/validations-common master: remove unicode prefix from code  https://review.opendev.org/c/openstack/validations-common/+/85346309:37
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Add ipv6 support in environments/deployed-network-environment.j2.yaml  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85324709:52
*** ysandeep|lunch is now known as ysandeep10:05
*** ysandeep is now known as ysandeep|ruck10:07
opendevreviewravi singh proposed openstack/tripleo-validations stable/wallaby: Run nova-svirt validation in nodes with nova_libvirt service  https://review.opendev.org/c/openstack/tripleo-validations/+/85362210:47
opendevreviewMaor Blaustein proposed openstack/tripleo-heat-templates master: Adds SELinux relabeling before puppet host configuration  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85362310:51
odyssey4memarios looks like https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/817552 just needs +w - it looks like it was tested in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/823966 ?11:00
mariosodyssey4me: looking11:00
mariosodyssey4me: so tripleo-quickstart-extras/+/817552 should be rebased and same for the test (to get latest tht/wallaby). the implied context here is actually adding the standalone-upgrade-ffu job? (note the test is currently titled for the undercloud-upgrade-ffu) 11:02
mariosodyssey4me: that would need update though https://review.opendev.org/c/openstack/tripleo-heat-templates/+/823966/21/zuul.d/layout.yaml 11:03
mariosodyssey4me: does not appear to have any templates or extra jobs wired up here and no ffu job 11:03
mariosah it has  - tripleo-upgrades-wallaby-pipeline11:03
mariosso you'd have to add the stnadlone-upgrade-ffu job to that pipeline or manually into the test zuul layout11:04
mariosodyssey4me: that ok for now? if we get good runs on that job then yeah lets go and merge 11:07
odyssey4memarios I think the intent was to wire that up downstream - or in a periodic... wherever it can happen, but take the time it needs11:22
opendevreviewJesse Pretorius proposed openstack/tripleo-quickstart-extras master: Adapt standalone-upgrade to run tripleo operator  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81755211:22
opendevreviewVeronika Fisarova proposed openstack/validations-common master: Add the default filter so the validations have default fallback  https://review.opendev.org/c/openstack/validations-common/+/85362611:25
mariosodyssey4me: i see.. but does it really take longer than 3 hours ... i mean unless it includes some kind of leap upgrade then we could try it upstream?11:28
mariosi mean we have undercloud-upgrade train to w ffu voting/gating now odyssey4me 11:28
marioscould make your life better to catch at least some things here in gate11:28
opendevreviewSofer Athlan-Guyot proposed openstack/tripleo-quickstart master: Add support for 3 controller 1 compute for deployed servers.  https://review.opendev.org/c/openstack/tripleo-quickstart/+/84091411:30
opendevreviewSofer Athlan-Guyot proposed openstack/tripleo-quickstart master: Add a update job for 3ctl 1comp and with a full fledge overcloud.  https://review.opendev.org/c/openstack/tripleo-quickstart/+/84091511:30
mariosk thanks for rebase i'll add that to my reviews list to check (though rebase didn't wipe votes). would be good to see that tested... if you're talking periodic instead of check/gate then lets try running it in rdo?11:30
mariosodyssey4me: ^ 11:30
odyssey4memarios yeah, it'd be really good to see a standalone upgrade in check/gate - is there a way to make that happen for any existing scenario, or would it require a new scenario? cc damani 11:31
mariosodyssey4me: i think the undercloud-upgrade-ffu is a good job to look at damani it should be very similar to what you want to add11:33
mariosodyssey4me: i think we'd have to add a new job. for wallaby 8 specifically (not 9 cos cant have train on 9 for ffu there)11:34
mariosodyssey4me: so added in gate next to undercloud-upgrade-ffu for wallaby gate across tripleo 11:34
odyssey4medamani jbadiapa marios looks like some work was started - https://review.opendev.org/c/openstack/tripleo-ci/+/829713 / https://review.opendev.org/c/openstack/tripleo-quickstart/+/82971111:34
mariosodyssey4me: damani: see here for the undercloud-ugprade-ffu https://opendev.org/openstack/tripleo-ci/src/commit/78a0506b67463f18e0650c2602387164bb92fe30/zuul.d/upgrades-jobs-templates.yaml#L339 added by jfrancoa (and wired up in that file also in the tripleo-upgrades-wallaby-pipeline 11:35
mariosso we'd add it next to that ^^ 11:35
*** dviroel|afk is now known as dviroel11:39
mariosack odyssey4me 14:34 < odyssey4me> damani jbadiapa marios looks like some work was started - https://review.opendev.org/c/openstack/tripleo-ci/+/829713 / 11:39
mariosthat looks about right so lets get those rebased/fresh runs and see what happens? 11:39
mariosodyssey4me: can bring that to the next ci community call? 11:40
mariosif it helps otherwise lets just discuss in gerrit/here11:40
jbadiapamarios, yes we need to rebase the patche first11:41
odyssey4mejbadiapa damani please decide who will be taking that forward, then attend the ci meeting to get any questions answered or get any assistance you need11:42
odyssey4methe sooner we can get a wallaby->master job in place, the better11:43
odyssey4memany thanks marios for your help!11:43
mariosodyssey4me: ack but note if you're talking wallaby to zed then it will be *9* not 811:44
dviroelfpantano: marios: ysandeep|ruck: it is that you folks finished quincy promotion11:44
mariosi was talking about t to w so far like we have to for undercloud-ffu 11:44
mariosodyssey4me: ^ 11:44
mariosodyssey4me: and w to z on 9 is a fine idea too ... just clarifying11:45
ysandeep|ruckdviroel, yes we promoted new images for quincy11:45
odyssey4memarios ack, if it is easier to do train->wallaby for now, that's fine... but we do need to follow that on with a wallaby->zed asap11:45
ysandeep|ruckdviroel, https://review.rdoproject.org/r/c/testproject/+/4456211:45
ysandeep|ruckI have tested sc001 afterwards - its working now11:46
mariosdviroel: ack as ysandeep|ruck is telling you we weren't sure of the status but pieced it together from the evidence (abandoned testproject). would have been nice to drop us a note/email/fax about it :) and save some panic11:46
mariosdviroel: i posted and then abandoned the revert 11:46
mariosodyssey4me: yeah probably you'd want both in the end 11:47
* bshephar considers faxing code reviews to marios11:47
marios:D11:47
dviroelmarios: thats true, i was about to drop that info in RR hackmd, but i wasn't expecting you folks recheck quickstart change before I start the day11:47
mariosdviroel: gate was broken as the tqe patch merged ~2 hours after your abandon so we had to react to it11:48
dviroelmarios: sorry about that, i saw this patch failing on gate before going to bed, thats why I wasn't so worried. But it merged :P11:49
mariosdviroel: i've broken the gate many times so its not a personal thing... lucky ysandeep|ruck could get it fixed relatively quickly but it should have been an automatic note to the ruck|rover out of courtesy. as it was if ysandeep|ruck dind't jump on it this mornign we'd have broken gate all day (even if the fix only took 20 mins, you were the only one with that info this morning)11:51
mariosdviroel: we are ready to workflow that one to switch the promotion too sec fetching link11:52
mariosdviroel: that https://review.rdoproject.org/r/c/rdo-jobs/+/44518/1#message-b1e08af2b1094bed9cb949e5177eef41d496c61b 11:52
waleedmHi guys, I'm deploying tripleoyoga in cenots 8 stream and the overcloud deplyemnt failed with this error CREATE_FAILED  resources.ControllerServiceChain: Error in 21 output role_data: The values of the "for_each" argument to "repeat" must be lists or maps11:53
waleedmhow to debug it ?11:53
dviroelmarios: yes, this one is ready to w+11:53
Tengutkajinam: is this kind of issue related to some puppetlabs-apache update? https://8b5ec10dd250ff86da53-b37c466c2c0a96beb3466c3b3d4e1c87.ssl.cf2.rackcdn.com/852733/2/gate/tripleo-ci-centos-9-undercloud-upgrade/3a63ea6/logs/undercloud/home/zuul/undercloud_upgrade.log (search for "parameter 'port' variant 0 expects an Undef value, got Tuple")11:55
tkajinamTengu, very likely11:56
Tenguakahat: -^^  fyi11:57
Tengutkajinam: one more motivation to move on that task of mine.11:57
bshepharwaleedm: Might be something wrong in your roles_data.yaml file. 11:57
tkajinamwhat is the upgrade being tested. wallaby to master ?11:58
ysandeep|ruckbshephar, do we support Yoga with C8? may be better to use c9/wallaby or c9/master.11:59
ysandeep|ruckwaleedm, ^^11:59
ysandeep|ruckI don't think c8 is supported after W11:59
bshepharYeah, I wouldn't use c8 now. But that particular error wont be related to the OS. 11:59
bshepharYeah, I belive w is the last tripleo release supporting CentOS8 as well11:59
tkajinamTengu, I believe we fixed it in maser https://review.opendev.org/c/openstack/tripleo-heat-templates/+/84531712:01
waleedmmmm I only generated the file using the using this command, "openstack overcloud roles generate -o /home/stack/roles_data.yaml ComputeSriovIB Controller"12:01
tkajinamhttps://bugs.launchpad.net/tripleo/+bug/197813112:01
waleedmfor the controller node i added a new network "    OcProvisioning:12:02
waleedm      subnet: oc_provisioning_subnet12:02
waleedm" 12:02
waleedmbu I already defined this network in network_data and VIP files and also the control.yaml file 12:03
waleedmok I can gitve it a try with centos 9 master 12:05
akahatTengu, thanks!12:06
akahattkajinam, Today i've hit it (again)12:06
Tenguakahat: np. fyi, working on dropping that module from tripleo, replacing it by some ansible in tripleo-ansible. Will take some time though, of course ^^'12:06
tkajinamakahat, what is the source/dest version used in that job ?12:07
tkajinamakahat, train -> wallaby or wallaby -> master ?12:07
akahattkajinam, wallaby c9 -> master12:09
akahatTengu, Thanks for info.12:10
bshepharwaleedm: Hmm ok, the error might be coming from here then: https://github.com/openstack/tripleo-heat-templates/blob/master/network/ports/net_ip_list_map.j2.yaml#L92-L9812:12
tkajinamakahat, openstack-tripleo-heat-templates-14.3.1-0.20220818082542.7907b48.el9.noarch12:13
tkajinamakahat, this is what is installed in undercloud12:13
bshepharProbably best to raise a bug and attach your templates though. Bit hard to guess without being able to check what you're trying to configure.12:13
tkajinamakahat, this one is wallaby, not master. I guess tht package was not updated properly12:13
waleedmbshephar, should I define the ctplane network in my network and vip yaml files ?12:15
waleedmbshephar, this my roles_yaml file https://paste.opendev.org/show/bhnMNq1hHYeS7C3Yv0H7/12:16
akahattkajinam, ok.. I'll rerun it.. let's see.12:16
waleedmbshephar, and this's my network and vip files https://paste.opendev.org/show/bDkA8b157PDYJ2hw2rts/12:18
*** deydra_ is now known as deydra12:18
waleedmbshephar, and this's my overcloud_baremetal_deploy.yaml file https://paste.opendev.org/show/bEJ2PURbcKjvVegwsQVV/12:19
waleedmif you just can take a quick view on it12:20
bshepharctlplane doesn't need to be in roles_data or network_data. You have vif: true in your baremetal_deploy file though. So you probably want to add it to the vips file12:24
bshepharNote the example file here: https://github.com/openstack/tripleo-heat-templates/blob/master/network-data-samples/vip-data-fixed-ip.yaml#L30-L3212:24
tkajinamwaleedm, which version of tht are you using ?12:25
bshepharAh vif is just for ironic actually12:25
waleedmit should the one installed from yoga repo branch 12:27
waleedmopenstack-tripleo-heat-templates-16.0.0-0.20220426101319.10707e9.el8.noarch12:27
tkajinamwaleedm, can you try the default ml2+ovn ? I suspect there might be an issue with mlnx agent templates12:30
waleedmok I'll try it 12:30
tkajinamI assume you are trying mlnx agent as you are using ComputeSriovIB role12:31
waleedmyes but also the error was in controller service chain 12:31
waleedmbut yeah it's better to try the default ml2-ovn firstly and see12:32
tkajinammight be an issue with OS::TripleO::Services::NeutronAgentsIBConfig12:33
tkajinamthough it's quite rough guess atm12:33
waleedmsure man, np I'll try it during the weekend or next week 12:34
waleedmalso, I'll generate a wallaby env with centos 8 to compare , because wallaby was working last time 12:34
fultonj19:09 <bshephar> fultonj: Do you have this one? https://review.opendev.org/c/openstack/tripleo-ansible/+/84565312:36
fultonjbshephar: following up on ^ from the other day...12:37
fultonji have that role, but I don't see it being run12:37
fultonjwhen i run the playbook on my external compute node i see http://ix.io/47WE12:38
fultonjclosest thing i see is tripleo_network_config running12:39
fultonjbshephar: if you have `sudo ovs-vsctl show` on your external and standalone node output handy, i'd be curious to see what it looks like.12:39
fultonjbogdando: regarding tripleo_nova_compute .... 12:40
fultonjMy inventory file has nova placement and neutron parameters yet I don't see them rendered into my nova.conf by the tripleo_nova_compute role12:40
fultonjhttps://paste.opendev.org/show/816007/12:40
fultonjwhen i add them to the nova.conf after ansible runs my external compute is discvered via 'nova-manage cell_v2 discover_hosts --verbose' however.12:42
bshepharHey fultonj. I currently don't have it deployed sorry, using that node for multi-rhel testing atm unfortunately.12:43
fultonjbshephar: ok, do you find it unsual that i don't see that role in the ansible output http://ix.io/47WE ?12:43
fultonjI suppose i'll read the playbook next to see where that role would get called to try to figure out why it's not getting called12:44
fultonjadvice welcome :)12:44
opendevreviewMerged openstack/puppet-tripleo master: Ceilometer: Replace hiera by lookup  https://review.opendev.org/c/openstack/puppet-tripleo/+/83928812:47
bshepharHmm yeah, should be there. Maybe I missed adding something12:47
fultonjbshephar: i take it you followed https://etherpad.opendev.org/p/tripleo-standalone-roles and then developed your ovn role and saw it work in that context12:50
bshepharfultonj: I have only been running it as either just the role itself. So a playbook that just calls that role. Or, with tripleo-heat-templates importing that role. Maybe theres something I'm missing for the wider standalone roles to leverage it.12:50
ysandeep|ruckdviroel, fpantano https://791d2cb36d0580e31ba5-4a7312796dc78ee19ab8c1340b1f0bd5.ssl.cf1.rackcdn.com/852713/2/gate/tripleo-ci-centos-9-scenario004-standalone/c08beaf/job-output.txt hit issue with ceph in wallaby too. 12:50
ysandeep|ruckdviroel, CONTAINER_NAMESPACE": "quay.rdoproject.org/tripleomastercentos9" something is wrong.. why wallaby job pulling from master12:51
fultonjbshephar: oh! 12:51
fultonjok, good this is why i'm asking :)12:51
bogdandofultonj: on my side, I'm preparing molecule coverage to catch things like that. But the config management scope, for PoC was something very limited, for crucial nova.conf things only, iirc? slagle has authored those parts of the roles...12:51
dviroelysandeep|ruck: make no sense12:52
tkajinamdviroel, fpantano, in case you want to check details ... https://zuul.opendev.org/t/openstack/build/0b34288d695945f39b4095745908a6e5 https://zuul.opendev.org/t/openstack/build/c08beaf980344601abda3daddb1055d4 https://zuul.opendev.org/t/openstack/build/b18fd0a8092a4ef499839aef7dff873a12:52
fultonjbogdando: was part of POC sucess criteria booting an instance?12:52
bogdandoI think, yes12:52
* fultonj doesn't want to be trying to get something working that never worked in the first place12:52
tkajinamdviroel, scenario 01, 04, 10 in stable/wallaby are affected now, somehow12:52
bogdandobut it can be booted w/o networks :)12:52
tkajinam(it's reasonable as these are jobs with ceph ,but I don't know how the change affected wallaby12:53
* fpantano checking12:53
dviroeltkajinam: ack - lets investigate12:53
fultonjbogdando: ah ha!12:53
fultonjbogdando: https://paste.opendev.org/show/b3q6EVwcQC4QLGh5qlr1/ 12:53
fultonjmy node fails to boot because i booted it on a network nad i hit this ml2 issue12:53
ysandeep|ruckdviroel tkajinam , https://paste.openstack.org/show/bLq342orfQEcJVo7pPnh/ CONTAINER_NAMESPACE is wrongly set here.12:53
fultonjRefusing to bind port ca3325a5-ba26-4fdd-8961-f133495f613c due to no OVN chassis for host: centos.example.com12:54
fultonjERROR neutron.plugins.ml2.managers [req-61e26461-b1ae-4cc0-95e5-9d123f04256e req-232b031b-8bf7-4f8c-9273-a803c239f9e1 3602f74844c2437eb5568190e5204231 29901c1248944cb690a9a74b1dff323a - - default default] Failed to bind port ca3325a5-ba26-4fdd-8961-f133495f613c on host centos.example.com for vnic_type normal using segments [{'id': '59537308-c147-4f22-8b26-12fc9630d15f',12:54
fultonj'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 981, 'network_id': '075bb485-f321-4208-aa42-c7c1a046b716'}]12:54
fultonjbogdando: ^ 12:54
fultonjif that's expected because that part of deployment is not implmented i'd like to know12:54
bogdandojust saying that 'booting an instance' may cover nova and glance only, or all openstack :)12:54
fpantanoysandeep|ruck: dviroel so CONTAINER_NAMESPACE": "quay.rdoproject.org/tripleomastercentos9 <= in wallaby ?12:55
fultonjbogdando: yes, i agree very much12:55
bogdandoit is normal that not everything is configured with ansible roles yet...12:55
fultonjbogdando: it would be good to know what is expected12:55
fpantanomaybe https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/tasks/ceph-install.yml#L13912:55
fpantanothis stuff ^^^ > dviroel ?12:55
bogdandothat's early poc, I haven't looked into new config manager, slagle and DF folks know more about it12:56
fpantanobecause that should be the point where tripleo-ansible-operator calls the ceph role w/ many parameters12:56
bogdandobtw, I've looked into multi rbd configs, and that was quite a riddle to translate it from tht/yaql to jinja12:57
ysandeep|ruckdviroel, fpantano we have ceph promotion pipelines for wallaby also, right?12:57
fpantanoysandeep|ruck: yes, but it's not about promotion there, it's because the namespace looks hardcoded12:57
bogdandobut for now we only have those as facts, config bits TBD12:57
fpantanoin quickstart12:57
dviroelfpantano: yes, might be12:58
fultonjbogdando: do you have a review i can look at?12:58
fpantanolet me try a fix --on-the-fly12:58
fultonjwith the riddle12:58
ysandeep|ruckdviroel, don't we have docker_ceph_namespace in relase file?12:58
* ysandeep|ruck checking12:58
fultonji assume you're talking about multirbd for cinder config12:58
dviroelysandeep|ruck: no, we don't have12:59
fpantanodviroel: or should we set {{ release }} on https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/tasks/ceph-install.yml#L139 ?12:59
fpantanotripleo{{release}}centos9 ?12:59
dviroelfpantano: yes, this should fix13:00
fultonjslagle: I followed [1] but was unable to boot a VM on the network [2]. Were you able to boot a VM on the network? 13:00
fultonj[1] https://etherpad.opendev.org/p/tripleo-standalone-roles13:00
fultonj[2] https://paste.opendev.org/show/b3q6EVwcQC4QLGh5qlr1/13:00
dviroelfpantano: and we only promote images for centos9 now, but should be the same for centos8, we can let centos9 hardcoded, please add a comment about that13:01
slaglefultonj: we need the ovn bits added at a minimum13:01
ysandeep|ruckdviroel, may be we use docker_registry_namespace instead https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-9/master.yml#L26 13:01
fpantanoyes13:01
fpantanodviroel: ack13:01
ysandeep|ruckfpantano, ^^13:01
fultonjslagle: so the OVN issue I'm hitting is expected becuase that part is not yet working via standalone ansible. True?13:02
fpantanoysandeep|ruck: not sure, I feel like we can break then the downstream periodics13:03
slaglefultonj: highly likely13:03
fultonji.e. https://review.opendev.org/c/openstack/tripleo-ansible/+/840509 needs to include a call to https://review.opendev.org/c/openstack/tripleo-ansible/+/84565313:03
fultonji don't see that role getting called when i run the playbook as per the etherpad13:03
fultonjhttp://ix.io/47WE13:03
dviroelysandeep|ruck: there is this docker_ceph_namespace var that is being used for long time, and it is the way to override only ceph namespace13:04
dviroelysandeep|ruck: that's why it is there13:04
fultonjthanks all for the tips. i feel better now that it's not supposed to work yet. it's not just me :)13:04
ysandeep|ruckdviroel, yeah i think we should have docker_ceph_namespace in release file13:04
tkajinamit seems docker_ceph_namespace is not defined in any place ?13:04
tkajinamI found one in tripleo-ci but it is not used in these wallaby standalone jobs13:04
* tkajinam noticed the same was already discussed 13:05
opendevreviewFrancesco Pantano proposed openstack/tripleo-quickstart-extras master: Update CEPH_NAMESPACE and rely on the "release" variable  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/85365213:05
* fultonj looks at 85365213:06
opendevreviewFrancesco Pantano proposed openstack/tripleo-quickstart-extras master: Update CEPH_NAMESPACE and rely on the "release" variable  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/85365213:06
fpantanodviroel: not sure is the right way but let me know your feedback on this ^^13:06
fpantanoysandeep|ruck: ^^13:06
slaglefultonj: it does not work end to end. still identifying what all contributions we need from other groups to close the gaps13:07
bogdandofultonj: I would say, to test nova scope roles, try booting it w/o network13:07
bogdandocannot really help with neutron things much, sorry13:08
bogdando> that part of deployment is not implmented - highly likely :)13:08
bogdandofultonj: sure, let me link it13:09
bogdandohttps://review.opendev.org/c/openstack/tripleo-ansible/+/842152/32/tripleo_ansible/roles/tripleo_nova_libvirt/tasks/main.yml (not nova compute)13:10
bogdandoin my plans to have some decent molecule coverage for those facts, then check it against expected config lines13:11
ysandeep|ruckfpantano, thanks! 13:13
dviroelfpantano: do you have a stable/wallaby test patch?13:14
fpantanodviroel: mmmm wait, isn't quickstart branchless? ^13:15
fpantanoah you mean a testproject?13:16
fpantanowhich depends-on that patch?13:16
fpantanodviroel: ah we can test it in tripleo yes13:16
dviroelfpantano: just to check if actually works with stable/wallaby13:17
fpantanolet me put a dnm patch up13:17
dviroelyes13:17
ysandeep|ruckdviroel, are we not getting docker_ceph_namespace from here: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/extras-common/defaults/main.yml#L7013:17
opendevreviewVeronika Fisarova proposed openstack/validations-common master: Add the default filter so the validations have default fallback  https://review.opendev.org/c/openstack/validations-common/+/85362613:17
opendevreviewVeronika Fisarova proposed openstack/validations-common master: Add the default filter so the validations have default fallback  https://review.opendev.org/c/openstack/validations-common/+/85362613:20
opendevreviewFrancesco Pantano proposed openstack/tripleo-heat-templates stable/wallaby: DNM - PACIFIC - Test PACIFIC_QUICKSTART  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85365513:21
fpantanodviroel: ^^^ 13:21
dviroelfpantano: thanks13:27
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates master: Configure OVN DB connections by ansible  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85365813:39
dviroelysandeep|ruck: so yeah, we should. You can see the default there with {{release}}13:43
opendevreviewBrendan Shephard proposed openstack/tripleo-heat-templates master: Add handling for Nova multi-rhel  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85366113:44
dviroelysandeep|ruck: so, it is because of the ternary condition on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/853652/2/roles/standalone/tasks/ceph-install.yml#13913:49
dviroelysandeep|ruck: maybe this should be moved to always get from "docker_ceph_namespace"13:49
dviroelsince we have all the logic in extras-common already13:49
bogdandofultonj: if you perchance have any inputs for https://review.opendev.org/c/openstack/tripleo-ansible/+/847347 please share13:53
bogdandowrt facts-or-not, given the above multi-store config example13:53
fultonjline 6213:55
fultonji see13:55
fultonjlet me read that and think about it before i comment13:55
ysandeep|ruckdviroel, sounds good to me13:56
dviroelysandeep|ruck: lets wait for results on fpantano fix, and fix the gate, we can propose that in a follow up change13:57
ysandeep|ruckdviroel, I think we should report a bug for wallaby.. let me do that before i leave for the day13:59
fpantanodviroel: ysandeep|ruck yeah, hopefully that's enough13:59
slaglefultonj: i'll take a look at the ovn integration today13:59
dviroelysandeep|ruck: ack - thanks ysandeep|ruck 13:59
fultonjslagle: ok, cool. thanks13:59
fultonji'll be happy to test it13:59
opendevreviewCedric Jeanneret proposed openstack/tripleo-ansible master: Update skeleton content to match standalone doc  https://review.opendev.org/c/openstack/tripleo-ansible/+/85271114:04
opendevreviewVeronika Fisarova proposed openstack/validations-common master: Add the default filter so the validations have default fallback  https://review.opendev.org/c/openstack/validations-common/+/85362614:13
bshepharslagle: Let me know if it's something that I have missed. Happy to rectify if I have overlooked something14:15
slaglebshephar: i'll add it to the playbooks today and see what happens :)14:20
bshepharIt'll work, I have confidence. 14:24
opendevreviewMerged openstack/tripleo-heat-templates master: OVN: Use puppet-ovn to set ovn-cms-options  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/83109814:25
*** ysandeep|ruck is now known as ysandeep|afk14:38
opendevreviewCedric Jeanneret proposed openstack/tripleo-heat-templates stable/wallaby: Ensure mode is correct for the tmp directory  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85344314:41
opendevreviewCedric Jeanneret proposed openstack/tripleo-ansible master: Update skeleton content to match standalone doc  https://review.opendev.org/c/openstack/tripleo-ansible/+/85271114:46
*** ysandeep|afk is now known as ysandeep14:57
opendevreviewBogdan Dobrelya proposed openstack/tripleo-ansible master: Add docs for standalone roles and playbooks  https://review.opendev.org/c/openstack/tripleo-ansible/+/84734714:59
opendevreviewTakashi Kajinami proposed openstack/tripleo-common master: Remove ceilometer-common from swift images  https://review.opendev.org/c/openstack/tripleo-common/+/85327815:01
fultonjFYI: 'openstack server create --nic none' works for my external compute node deployed by standalone ansible roles15:04
*** ysandeep is now known as ysandeep|PTO15:05
slaglecall it "network isolated environments"15:05
slagleit's a feature15:05
fultonj:)15:05
fultonji hear it's good for security15:06
opendevreviewMerged openstack/tripleo-ci master: Add tripleo_ipa under files filter for ipa job.  https://review.opendev.org/c/openstack/tripleo-ci/+/85342815:09
*** dasm|off is now known as dasm15:09
*** amoralej is now known as amoralej|off15:27
*** marios is now known as marios|out15:30
fpantanomarios|out: dviroel FWIW on the livestream I see: https://paste.opendev.org/show/bSA9XXsmC0F8bO4N8MCg/15:35
dviroelfpantano: great15:36
fpantanodviroel: and this is master log -> https://05c43b688f67d53aa028-5f4fcc390c732de527a6558cf26ba7f3.ssl.cf1.rackcdn.com/853652/2/check/tripleo-ci-centos-9-scenario001-standalone/a0f3b00/logs/quickstart_install.log15:36
fpantanowhile wallaby still running15:36
fultonjthanks fpantano 15:36
* dviroel lunch - brb15:37
opendevreviewMerged openstack/tripleo-quickstart-extras master: Adds dlrn_hash resolution tripleo-get-hash for periodic mixed rhel  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/85316715:43
opendevreviewMaor Blaustein proposed openstack/tripleo-heat-templates master: Adds SELinux relabeling before puppet host configuration  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85362316:02
opendevreviewJames Slagle proposed openstack/tripleo-ansible master: Clean up working dir from tox role-addition  https://review.opendev.org/c/openstack/tripleo-ansible/+/85368316:04
*** jpena is now known as jpena|off16:42
opendevreviewSofer Athlan-Guyot proposed openstack/tripleo-quickstart master: Add a update job for 3ctl 1comp and with a full fledge overcloud.  https://review.opendev.org/c/openstack/tripleo-quickstart/+/84091517:49
opendevreviewMerged openstack/python-tripleoclient stable/wallaby: Add cell export to overcloud deploy  https://review.opendev.org/c/openstack/python-tripleoclient/+/85264518:06
opendevreviewMerged openstack/tripleo-ci master: Bump standalone ceph nightly jobs to Quincy  https://review.opendev.org/c/openstack/tripleo-ci/+/85271518:06
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: tripleo_auditd: Clear unused default of tripleo_auditd_rules  https://review.opendev.org/c/openstack/tripleo-ansible/+/85329718:06
opendevreviewMerged openstack/tripleo-ansible master: Skip CephIngress daemon deploy  https://review.opendev.org/c/openstack/tripleo-ansible/+/85253218:06
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Fix ignored NovaLibvirtMaxQueues  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/84781418:06
opendevreviewClark Boylan proposed openstack/diskimage-builder master: DNM test if GRUB_DISABLE_UUID fixes rocky 9 boots  https://review.opendev.org/c/openstack/diskimage-builder/+/85369118:18
opendevreviewMerged openstack/puppet-tripleo master: Aodh: Sync default value in parameter description  https://review.opendev.org/c/openstack/puppet-tripleo/+/83895519:11
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Set OVSDB Connection.probe_interval  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85188919:11
*** rcastillo|rover_ is now known as rcastillo|rover19:16
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: Leave connection setup to THT  https://review.opendev.org/c/openstack/tripleo-ansible/+/85188819:52
opendevreviewSofer Athlan-Guyot proposed openstack/tripleo-quickstart master: Add a update job for 3ctl 1comp and with a full fledge overcloud.  https://review.opendev.org/c/openstack/tripleo-quickstart/+/84091520:00
opendevreviewJohn Fulton proposed openstack/tripleo-heat-templates master: Introduce ExternalCeph boolean  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85369920:09
opendevreviewJames Slagle proposed openstack/tripleo-ansible master: Set setype: container_file_t for kolla config dir  https://review.opendev.org/c/openstack/tripleo-ansible/+/85370020:11
opendevreviewJames Slagle proposed openstack/tripleo-ansible master: Add docs for standalone roles and playbooks  https://review.opendev.org/c/openstack/tripleo-ansible/+/84734720:13
opendevreviewJames Slagle proposed openstack/tripleo-ansible master: tripleo_logrotate_crond: add for standalone roles  https://review.opendev.org/c/openstack/tripleo-ansible/+/84199720:24
opendevreviewJames Slagle proposed openstack/tripleo-ansible master: Rename iscsid_install.yml to just install.yml  https://review.opendev.org/c/openstack/tripleo-ansible/+/85370120:26
*** dviroel is now known as dviroel|afk20:32
opendevreviewMerged openstack/puppet-tripleo master: Remove manifest for auditd service  https://review.opendev.org/c/openstack/puppet-tripleo/+/85259821:06
opendevreviewMerged openstack/puppet-tripleo master: Barbican: Replace hiera by lookup  https://review.opendev.org/c/openstack/puppet-tripleo/+/83922221:06
opendevreviewMerged openstack/tripleo-ansible master: Add tripleo_nova_migration_target role  https://review.opendev.org/c/openstack/tripleo-ansible/+/84326921:06
opendevreviewMerged openstack/tripleo-heat-templates master: Use tripleo_migration_target standalone ansible role  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85173921:06
opendevreviewMerged openstack/tripleo-quickstart master: Set dlrn_hash_tag to current-tripleo if mixed os job  https://review.opendev.org/c/openstack/tripleo-quickstart/+/85273321:06
*** dasm is now known as dasm|off22:08
*** rlandy_ is now known as rlandy|out23:13
*** rcastillo|rover is now known as rcastillo23:13

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!