Friday, 2018-11-23

rdogerritrdo-trunk proposed rdoinfo master: Bump rdoinfo stein-uc tags to latest upper-constraints  https://review.rdoproject.org/r/1736400:02
rdogerritrdo-trunk proposed rdoinfo master: Promote CBS tags update for queens-testing  https://review.rdoproject.org/r/1741900:03
rdogerritrdo-trunk created rdoinfo master: Promote CBS tags update for rocky-testing  https://review.rdoproject.org/r/1745800:04
*** hitch has quit IRC00:05
*** hitch has joined #rdo00:06
*** cyberworm54 has quit IRC00:43
*** diw has joined #rdo01:43
*** diw has quit IRC02:16
*** owalsh has quit IRC02:25
*** d0ugal has quit IRC02:33
*** owalsh has joined #rdo02:39
rdogerritSteve Baker created rdoinfo master: Rebuild tar-split in Stein deps  https://review.rdoproject.org/r/1745902:41
*** tonyb has quit IRC02:49
*** tonyb has joined #rdo02:52
*** udesale has joined #rdo03:13
*** Son_Goku has quit IRC03:15
*** ykarel|away has joined #rdo03:40
*** ykarel|away is now known as ykarel03:44
rdogerritMerged rdoinfo master: Promote CBS tags update for rocky-testing  https://review.rdoproject.org/r/1745804:10
*** diw has joined #rdo04:13
*** ramishra has joined #rdo04:15
*** vnogin has joined #rdo04:17
*** vnogin has quit IRC04:21
rdogerritYatin Karel proposed rdoinfo master: Promote CBS tags update for queens-testing  https://review.rdoproject.org/r/1741904:26
*** janki has joined #rdo04:29
*** dtrainor__ has joined #rdo04:42
*** dtrainor_ has quit IRC04:45
*** diw has quit IRC04:47
*** radeks__ has joined #rdo04:47
*** radeks_ has quit IRC04:50
*** dtrainor__ has quit IRC04:57
*** radeks__ has quit IRC05:00
*** ykarel has quit IRC05:07
*** Son_Goku has joined #rdo05:08
*** ykarel has joined #rdo05:24
rdogerritrdo-trunk created openstack/kuryr-kubernetes-distgit rocky-rdo: openstack-%service-0.5.2-1  https://review.rdoproject.org/r/1746005:37
rdogerritrdo-trunk created puppet/puppet-cinder-distgit queens-rdo: puppet-cinder-12.4.1-1  https://review.rdoproject.org/r/1746105:37
rdogerritYatin Karel proposed openstack/kuryr-kubernetes-distgit rocky-rdo: openstack-kuryr-kubernetes-0.5.2-1  https://review.rdoproject.org/r/1746005:39
rdogerritrdo-trunk created puppet/puppet-cinder-distgit rocky-rdo: puppet-cinder-13.3.2-1  https://review.rdoproject.org/r/1746205:40
*** pcaruana has quit IRC05:44
openstackgerrityatin proposed openstack/packstack stable/queens: Set metadata_host instead of metadata_ip  https://review.openstack.org/61966505:46
*** janki has quit IRC05:55
*** janki has joined #rdo05:56
*** dtrainor__ has joined #rdo05:58
rdogerritMerged puppet/puppet-cinder-distgit rocky-rdo: puppet-cinder-13.3.2-1  https://review.rdoproject.org/r/1746205:59
rdogerritMerged puppet/puppet-cinder-distgit queens-rdo: puppet-cinder-12.4.1-1  https://review.rdoproject.org/r/1746105:59
rdogerritMerged openstack/kuryr-kubernetes-distgit rocky-rdo: openstack-kuryr-kubernetes-0.5.2-1  https://review.rdoproject.org/r/1746006:01
*** Son_Goku has quit IRC06:03
*** dtrainor__ has quit IRC06:11
*** ccamposr has joined #rdo06:15
*** chandankumar has joined #rdo06:23
*** chandankumar is now known as chkumar|rover06:25
*** chkumar|rover is now known as chkumar|ruck06:25
*** jfrancoa has joined #rdo06:35
*** ksambor has joined #rdo06:40
*** quiquell|off is now known as quiquell06:41
*** diw has joined #rdo06:44
*** BOKALDO has joined #rdo06:50
rdogerritMerged rdoinfo master: Promote CBS tags update for queens-testing  https://review.rdoproject.org/r/1741907:03
*** mosulica has joined #rdo07:03
rdogerritchkumar246 proposed rdo-jobs master: [DNM] testing msster/rocky failed jobs  https://review.rdoproject.org/r/1734807:08
*** pcaruana has joined #rdo07:08
rdogerritMerged openstack/tempest-distgit rpm-master: Create python2/python3 package depending on distro  https://review.rdoproject.org/r/1742807:10
rdogerritYatin Karel created rdoinfo master: Enable congress-tempest-plugin, tempestconf for Fedora  https://review.rdoproject.org/r/1746307:13
rdogerritchkumar246 created rdo-infra/ci-config master: skip fs01 for rocky in order to have promotion  https://review.rdoproject.org/r/1746407:16
chkumar|ruckykarel, ^^07:16
*** diw has quit IRC07:17
*** kopecmartin|off is now known as kopecmartin07:17
ykarelchkumar|ruck, ack07:18
ykarelokk fs035 already in skip07:18
chkumar|rucknumber80, Hye07:20
chkumar|ruckHey07:20
chkumar|rucknumber80, https://review.rdoproject.org/r/17464 bless this patch :-)07:21
rdogerritMerged rdo-infra/ci-config master: skip fs01 for rocky in order to have promotion  https://review.rdoproject.org/r/1746407:32
chkumar|ruckykarel, do you know how we create patches branches in https://review.rdoproject.org/r/#/admin/projects/openstack/tempest,branches07:34
chkumar|ruck?07:34
ykarelchkumar|ruck, nope, but why u need that? iirc we are no longer keeping patches in RDO07:36
ykareleverything from upstream07:36
chkumar|ruckykarel, we have backported few patches in downstream for 1307:36
chkumar|ruckfor tempest07:36
chkumar|ruckykarel, I want to keep it sync in both side07:36
ykarelchkumar|ruck, ack07:38
*** dtantsur|afk is now known as dtantsur07:38
quiquellykarel: I think the podman upgrade also needs something from the kernel07:43
quiquellykarel: This is going to be a never ending thing07:43
ykarelquiquell, what's the issue07:44
quiquellykarel: last comment here https://bugs.launchpad.net/tripleo/+bug/180464907:44
openstackLaunchpad bug 1804649 in tripleo "f28 standalon fails at timeout" [High,Triaged] - Assigned to Quique Llorente (quiquell)07:44
* ykarel looks07:44
quiquellykarel: tldr "kernel: overlayfs: unrecognized mount option "nodev" or missing value"07:44
quiquellykarel: fels like kernel module is old too or kernel itself07:44
quiquellykarel: The error does not appear at centos-7 standalone07:45
ykarelFedora kernel is 4.something i think07:45
*** rcernin has quit IRC07:48
*** cylopez has joined #rdo07:48
quiquellykarel: kernel-4.18.18-200.fc28.x86_6407:48
quiquell centos-7 kernel-3.10.0-862.14.4.el7.x86_6407:48
*** slaweq has joined #rdo07:49
ykarelquiquell, u said The error does not appear at centos-7 standalone, is there docker or podman?07:49
quiquellykarel: podman07:49
ykarelokk07:49
ykarelquiquell, GraphOptions:07:52
ykarel  - overlay.mountopt=nodev07:52
ykarel  - overlay.override_kernel_check=true07:52
ykarelquiquell, is ^^ set intentionally ?07:52
ykarel - overlay.mountopt=nodev07:53
ykarelspecifically ^^07:53
*** quiquell is now known as quiquell|brb07:56
*** cylopez has quit IRC07:56
rdogerritchkumar246 proposed rdo-jobs master: [DNM] testing msster/rocky failed jobs  https://review.rdoproject.org/r/1734807:57
Tenguykarel: that said, the overlay.mountopt is set in the example here: https://www.mankier.com/1/podman-info07:57
*** paramite has joined #rdo08:00
ykarelTengu, hmm, no clue why it's not set in CentOS08:02
Tenguprobably something linked to the kernel.08:02
Tengulemme check the content of the libpod.conf08:02
*** rpittau has joined #rdo08:02
Tengu(on my centos undercloud)08:03
ykarelcan it be runc version?08:03
Tengujust have to boot it up..08:03
Tenguhmmm runc is provided by the container team, isn't it?08:03
*** jtomasek has joined #rdo08:04
*** amoralej|off is now known as amoralej08:05
amoralejgood morning08:06
Tenguykarel: so nothing in is libpod.conf, at least on centos.08:06
Tenguthat said, fedora build uses the same repository as centos, regarding delorean and containers, right?08:07
Tenguor does it come from some other place?08:07
ykarelTengu, ack, nope CentOS and Fedora repos are different08:11
Tenguoh? care to point me the one providing the podman thingy?08:11
ykarelpodman buid Fedora: https://copr.fedorainfracloud.org/coprs/g/openstack-sig/fedora-overrides/build/827040/08:12
Tenguhmmm.08:13
Tenguso nothing regarding overlay in the provided libpod either.08:14
*** cylopez has joined #rdo08:14
*** apevec has joined #rdo08:14
Tenguwondering if this wouldn't require some issue against libpod on github.08:15
Tenguquiquell|brb: -^ I can create it if you want ;)08:15
kopecmartinamoralej, hi08:17
*** quiquell|brb is now known as quiquell08:17
kopecmartinamoralej, do you know how to push tempest patches to queens?08:17
*** jpena|off is now known as jpena08:17
amoralejkopecmartin, you mean patches to tempest code?08:18
quiquellTengu: wait, want to check first where is the place we us nodev08:18
kopecmartinamoralej, yes, I followed a RDO blog, but I'm struggling with it, i've got this issue http://paste.openstack.org/show/735960/08:18
rdogerritTobias Urdin created openstack/neutron-dynamic-routing-distgit rpm-master: Fix systemd unit file Restart line  https://review.rdoproject.org/r/1746508:18
frippe75Sorry for repeating myself. Has anybody succesfully deployed Rocky using packstack on CentOS recently (using stable)?08:24
apeveckopecmartin, you want patch  in RPM ?08:25
*** ykarel is now known as ykarel|lunch08:25
apevecin rdoinfo we have pin on tags:08:26
apevec  queens:08:26
apevec    source-branch: 18.0.008:26
frippe75What step is causing the the db record for the host to be created when adding a new nova compute to the controlplane? looked in nova.cmd.manage but not really sure how it meant to be added in.08:27
kopecmartinapevec, yes, i want to backport a few patches from master to queens rpm , it's in progress in downstream  and we wanted to sync that with upstream08:27
kopecmartinapevec, is there any doc or manual how to do it?08:27
rdogerritJavier Peña proposed config master: Initial addition of ceph-ansible to projects  https://review.rdoproject.org/r/1745608:28
apeveckopecmartin, as it says in your paste patches/queens-rdo-patches does not exist i.e. we did not have patches in queens RPM08:28
apevecis it required for upstream oooci?08:28
*** frippe75 has quit IRC08:30
*** akrivoka has joined #rdo08:30
kopecmartinapevec, not necessarily, chkumar|ruck what do you think? is the backport required for oooci?08:30
apeveckopecmartin, in RDO we avoid any patches since RDO is defined as vanilla upstream packaging08:31
apeveckopecmartin, which patch is it?08:31
kopecmartinapevec, for example this lp https://bugs.launchpad.net/tempest/+bug/179086408:32
openstackLaunchpad bug 1790864 in tempest "tempest ignors network.public_network_id values and discovers other shared network" [Undecided,New]08:32
kopecmartinand also this patch https://review.openstack.org/#/c/591595/08:32
kopecmartinand this one https://review.openstack.org/#/c/558417/08:32
*** holser_ has joined #rdo08:33
apevecso both merged on master and Tempest is branchless - I'd bump tag on both queens and rocky then to the latest08:33
*** ramishra has quit IRC08:33
apevecnow we have:08:34
*** holser_ has joined #rdo08:34
apevectags:08:34
apevec  ocata:08:34
apevec    source-branch: 16.1.008:34
apevec  pike:08:34
apevec    source-branch: 17.2.008:34
apevec  queens:08:34
apevec    source-branch: 18.0.008:34
apevec  rocky:08:34
apevec    source-branch: 19.0.008:34
apevecand stein following master08:34
apevecit's up to maintainers:08:34
apevec- chkumar@redhat.com08:34
apevec- slinaber@redhat.com08:34
apevec- ltoscano@redhat.com08:34
apevecit's branchless fun!08:35
kopecmartinapevec, hm, so it means, if i want to get a patch which was merged only couple of days ago to queens package, i would need to use practically the same version of tempest rocky uses (the latest one)08:37
kopecmartinand it may not work with queens at all (i mean the other changes we would include), really fun :D08:38
*** ramishra has joined #rdo08:38
rdogerritMerged rdo-jobs master: Remove periodic-tripleo-ci-centos-7-standalone-rocky job  https://review.rdoproject.org/r/1743108:38
apeveckopecmartin, well if it does not work with queens at all, then it's not branchless :)08:40
sfbenderJavier Peña proposed DLRN master: Move the pre-processing steps in the downstream driver  https://softwarefactory-project.io/r/1420908:41
chkumar|ruckapevec, kopecmartin bumping a new version will bring new tests and will more regression to downstream and most blacklisting08:42
chkumar|ruckwhich I wanted to avoid that08:42
*** lvdombrkr has joined #rdo08:44
apevecregression?08:45
*** gfidente has joined #rdo08:46
apevecper per https://github.com/openstack/tempest/blob/master/doc/source/stable_branch_support_policy.rst Tempest master supports at Pike-Stein now08:46
apevecand Tempest is now containerized, so new deps should be isolated08:47
chkumar|ruckapevec, with each new tempest tag, new tests got introduced and these new tests might not work sometimes on downstream and new tests failing on deployment give extra work to fix that or otherwise blacklist them that's why we tagged tempest for a particular release08:49
chkumar|ruckapevec, containerization solves the problem of dependency but tempest container is supported from rocky cycle08:49
chkumar|ruckapevec, sorry regression I mean to say some one wants to make new tests passing they need to backport patches for their components08:51
apevecthen Tempest's stable_branch_support_policy.rst is not correct?08:52
apeveci.e. aren't all new tests passing on Pike-Stein before they're merged?08:53
rdogerritDaniel Alvarez proposed rdoinfo master: Bump OVS to 2.10.0-6 in Rocky and Stein  https://review.rdoproject.org/r/1745508:53
apeveceither way, -1 for adding patches to openstack-tempest RPM unless it's blocking CI08:53
chkumar|ruckapevec, on Tripleo CI side, it is not blocking anything08:54
apevecleaving +-2 to maintainers of course08:54
apevecok then it can be d/s patch08:54
apeveckopecmartin, ^08:54
*** d0ugal has joined #rdo08:54
chkumar|ruckapevec, as per stable branch policy, a particular tempest tag support previous two release08:55
apevecchkumar|ruck, that's not what stable_branch_support_policy.rst says08:55
apevec"Tempest will only provide support for branches in the "Maintained" phase"08:56
apevecthese are documented in releases.openstack.org08:56
chkumar|rucksorry re-reading it again08:56
apevecit includes >= Pike until est. March 201908:56
apevecand Queens is Maintained until est. Aug 201908:57
*** brault has joined #rdo08:57
*** jpich has joined #rdo08:59
apevecchkumar|ruck, yeah, I see -pike in recent Tempest check09:00
apevecBTW I also saw now I have old unmerged review in Tempest :) https://review.openstack.org/30083209:01
* apevec wonders if it is still relevant09:01
chkumar|ruckapevec, I think this patch is no longer needed09:02
chkumar|ruckapevec, for handling plugins, we have got some many improvements09:02
*** mosulica has quit IRC09:02
apevecthanks, I'll abandon it with a comment09:02
chkumar|ruckapevec, on requirements side we use pinned tempest version https://github.com/openstack/requirements/blob/stable/queens/global-requirements.txt#L31609:04
*** florianf is now known as florianf|pto09:04
chkumar|ruckas it breaks tempest plugin interface also09:05
apevecright, so tempest plugins are also branchless and should move all together w/ Tempest09:07
apevecanyway, I shut up now as I'm not maintainer :)09:08
chkumar|ruckapevec, from rocky release, we have tagged almost all the tempest plugins09:08
*** ykarel|lunch is now known as ykarel09:10
*** diw has joined #rdo09:14
*** tosky has joined #rdo09:15
*** hseipp has joined #rdo09:16
amoralejapevec, chkumar|ruck kopecmartin but, will latest release of tempest work with libs, tempest plugins and other services versions in queens?09:17
*** akrivoka has quit IRC09:17
amoralejwe had problems bumping tempest in the past09:17
*** akrivoka has joined #rdo09:17
amoralejwhat version of tempest is used in stable/queens branches upstream?09:18
kopecmartinamoralej, yeah, that's the problem, so as apevec said, we're not gonna bump tempest in queens unless there is a CI requirement for it09:18
amoralej+109:19
amoralejif you want to push patches to tempest it's possible09:19
amoraleji can create the patches branch if needed09:19
number80tobias-urdin: there's a legit unit test errors (they seem to require network?) unrelated to the change => https://review.rdoproject.org/r/#/c/1746509:20
number80This might become a promotion blocker09:20
tobias-urdinhm09:21
number80This is weird because last commit was successully built09:23
apevecamoralej, please don't create patches branch unless there's critical CI fix09:23
amoralejyeah, i meant, in case we need to unblock09:23
amoraleji think patches is better that rebase in queens at this point09:23
apevecnumber80, the same commit?09:24
apevecDLRN-rpmbuild builds with latest master code09:24
number80apevec: same commit09:24
number80only package changed and it should not affect build (a typo fix in systemd unit)09:25
chkumar|ruckamoralej, do we create patches branches manually?09:26
apevecsame buildroot?09:26
number80apevec: that's what I'm investigating09:26
apevecchkumar|ruck, yes, again we try to avoid them09:27
*** mrunge_ is now known as mrunge09:27
chkumar|ruckack09:27
openstackgerritMerged openstack/packstack stable/rocky: Set metadata_host instead of metadata_ip  https://review.openstack.org/61960509:31
amoralejchkumar|ruck, normally yes09:31
amoralejbut try to avoid as apevec said09:32
*** frippe75 has joined #rdo09:33
rdogerritAlfredo Moralejo proposed rdo-jobs master: Add new oooq jobs running FS030 (ovn) to master, rocky and queens  https://review.rdoproject.org/r/1745309:41
amoralejdalvarez, ykarel ^ that should be the good one09:42
ykarelamoralej, ack09:42
dalvarezoh yeah!09:42
frippe75Since puppet by itself is not top-bottom execution. I noticed alot of begin/end, anchor syntax in the manifests. Having issues getting packstack to deploy rocky release and was wondering if this could mess up the "onboarding" of a new nova compute (i.e nova-manage commands is out of order) ??09:42
frippe75packstack question that is....09:42
amoralejapevec, btw, i'm adding ovn jobs to rdoinfo experimental pipeline to gate ovs09:42
*** derekh has joined #rdo09:44
*** diw has quit IRC09:47
ykarelfrippe75, what issue u are facing09:50
frippe75ykarel, sorry, have been bashing you for two days. realize not the best method but i cannot get the compute hosts to be discovered. packstack/centos 7.5 rocky release.09:52
*** sshnaidm is now known as sshnaidm|off09:52
*** holser_ has quit IRC09:52
*** holser_ has joined #rdo09:52
ykarelfrippe75, u doing multinode or singlenode deployment09:53
frippe75ykarel, nova-manage cell_v2 discover_hosts  says "Found 0 unmapped"... this is allinone09:53
frippe75ykarel, list_hosts lists 0 hosts09:53
frippe75ykarel, openstack compute service list --service nova-compute   lists my compute a up/enabled09:54
ykareli remeber there was some issue related to host entry in /etc/hosts09:54
ykarelamoralej, any idea ^^09:55
frippe75using FQDN09:55
amoralejmmm09:55
frippe75resolves both ways09:55
amoralejit shouldn't matter09:55
*** jfrancoa has quit IRC09:55
amoralejmain problem are renames09:55
amoralejbut i guess that's not the case09:55
amoralejfrippe75, is that allinone?09:56
frippe75amoralej, yes09:56
amoralejand service list --service nova-compute shows just one node, right?09:56
frippe75redeployed with selinux  on/off/permissive all the way from bare metal install09:56
amoralejyou are using centos 7.5?09:56
frippe75amoralej, yes one node09:56
frippe757.5.1804 updated09:56
amoralejlet me launch a deployment09:57
amoralejfrippe75, we test it regularly, should work09:57
amoralejhow did you enabled repos?09:57
frippe75wow, would be amazing! noticed a few more users reporting the same on https://ask.openstack.org09:57
frippe75i copy pasted the instructions for packstack on rdo09:58
amoralejwhat link?09:58
amoralejy think instructions are fine, just to double check09:58
frippe75https://www.rdoproject.org/install/packstack/09:58
amoralejso09:59
amoralejyum install -y centos-release-openstack-rocky09:59
amoralejright?09:59
frippe75yes10:00
jpenachkumar|ruck, panda|rover: can you check my last comment on https://bugs.launchpad.net/tripleo/+bug/1803964 ?10:01
openstackLaunchpad bug 1803964 in tripleo "post failure on multiple periodic jobs with SSH Error: data could not be sent to remote host" [Critical,Triaged]10:01
jpenaif we could get the cleaning_stacks.log file from the te-broker, maybe we could compare some timestamps10:01
*** jfrancoa has joined #rdo10:08
*** ramishra has quit IRC10:12
*** moguimar has joined #rdo10:12
panda|roverjpena: no way to get that log, it's overwritten every day.10:12
*** ramishra has joined #rdo10:12
frippe75amoralej, do you handle nsswitch.conf in anyway?10:12
frippe75i.e revert back to old behaviour.....10:13
panda|roverjpena: it's created with a very ppo command in crontab10:13
panda|roverpoor*10:14
amoralejfrippe75, we don't touch nsswitch.conf afaik10:14
panda|roverjpena: I'll try to propose a patch to do something smarter, but those logs are gone10:14
jpenapanda|rover: ok. I'm trying to figure out with kforde when was that VM deleted. At least it should give us some information10:15
jpenapanda|rover: actually, logs are overwritten on every execution -> https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/te-broker/tasks/main.yml#L15310:17
panda|roverjpena: which happens every day10:18
panda|roveroh10:18
panda|roverno was reading the cron wrong10:18
jpenaonce an hour10:18
panda|roveryes we definitely need better logging10:19
*** chkumar|ruck has quit IRC10:22
amoralejfrippe75, running packstack --allinone right now10:26
amoraleji'll let you know10:26
frippe75amoralej, much appreciated!10:26
*** moguimar has quit IRC10:27
*** moguimar has joined #rdo10:29
sshnaidm|offjpena, do you know where can I find zuul config for rdo ci?10:32
*** dpawlik_ has quit IRC10:32
jpenasshnaidm|off: I guess you are looking for https://github.com/rdo-infra/review.rdoproject.org-config/tree/master/zuul.d10:33
*** Caterpillar has quit IRC10:34
sshnaidm|offjpena, I need zuul config itself10:34
*** chandan_kumar has joined #rdo10:35
*** chandan_kumar is now known as chkumar|ruck10:35
jpenasshnaidm|off: then https://softwarefactory-project.io/r/gitweb?p=config.git;a=tree;f=zuul;h=ab54183aabbb51ec525dd36f512fc06036e5b46e;hb=refs/heads/master10:35
sshnaidm|offjpena, yeah, exactly, thanks10:35
*** Caterpillar has joined #rdo10:37
*** dpawlik has joined #rdo10:37
ykareljpena, are u looking for HOST KEY changed job10:47
ykareli saw that today10:47
jpenaykarel: yep10:47
ykareljpena, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/2861646/job-output.txt.gz#_2018-11-23_06_54_50_56690810:47
jpenawhich job? I can see if the logs match what I saw on Nov 1910:47
ykarel^^ from today10:47
jpenaykarel: interesting, I think we have similar symptoms. Look at this: https://softwarefactory-project.io/paste/show/1310/10:53
jpenathat is the nodepool log related to the IP. The first node is the one in the log (311507). Then, at 05:08:41, another nodepool node got the same IP10:54
jpenathat means either the FIP was removed, or the node was deleted before that time10:54
ykareljpena, ack, so need to see rdo-cloud logs what happend when10:55
jpenaand nodepool only tried to delete the node at 06:5510:55
jpenathe nova id for the node was 2fb9e2fe-1a98-4d15-b4c4-75cf37935abc10:55
*** udesale has quit IRC11:07
*** holser_ is now known as holser|lunch11:08
*** cylopez has quit IRC11:11
rdogerritJavier Peña proposed rdo-infra/puppet-dlrn master: Add support for build_driver and some driver parameters  https://review.rdoproject.org/r/1431111:11
*** jchhatbar has joined #rdo11:13
*** Son_Goku has joined #rdo11:13
*** janki has quit IRC11:15
chkumar|ruckjpena, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/40d580a/job-output.txt.gz#_2018-11-23_08_21_30_30818811:16
chkumar|ruckjpena, it is the recent ssh timedout leading to post_failure11:16
jpenachkumar|ruck: yes, we're tracking another similar instance to find out what happened11:16
*** akrivoka has quit IRC11:17
*** akrivoka has joined #rdo11:18
ykarelinteresting, same ip 38.145.34.15611:18
*** Son_Goku has quit IRC11:25
*** jchhatba_ has joined #rdo11:29
*** rpittau_ has joined #rdo11:31
*** jchhatbar has quit IRC11:32
*** rpittau has quit IRC11:35
*** rpittau_ is now known as rpittau11:43
*** diw has joined #rdo11:44
frippe75amoralej, reinstall, re-packstated, same result. This time a removed the myhostname option from hosts in nsswitch.conf since it appears to resolve to an ipv6 address... Still not working. On your end?11:48
amoralejfrippe75, my installating finished11:48
amoralejlet me check11:48
frippe75I have cell0 and default cell but no mappings11:50
amoralejfrippe75, how do you see what nodes are in what cell?11:52
amoralejwhat's the actual problem you find?11:52
frippe75nova-manage --debug  cell_v2 list_hosts11:53
frippe75during spawn of instance... "No valid host"11:53
amoralejmy list_hosts looks fine11:53
frippe75ahhh.... for real?11:53
amoralejit has the node in the default cell11:53
frippe75not on my side. this is really strange... centos 7.5.1804?11:54
frippe75FQDN?11:54
amoralejfrippe75, https://pastebin.com/t1ySjbvZ11:54
amoralejwhat do you see?11:54
frippe75https://pastebin.com/9izPamhQ11:55
amoralejfrippe75, and don't you see any error in nova logs?11:57
amoralejabout what's preveng the node from registering correctly?11:58
frippe75check scheduler,placement,compute... Not any error message I can relate.11:58
amoralejcan you paste  nova-manage cell_v2 list_cells too?11:58
frippe75well besides ... "No compute node record for host openstack.frippe.com: ComputeHostNotFound_Remote:"   this is the issue I'm trying to solve.11:59
frippe75amoralej, https://pastebin.com/raw/4GZsREVn12:00
amoralejfrippe75, in which log do you find that?12:00
amoralejcompute or server?12:00
frippe75compute12:01
amoralejit's expected that first time, it gives that error iirc12:02
amoralejbut then, it creates the compute node object12:02
amoralejyeah, i get the same error in my log12:03
amoralejthat's not an actual error12:03
amoralejit's just until is created12:03
amoralejhow many times do you find that frippe75 ?12:03
amoralejtwo?12:03
frippe75looked in ../nova/cmd/manage.py trying to figure out how it's suposed to get created. looks like via host_mapping_obj.discover_hosts12:04
*** ramishra has quit IRC12:04
frippe75amoralej, 30 times atleast12:04
amoralejcould you enable debug in nova.conf12:04
amoralejups12:04
amoralejmmm12:04
amoralejweird12:04
frippe75to say the least :-)12:05
ykarelfrippe75, "nova-status upgrade check" shows any warning?12:05
frippe75looks like packstack default options is not to use the "scheduled discovery"12:05
amoralejdid you enable debug in nova.conf and restart nova services?12:05
frippe75yes12:05
frippe75missing resource provider12:05
amoralejand didn't find something in nova-api or nova-scheduler logs?12:06
amoralejsome ERROR i mean12:06
frippe75ykarel, https://pastebin.com/raw/Fg5FuWZE12:06
ykarelso both placement and cells have issues12:07
frippe75amoralej, yes actually now in api ... reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(110, 'Connection timed out'))"12:07
amoralejmm12:08
amoralejhow many vcpus and memory do you have?12:08
frippe754  core / 32 GB12:08
amoralejyou have many of those errors?12:08
amoralejor just one?12:08
frippe75112:08
amoralejit's so weird...12:09
frippe75no errors in placement-api12:09
amoralejso12:09
frippe75no errors in scheduler12:09
amoralejgrep ERROR /var/lib/nova/*log12:09
amoralejreturns just that one?12:10
frippe75and the host mappings in compute...12:10
frippe75amoralej, looking in objects/host_mapping.py   it gets "created"   via    compute_nodes = objects.ComputeNodeList.get_all_by_not_mapped12:14
frippe75But that kind of depends on them being in the db in the first place right?? What actually populates the nodes in the nova db to begin with?12:14
*** EmilienM is now known as EvilienM12:14
*** diw has quit IRC12:17
*** Son_Goku has joined #rdo12:17
frippe75amoralej, and that stems from  a call to db_computes = db.compute_node_get_all_mapped_less_than12:17
frippe75trying to figure out what's needed to get populated into the db...12:18
amoralejapevec, number80 ykarel bumping evenlet https://review.rdoproject.org/r/#/c/17401 and crossing fingers12:18
amoralejfrippe75, nodes are self-registered12:19
ykarelamoralej, ack12:19
number80amoralej: ack12:19
amoralejfirst time they try to contact to some api or scheduler, iirc12:19
frippe75yes, but how. Where does that registration take place?12:19
amoralejmmm12:19
amoralejdon't remember who actually creates it12:20
amoralejwhat is strange is that node is registered12:20
amoralejyou se in the service list command12:20
frippe75I'm trying to trace that back.12:20
amoralejcould you remove it?12:20
*** jpena is now known as jpena|lunch12:20
*** shardy has joined #rdo12:20
amoralejand see what happens?12:20
frippe75db row delete?12:20
amoralejnop12:20
amoraleji think there is a command?12:21
frippe75ok12:21
amoralejopenstack compute service delete12:21
frippe75found it! thanks!12:22
ykarelatleast for placement resource provides it logs in nova-compute when creating those, for cellv2 i think it's seperate12:22
ykarelcommand12:22
amoralejyou may need to restart nova-compute12:22
rdogerritMerged rdoinfo master: Update eventlet to 0.24.1  https://review.rdoproject.org/r/1740112:23
frippe75amoralej, 1 of 1 compute services failed to delete.12:23
amoralejwth12:23
frippe75providing the id for the delete12:23
number80Interesting, eventlet in fedora uses new python dependency generator12:23
amoralejfrippe75, that's output of the command?12:23
frippe75Unexpected API Error. Please report this at ...bugs...12:23
amoralejin my environment it went fine12:24
amoralejyou must have some error12:24
amoralejin some log12:24
amoralejfrippe75, i removed it, restart nova-compute and it gets recreated12:24
frippe75Unexpected API Error. Please report this at12:24
frippe75in the nova-api.log ..... return db.compute_node_get_all_by_host(context, host)12:25
amoralejand then nova-manage cell_v2  discover_hosts12:25
frippe75it tries to find the host entry but it's not there (as we discovered) :-)12:25
amoralejadds it to the cell12:25
amoralejcan you paste the full error messate?12:25
amoralejmessage?12:26
*** pcaruana has quit IRC12:26
frippe75amoralej, https://pastebin.com/raw/V7N9XmLq     the delete requires it to find it in the db first.12:27
amoralejbut it's in openstack compute service list --service nova-compute12:27
amoralejso, it must me something like half-created12:27
frippe75The discovery only maps it into the cell, it has to be there first. thats whats missing.12:27
frippe75on my install table compute_nodes in nova db is empty.12:28
amoralejmmm12:28
amoralejso, it does not exist as compute12:29
frippe75question is what gets it in there in the first place.12:29
amoralejbut it must exist as service12:29
amoralejsomething like that12:29
amoralejwe can try something12:29
amoralejthere is a parameter host=12:29
amoralejin /etc/nova/nova.conf12:29
frippe75haven't seen that one. I think packstack relies heavily on defaults.12:30
amoralejyeah, i know12:30
amoralejit's not modified12:30
amoralejit's recomended to keep it default12:30
amoralejbut just to test12:30
amoralejset it12:30
amoralejhost=openstack12:30
frippe75you know which section?12:30
amoralejif you look for current_hostname in nova.conf you'll find it12:31
amoralejyou can "ping openstack" ?12:31
amoralejusing short name?12:31
frippe75found it12:31
frippe75i mean the host entry... not the solution12:32
frippe75testing12:33
amoralejchange that and restart openstack-nova-compute12:33
amoralejyou should see a new service appear with name openstack12:34
frippe75amoralej,  ComputeHostNotFound_Remote: Compute host openstack could not be found.12:37
amoralejthat's expected12:38
amoralejfirst time12:38
amoralejbut does it creates it?12:38
frippe75amoralej, two service short and fqdn... short is enabled12:38
amoralejgood12:38
amoralejnow, try the discover_hosts12:38
frippe75ran discover_hosts and list_hosts   nothing12:39
amoralejouch12:39
amoralejand you can't find any new ERROR in logs?12:39
frippe75so the actual start-up should create the entry. thats the plan?12:40
amoralejyes12:40
amoralejstart-up creates the entry but does not discover12:40
amoralejthen you run discover and they appear12:40
amoralejthat's the expected behavior12:40
amoralejand what i found12:40
amoralejyou can resolve12:40
amoralejopenstack12:40
amoralejand fqdn?12:40
frippe75yes resolves both short and fqdn.12:41
amoralejand don't you see any other error?12:42
amoralejit may not be an error12:42
amoralejmmm12:42
amoralejbut something in api.log12:42
frippe75All my errors since deploy... https://pastebin.com/raw/D2hpLhwq12:43
*** moguimar has quit IRC12:44
amoralejthere has to be something at around 2018-11-23 13:34:02.54312:49
amoralejin api.log12:49
amoralejwarnings or something12:49
amoralejwhen trying to add the new compute node12:49
amoralejyou are in debug, right?12:49
*** moguimar has joined #rdo12:50
rdogerritAlfredo Moralejo proposed rdoinfo master: Bump OVS to 2.10.0-6 in Rocky and Stein  https://review.rdoproject.org/r/1745512:51
*** jfrancoa has quit IRC12:52
frippe75amoralej, debug=True in defaults section of nova.conf... stop/start12:53
amoralejyeap12:53
*** BOKALDO has quit IRC12:59
*** jfrancoa has joined #rdo13:06
frippe75amoralej, thanks for all your support! friday and i'm gonna not do openstack for 24 hours at least. Gonna try to deploy queens next time just to see if that works out. Coming from newton....13:07
amoralejok13:08
amoralejbut rocky should work13:08
amoralejthere must be something environment-specific13:08
amoralejbut dunnow what13:08
frippe75will try other hardware as well13:08
frippe75not that it should matter. let's see.13:09
*** jpena|lunch is now known as jpena13:15
*** holser|lunch is now known as holser_13:31
*** hseipp has quit IRC13:36
ykarelamoralej, jpena https://review.rdoproject.org/r/#/c/17463/13:37
ykarelamoralej, can u also clear vitrage FTBFS13:37
ykarelhttps://review.rdoproject.org/r/#/c/17440/13:38
ykarelthere is openstack-placement FTBFS also13:38
ykarelmay be lyarwood is taking care off ^^13:38
amoralejykarel, yeah, let me fix vitrage13:39
ykarelack13:39
*** quiquell is now known as quiquell|lunch13:39
lyarwoodykarel: yeah there's a placement bugfix posted that should land today13:39
ykarellyarwood, okk cool13:40
ykarelamoralej, so should i send patch to run oooq CloudSIG running with trunk to not run on changes with rdo.yml or u are taking care with some other patch?13:42
rdogerritMerged rdoinfo master: Enable congress-tempest-plugin, tempestconf for Fedora  https://review.rdoproject.org/r/1746313:42
*** lvdombrkr has quit IRC13:42
ykarelamoralej, also the removal -release CloudSig jobs we discussed today13:42
amoralejykarel, can you send the one to ignore rdo.yml and remove -release ones13:43
amoralejyeap13:43
ykarelamoralej, okk sending13:43
rdogerritAlfredo Moralejo proposed openstack/vitrage-distgit rpm-master: Remove collector subpackage  https://review.rdoproject.org/r/1744013:44
rdogerritYatin Karel created config master: Do not run oooq jobs when non deps are updated  https://review.rdoproject.org/r/1746613:49
ykarelamoralej, ^^13:49
*** ksambor has quit IRC13:53
ykarelamoralej, don't we need to consider: Shouldn't the -graph package provide openstack-vitrage-collector for some time to not break deployment tools13:54
amoraleji think it's not a good idea13:55
amoralejas it's not providing collector service13:55
*** moguimar has quit IRC13:55
amoralejso, even if it installs it, it will fail to enable services etc...13:55
amoralejso i think we need to make sure upgrade path works13:55
amoralejusing obsoletes13:56
amoralejbut not provides13:56
ykarelhmm okk, so deployment tools should fix the usage then13:56
amoralejanyway, i'd prefer the maintainer to provide feedback13:56
amoralejykarel, yes13:56
amoralejit's vitrage in p-o-i?13:56
amoraleji think so, iirc13:56
ykarelyes13:56
ykarelscenario00113:57
ykarelokk seems disabled13:57
ykareli see  $enable_vitrage          = false with RedHat family, so disabled13:57
*** moguimar has joined #rdo13:58
amoralejok13:59
amoralejthen we will not break the promotion13:59
*** amoralej is now known as amoralej|lunch14:00
ykarelyes14:01
*** ykarel is now known as ykarel|away14:01
*** quiquell|lunch is now known as quiquell14:01
*** kushalAgrawal has joined #rdo14:03
*** BOKALDO has joined #rdo14:04
*** akrivoka has quit IRC14:11
*** akrivoka has joined #rdo14:12
*** diw has joined #rdo14:14
*** chkumar|ruck has quit IRC14:15
apevecdalvarez, amoralej, bummer, only ppc64le failed :( https://cbs.centos.org/koji/taskinfo?taskID=62107414:21
apevecI thought tests are disabled on non-x86_6414:21
dalvarez:(14:22
dalvarezyeah they were... amoralej|lunch remember?14:22
apevecmaybe it was in 2.10, I'll cherrypick14:24
*** ykarel|away has quit IRC14:25
*** Son_Goku has quit IRC14:30
*** mcornea has joined #rdo14:37
*** frippe75 has quit IRC14:40
apevecyeah, that was it14:41
*** jfrancoa has quit IRC14:41
apevecdalvarez, amoralej FYI branch with 2.9.3 rebase is https://github.com/rdo-common/openvswitch/tree/common-rdo-2.9.x14:41
apevecI've cleaned up some old branches in that repo14:41
apevecin -2.9.0 it's build for ocata used by rdocloud14:42
apevecI'll add review on top which updates it in Ocata14:42
*** diw has quit IRC14:47
*** ccamposr has quit IRC14:47
*** ykarel|away has joined #rdo14:54
*** trown|outtypewww has quit IRC14:54
*** jfrancoa has joined #rdo14:55
*** quiquell is now known as quiquell|off14:55
*** hseipp has joined #rdo14:55
*** trown|brb has joined #rdo14:58
rdogerritMerged config master: Do not run oooq jobs when non deps are updated  https://review.rdoproject.org/r/1746615:01
*** rgogunskiy has quit IRC15:04
*** moguimar has quit IRC15:05
*** dpawlik has quit IRC15:06
dalvarezamoralej|lunch: apevec https://review.rdoproject.org/r/17455 zuul liked it ! :)15:10
dalvarezand i tested the update on a packstack setup15:10
rdogerritMerged openstack/vitrage-distgit rpm-master: Remove collector subpackage  https://review.rdoproject.org/r/1744015:11
*** amoralej|lunch is now known as amoralej15:11
amoralejdalvarez, cool15:11
rdogerritAlfredo Moralejo created config master: Add oooq fs030 (ovn) jobs to experimental pipeline  https://review.rdoproject.org/r/1746715:19
rdogerritAlan Pevec proposed rdoinfo master: Bump OVS package to 2.9.3 for Queens  https://review.rdoproject.org/r/1383915:19
amoralejapevec, jpena let's merge https://review.rdoproject.org/r/#/c/17453/ and https://review.rdoproject.org/r/#/c/17467/ to test ongoing ovs updates15:19
amoraleji've tested it in https://review.rdoproject.org/r/17454 and works fine15:20
amoraleji'll do the same and add other FS which includes ceph that we can use for ansible updates, i.e.15:21
rdogerritAlfredo Moralejo proposed config master: Add oooq fs030 (ovn) jobs to experimental pipeline  https://review.rdoproject.org/r/1746715:26
rdogerritAlfredo Moralejo proposed config master: Add oooq fs030 (ovn) jobs to experimental pipeline  https://review.rdoproject.org/r/1746715:27
apevecamoralej, virtual +2 from me, need to leave now15:28
*** apevec has quit IRC15:28
amoralejok15:28
amoralejthx15:28
*** shardy has quit IRC15:36
*** cylopez has joined #rdo15:40
*** ykarel|away has quit IRC15:43
rdogerritMerged rdo-jobs master: Add new oooq jobs running FS030 (ovn) to master, rocky and queens  https://review.rdoproject.org/r/1745315:44
*** ykarel|away has joined #rdo15:45
rdogerritMerged config master: Add oooq fs030 (ovn) jobs to experimental pipeline  https://review.rdoproject.org/r/1746715:52
*** fultonj has quit IRC15:57
rdogerritAlfredo Moralejo proposed rdoinfo master: Bump rdoinfo stein-uc tags to latest upper-constraints  https://review.rdoproject.org/r/1736415:57
rdogerritAlfredo Moralejo proposed rdoinfo master: Bump rdoinfo stein-uc tags to latest upper-constraints  https://review.rdoproject.org/r/1736415:57
*** pcaruana has joined #rdo16:00
*** kopecmartin is now known as kopecmartin|off16:01
*** dpawlik has joined #rdo16:04
*** dpawlik has quit IRC16:08
*** jfrancoa has quit IRC16:18
rdogerritrdo-trunk created openstack/tripleo-common-distgit rpm-master: openstack-tripleo-common: failed to build 1b589e3  https://review.rdoproject.org/r/1746816:27
rdogerritJavier Peña proposed openstack/placement-distgit rpm-master: Add placement-status and placement-manage  https://review.rdoproject.org/r/1741016:28
lyarwoodthanks jpena16:31
rdogerritrdo-trunk created openstack/tripleoclient-distgit rpm-master: python-tripleoclient: failed to build 2a2fe4f  https://review.rdoproject.org/r/1746916:32
*** diw has joined #rdo16:44
*** rpittau has quit IRC16:45
rdogerritrdo-trunk created openstack/tripleo-validations-distgit rpm-master: openstack-tripleo-validations: failed to build ef60d4b  https://review.rdoproject.org/r/1747016:49
*** cylopez has quit IRC16:53
*** jchhatba_ has quit IRC16:54
rdogerritrdo-trunk created openstack/tripleo-heat-templates-distgit rpm-master: openstack-tripleo-heat-templates: failed to build 4030103  https://review.rdoproject.org/r/1747116:57
*** holser_ has quit IRC17:00
rdogerritrdo-trunk created openstack/tripleo-heat-templates-compat-distgit rpm-master: openstack-tripleo-heat-templates-compat: failed to build aed844e  https://review.rdoproject.org/r/1747217:01
*** jpich has quit IRC17:07
*** dtantsur is now known as dtantsur|afk17:07
*** slaweq has quit IRC17:16
*** diw has quit IRC17:17
*** hseipp has quit IRC17:27
*** jpena is now known as jpena|off17:33
*** lsm5 has quit IRC17:38
*** AppelonD has quit IRC17:51
*** derekh has quit IRC18:00
*** ykarel|away has quit IRC18:22
*** amoralej is now known as amoralej|off18:40
*** d0ugal has quit IRC18:41
*** d0ugal has joined #rdo18:57
*** paramite has quit IRC19:13
*** Son_Goku has joined #rdo19:16
*** akrivoka has quit IRC19:18
*** artom has quit IRC19:19
*** artom has joined #rdo19:19
*** Son_Goku has quit IRC19:26
*** Son_Goku has joined #rdo19:29
*** Son_Goku has joined #rdo19:33
*** Son_Goku has quit IRC19:34
*** jtomasek has quit IRC19:40
*** rfolco has quit IRC19:40
*** gregwork has joined #rdo19:42
gregworkdoes anyone know if rhosp is supported on ibm p-series/power arch platform?19:42
gregworkbeing able to have power arch az's to run that arch specific vm's would be pretty fantastic if it exists19:43
*** Son_Goku has joined #rdo19:44
*** Son_Goku has joined #rdo19:53
*** Son_Goku has quit IRC19:54
*** Son_Goku has joined #rdo20:03
*** Son_Goku has quit IRC20:05
*** Son_Goku has joined #rdo20:12
rdogerritrdo-trunk created openstack/heat-agents-distgit rpm-master: openstack-heat-agents: failed to build 08ffe6b  https://review.rdoproject.org/r/1747320:13
*** Son_Goku has quit IRC20:14
*** diw has joined #rdo20:16
rdogerritrdo-trunk created openstack/tripleo-common-distgit rpm-master: openstack-tripleo-common: failed to build 1b589e3  https://review.rdoproject.org/r/1747420:24
*** gfidente has quit IRC20:25
rdogerritrdo-trunk created openstack/tripleoclient-distgit rpm-master: python-tripleoclient: failed to build 2a2fe4f  https://review.rdoproject.org/r/1747520:29
rdogerritrdo-trunk created openstack/tripleo-heat-templates-compat-distgit rpm-master: openstack-tripleo-heat-templates-compat: failed to build aed844e  https://review.rdoproject.org/r/1747620:34
*** diw has quit IRC20:38
*** slaweq has joined #rdo20:40
*** dtrainor__ has joined #rdo20:41
*** kushalAgrawal has quit IRC20:42
*** BOKALDO has quit IRC20:48
rdogerritrdo-trunk created openstack/tripleo-heat-templates-distgit rpm-master: openstack-tripleo-heat-templates: failed to build 0eeb0d4  https://review.rdoproject.org/r/1747720:48
rdogerritrdo-trunk created openstack/tripleo-validations-distgit rpm-master: openstack-tripleo-validations: failed to build e7ee943  https://review.rdoproject.org/r/1747820:54
*** holser_ has joined #rdo21:12
*** slaweq has quit IRC21:21
*** Son_Goku has joined #rdo21:35
*** pcaruana has quit IRC21:46
*** chem` has quit IRC21:54
*** mcornea has quit IRC22:10
*** holser_ has quit IRC22:10
*** Son_Goku has quit IRC22:19
*** Son_Goku has joined #rdo22:29
*** diw has joined #rdo22:38
*** Son_Goku has quit IRC22:41
*** diw has quit IRC22:43
*** Son_Goku has joined #rdo22:43
*** dhill_ has quit IRC22:56
*** slaweq has joined #rdo23:18
*** slaweq has quit IRC23:27
*** Son_Goku has quit IRC23:40
*** Son_Goku has joined #rdo23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!