Thursday, 2020-07-16

« Wednesday, 2020-07-15 Index Friday, 2020-07-17 »
*** rcernin has quit IRC00:06
*** rcernin has joined #openstack-ironic00:13
*** Lucas_Gray has joined #openstack-ironic01:01
*** Goneri has quit IRC01:23
*** uzumaki has joined #openstack-ironic01:23
openstackgerritVerification of a change to openstack/ironic failed: Use min_command_interval when ironic does IPMI retries  https://review.opendev.org/74116901:30
*** tkajinam has quit IRC01:55
*** tkajinam has joined #openstack-ironic01:55
*** mkrai has joined #openstack-ironic02:19
*** Lucas_Gray has quit IRC02:20
*** Lucas_Gray has joined #openstack-ironic02:32
*** mkrai has quit IRC02:33
*** Lucas_Gray has quit IRC02:40
*** ociuhandu has joined #openstack-ironic02:46
*** ociuhandu has quit IRC02:50
*** Lucas_Gray has joined #openstack-ironic02:55
*** tzumainn has quit IRC02:55
*** mkrai has joined #openstack-ironic03:04
*** ricolin_ has joined #openstack-ironic03:25
*** rh-jelabarre has quit IRC03:46
*** gyee has quit IRC04:15
*** diurnalist has quit IRC04:16
*** yaawang has quit IRC04:25
*** yaawang has joined #openstack-ironic04:26
openstackgerritShivanand Tendulker proposed openstack/ironic master: Fixes to skip validation of in-band deploy steps before agent boot  https://review.opendev.org/74126704:51
*** abdysn has joined #openstack-ironic04:56
*** yaawang has quit IRC05:26
*** yaawang has joined #openstack-ironic05:27
*** Lucas_Gray has quit IRC05:32
*** Lucas_Gray has joined #openstack-ironic05:33
*** Lucas_Gray has quit IRC05:50
arne_wiebalckGood morning, ironic!06:09
*** diurnalist has joined #openstack-ironic06:13
*** janders has quit IRC06:14
*** diurnalist has quit IRC06:18
iurygregorygood morning arne_wiebalck and Ironic!06:43
*** mkrai has quit IRC06:43
*** mkrai_ has joined #openstack-ironic06:43
arne_wiebalckhey iurygregory o/06:43
rpittaugood morning ironic! o/06:49
*** mkrai_ has quit IRC06:57
iurygregorygood morning rpittau o/06:59
rpittauhey iurygregory :)07:00
*** ociuhandu has joined #openstack-ironic07:02
openstackgerritRiccardo Pittau proposed openstack/ironic master: DNM TEST CI  https://review.opendev.org/74138907:06
*** janders has joined #openstack-ironic07:19
uzumakigood morning! o/07:22
uzumakiiurygregory, rpittau arne_wiebalck \o/07:22
iurygregoryhey uzumaki good morning!07:23
rpittauhey uzumaki :)07:23
uzumakihow's it going y'all07:23
jandersg'day guys07:23
uzumakihope no more CI breaks iurygregory07:23
uzumakimorning janders !07:24
iurygregoryhey janders o/07:24
iurygregoryuzumaki, lol many problems this week =)07:24
rpittaugood prevening janders :)07:24
arne_wiebalckhey uzumaki rpittau janders o/07:25
rpittauhey arne_wiebalck :)07:25
uzumakiiurygregory, argh.. I'd say that sucks, but I have a feeling you love the problems xD iurygregory07:29
uzumakibonjour arne_wiebalck ! o/07:30
iurygregoryeveryone here love problems07:30
iurygregorywe work with hardware lol07:30
uzumakiiurygregory, haha! that's so true xD07:31
*** yaawang has quit IRC07:32
*** yaawang has joined #openstack-ironic07:33
*** ociuhandu has quit IRC07:35
*** rcernin has quit IRC07:48
*** tkaprol has joined #openstack-ironic07:59
*** yaawang has quit IRC08:02
*** yaawang has joined #openstack-ironic08:03
*** k_mouza has joined #openstack-ironic08:04
*** lucasagomes has joined #openstack-ironic08:09
*** tkaprol has quit IRC08:12
*** tkaprol has joined #openstack-ironic08:12
arne_wiebalcksalut uzumaki o/08:13
*** bnemec has quit IRC08:14
*** rcernin has joined #openstack-ironic08:14
uzumakicomment ca va monsieur arne_wiebalck ?08:14
uzumakij'espere tu vas bien?08:14
arne_wiebalckça va très bien, et toi?08:16
*** bnemec has joined #openstack-ironic08:17
*** mkrai has joined #openstack-ironic08:24
iurygregoryyay CI is working a bit better after the devstack change merged08:25
iurygregoryfingers crossed08:25
jandersfor https://storyboard.openstack.org/#!/story/1618014 I suppose the main area of interest is ironic/conductor/steps.py, correct?08:31
jandersthings seem to have moved around a little since the old patches are proposed, just checking if my thinking is correct08:31
janderss/are/were08:34
jandersdropping out for ~10mins, brb08:50
*** janders has quit IRC08:50
uzumakica va bien aussi.. merci08:51
*** derekh has joined #openstack-ironic08:53
*** rcernin has quit IRC09:07
openstackgerritRiccardo Pittau proposed openstack/ironic master: Force autospec in test_deployments  https://review.opendev.org/74140809:13
*** dtantsur|afk is now known as dtantsur09:15
dtantsurmorning ironic09:15
rpittauhey dtantsur :)09:15
dtantsurokay, do we have the CI back already? it seems that one of the devstack fixes has merged.09:18
iurygregorydtantsur, seems to be happy09:18
iurygregoryat least gave some +109:18
dtantsurokie cool09:18
rpittauI put up a test patch, waiting for results09:19
rpittauit looks kinda good so far09:19
iurygregoryironic-lib and ironic are happy at least =)09:19
iurygregoryI got +1 on the fixes09:19
iurygregoryand they are on gate09:19
rpittauyeah, I confirm the status is all green09:20
*** janders has joined #openstack-ironic09:23
jandersback09:24
rpittaujanders: that's the right place to start looking, yeah :)09:26
jandersrpittau: thank you! :)09:28
rpittaunever seen the CI so green at the first try since a while https://review.opendev.org/#/c/741389/ :)09:30
patchbotpatch 741389 - ironic - DNM TEST CI - 1 patch set09:30
dtantsurheh09:33
*** ociuhandu has joined #openstack-ironic09:35
jandersgreen is good red is bad09:36
*** ociuhandu has quit IRC09:40
*** tkaprol has quit IRC09:40
*** ricolin_ has quit IRC09:45
*** Lucas_Gray has joined #openstack-ironic09:50
*** k_mouza has quit IRC09:51
openstackgerritRiccardo Pittau proposed openstack/ironic master: Enforce autospec in test_notification_utils  https://review.opendev.org/74141909:54
openstackgerritRiccardo Pittau proposed openstack/ironic master: Enforce autospec in test_deployments  https://review.opendev.org/74140809:55
openstackgerritRiccardo Pittau proposed openstack/ironic master: Enforce autospec in test_notification_utils  https://review.opendev.org/74141909:55
iurygregoryred on CI is bad09:55
*** k_mouza has joined #openstack-ironic09:57
*** dmellado has quit IRC09:59
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Rework installation documentation for an easier start  https://review.opendev.org/74142010:06
dtantsurcould I get some reviews on ^^^ please?10:06
dtantsurit's needed for our web site launch10:06
*** dmellado has joined #openstack-ironic10:07
jandersif 1) my review is any good (no +2) and 2) can be done tomorrow, I am happy to be one of the reviewers dtantsur would that help?10:13
*** rcernin has joined #openstack-ironic10:15
dtantsurjanders: yes, any review is good. and even if it merges by tomorrow, your feedback is still welcome10:16
dtantsuriurygregory: ugh, I'm afraid we're actually running jobs with 2x partition images10:16
dtantsurI'm looking at https://zuul.opendev.org/t/openstack/build/1db56b6892ec4c169ffb133496c6a647/logs (from the ironic-lib patch)10:17
iurygregorydtantsur, in the main config ?10:17
dtantsurit looks like in both runs partition images are used10:17
jandersdtantsur ok! will do, thanks!10:18
iurygregoryWUT10:19
jandersI will probably wrap up for the day after the meeting at 2100hrs my time (42min from now) hence tomorrow timeframe10:19
iurygregorybut on tempest the test is setting to wholedisk10:19
dtantsurI'm not sure how it can be happening.. but I put W-1 on the ironic-lib patch until we understand10:19
iurygregoryyeah10:19
iurygregoryI will take a look at it10:19
iurygregoryand other jobs10:19
dtantsurit shouldn't be happening, it works the same in the standalone job.. but let's check once more10:20
iurygregoryyeah, weird10:21
dtantsuriurygregory: I actually remember having this concern back when reviewing the patch: we don't pass image_id to create_server10:23
dtantsuriurygregory: https://opendev.org/openstack/tempest/src/branch/master/tempest/common/compute.py#L113 :(10:23
dtantsurthis uses https://opendev.org/openstack/ironic/src/branch/master/devstack/lib/ironic#L300910:24
dtantsurwhich is per job, not per test10:24
dtantsurwe need to update boot_instance to explicitly pass image_id=self.image_ref to create_server10:25
dtantsuriurygregory: ^^10:25
iurygregoryoh10:29
* iurygregory updates tempest plugin10:29
iurygregorysorry for that .-.10:30
*** rcernin has quit IRC10:30
openstackgerritVerification of a change to openstack/ironic-inspector failed: Respond so a ContentType is included  https://review.opendev.org/74034410:34
dtantsuriurygregory: don't worry, happens. I should have noticed that when reviewing..10:35
iurygregory{'image_id': self.image_ref}) would do the job I think10:38
iurygregorylet's see10:38
dtantsuriurygregory: could you also check the tempest plugin docs?10:39
dtantsurI think we don't cover this new aspect there10:39
iurygregorythis would need to go into **create_kwarg10:39
iurygregorycreate_kwargs10:40
iurygregoryfor boot_instance10:40
iurygregoryso basically I only need to add param in def baremetal_server_ops(self)10:42
iurygregoryso we can pass the info to boot_instance10:42
iurygregorydtantsur, does it makes sense to you?10:44
*** janders has quit IRC10:48
dtantsuriurygregory: yep. and let's blow up in boot_instance if we don't receive image_ref (either directly or from self)?10:50
*** janders has joined #openstack-ironic10:52
iurygregorydtantsur, ++10:54
iurygregoryI need to eat something (have a meeting in 5 min) the delivery just arrived10:55
iurygregorywill push the change in a few minutes10:55
*** ricolin_ has joined #openstack-ironic11:23
iurygregorynow back to work11:58
*** rh-jelabarre has joined #openstack-ironic11:59
janderswrapping up for the day, I will see you tomorrow12:16
janderso/12:16
dtantsurg'night janders12:18
*** rcernin has joined #openstack-ironic12:21
openstackgerritLucas Alvares Gomes proposed openstack/ironic master: DO NOT REVIEW: Test Ironic with DevStack's neutron-legacy plugin  https://review.opendev.org/74125612:44
iurygregoryI know lucasagomes will make things work for us =) including the ipv6 job =P12:54
lucasagomesiurygregory, I hope so heh... I just changed a few NEUTRON_* variables from that job because I believe that was what caused it to fail12:55
* lucasagomes fingers crossed12:55
iurygregoryfingers crossed also12:55
iurygregorydtantsur, I think I know what happen with tempest .-. since the inital value was None set on the function didn't work, I've extracted to class and added mandatory_attr = ['image_ref']12:56
*** k_mouza has quit IRC12:57
iurygregoryI will push the change and test with the ironic-lib to see how it goes12:57
TheJuliagood morning13:02
dtantsurmorning TheJulia13:02
dtantsuriurygregory++13:02
rpittaugood morning TheJulia :)13:02
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops  https://review.opendev.org/74144513:03
TheJuliaugh https://review.opendev.org/#/c/74034413:04
patchbotpatch 740344 - ironic-inspector - Respond so a ContentType is included - 6 patch sets13:04
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-lib master: Update ironic-lib jobs  https://review.opendev.org/74110513:04
* iurygregory fingers crossed13:04
iurygregorygood morning TheJulia =)13:04
TheJuliaBad gateway on explicit content type. I'm wondering if apache really wants body content13:04
dtantsurI wonder if it sends content-length correctly..13:09
TheJuliaI guess the apache source is going to be after I have some coffee13:11
TheJuliaI've never seen apache have an issue like this with a back-end app or webserver before13:13
TheJuliayeah, coffee and mod_proxy.c13:14
*** ociuhandu has joined #openstack-ironic13:15
*** rloo has joined #openstack-ironic13:20
openstackgerritJakub Libosvar proposed openstack/ironic master: DNM: Just test the patch under with devstack change  https://review.opendev.org/74122613:20
*** k_mouza has joined #openstack-ironic13:24
dtantsurTheJulia: wait, are we even using apache or uwsgi in the CI?13:30
TheJuliaeveryone is13:31
TheJuliait is the devstack default afaik13:31
*** rcernin has quit IRC13:31
dtantsurI think it's uwsgi13:31
TheJuliaI think it is and I think the proxy code path is still invoked in that13:31
dtantsur(unless we're using uwsgi behind apache)13:31
TheJuliawe are13:32
dtantsursigh13:32
dtantsurtoo many components, each may be misbehaving13:32
TheJuliayup13:32
dtantsurwhat I don't get is why it's different for ironic13:32
dtantsurwe're setting up inspector and ironic using the same devstack functions13:33
* dtantsur wonders if we can set up tcpdump to print all headers that are sent13:33
TheJuliayup, proxypass to uwsgi's socket13:34
dtantsurmm, yeah, and tcpdump probably won't help13:34
TheJuliadtantsur: do we return truly bare 202 and 204 messages with ironic?13:35
dtantsurI *think* we do at least one of them13:35
dtantsurI think our 'created' responses do have bodies though13:35
TheJulialooks like we do have a couple13:37
TheJuliabut a fair number that seem to be expected to kind of return something based on test13:38
* dtantsur is going to test on bifrost13:38
TheJulia++ less moving parts13:40
TheJuliaflask does return a Content-Length header13:41
dtantsur< HTTP/1.1 202 Accepted13:42
dtantsur< Content-Length: 013:42
dtantsurthis is ironic provision action13:42
iurygregorywe finally have the upstream issue for mariadb =) https://jira.mariadb.org/browse/MDEV-2319313:44
iurygregory\o/13:44
dtantsur< HTTP/1.1 202 ACCEPTED13:44
dtantsur< Content-Type: text/html; charset=utf-813:44
dtantsur< Content-Length: 013:44
dtantsurthis is inspector ^^^ we do have content-type which we don't have with ironic13:44
*** tzumainn has joined #openstack-ironic13:45
dtantsurTheJulia: the only difference I can see from curl -v ^^^13:45
dtantsuriurygregory: \o/13:45
TheJuliaiurygregory: does this mean we broke mariadb!?!?13:47
iurygregorythey broke us13:47
TheJuliaI know :)13:47
iurygregory=)13:47
TheJuliadtantsur: It is clear, the question is why does apache hate us13:48
iurygregory"The bug must have something with the version rebase from 10.4.12 to 10.4.13"13:48
*** abdysn has quit IRC13:48
iurygregoryso they just need to figure out and fix13:48
*** yaawang has quit IRC13:49
TheJuliawe likely need a release note noting the issue for now13:49
TheJuliasince it is having to go upstream, it may take a while13:49
dtantsurTheJulia: could you check https://review.opendev.org/#/c/739964/ at some point?13:50
patchbotpatch 739964 - ironic - Wipe agent token during reboot or power off - 3 patch sets13:50
*** yaawang has joined #openstack-ironic13:50
TheJuliaapproved13:51
dtantsurthx13:56
dtantsuralso a bit less of a priority, but we may need https://review.opendev.org/741420 before we get a new website pointing at it13:56
patchbotpatch 741420 - bifrost - Rework installation documentation for an easier start - 1 patch set13:56
TheJulialooks like the apache options/knobs are really all centered around socket... I wonder if uwsgi is causing apache to be offended13:58
dtantsurbut not in ironic?14:00
TheJuliaI wonder what is in the nova logs14:01
TheJuliahttps://bugs.launchpad.net/ubuntu/+source/uwsgi/+bug/1776624 <-- sure sounds familiar14:02
dtantsurand then we have these amqp error in inspector https://zuul.opendev.org/t/openstack/build/29531ff158ec4bbba2298a5d5c43a019/log/controller/logs/screen-ironic-inspector-api.txt?severity=414:02
openstackLaunchpad bug 1776624 in uwsgi (Ubuntu) "apache2 proxy reports internal error when proxying requests to uwsgi on Ubuntu 16.04" [Undecided,New]14:02
*** Goneri has joined #openstack-ironic14:02
dtantsuryeah, but this mentions big endian machine14:02
TheJuliayeah14:02
dtantsurI talked to oslo folks about these error, they could only suggest something locking up the heartbeating thread14:03
TheJuliaSo the thing that makes me think the message bus is not involved is the fact that ironic-inspector logs the 202 and no errors on the webserver14:03
TheJuliathe heartbeat thread to the message bus?14:04
*** ianychoi has joined #openstack-ironic14:05
openstackgerritMerged openstack/ironic master: Use default timeout for all jobs  https://review.opendev.org/74112314:05
dtantsurTheJulia: yeah14:05
TheJuliaand the first logs I looked into, I think I spotted weirdness with the message bus like 1.5-2 minutes after post14:07
openstackgerritMerged openstack/ironic master: Allow deleting nodes with a broken driver  https://review.opendev.org/73970214:07
openstackgerritVerification of a change to openstack/ironic failed: Do not validate driver on changing non-driver fields  https://review.opendev.org/73969914:07
dtantsurhere goes the 16th recheck \o/14:08
openstackgerritDmitry Tantsur proposed openstack/ironic stable/ussuri: Allow deleting nodes with a broken driver  https://review.opendev.org/74146014:08
openstackgerritDmitry Tantsur proposed openstack/ironic stable/train: Allow deleting nodes with a broken driver  https://review.opendev.org/74146114:08
openstackgerritMerged openstack/bifrost master: Rework installation documentation for an easier start  https://review.opendev.org/74142014:13
TheJulialooks like we're fairly locked into the apache -> uwsgi pattern at this point14:14
TheJuliathe only thing I can think of at this point is to fire up a fresh test VM with the same devstack config and see if I can reproduce it14:15
dtantsur++14:15
TheJuliawhat is interesting is I did ?5? runs sending a dict back and none of them threw the error. Changing it to just be a content type brought it back14:16
dtantsuradding content-type does nothing, flask sends text/html by default anyway14:17
dtantsurI wonder if we could *remove* content-type14:18
dtantsur(ironic does not send it for empty responses)14:18
TheJuliawhy yes, I think we can remove it14:19
TheJuliahmm I think I saw an example of doing so somepalce14:19
dtantsuralso, returning a body with HTTP 204 is, strictly speaking, incorrect14:19
*** mkrai has quit IRC14:21
TheJuliayeah14:23
openstackgerritRiccardo Pittau proposed openstack/ironic master: Enforce autospec in some api tests  https://review.opendev.org/74146814:28
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Agent power interface  https://review.opendev.org/73559114:28
* TheJulia cracks open the flask source code14:34
openstackgerritRiccardo Pittau proposed openstack/ironic master: migrate testing to ubuntu focal  https://review.opendev.org/73944814:35
openstackgerritRiccardo Pittau proposed openstack/ironic master: Enforce autospec in some api tests  https://review.opendev.org/74146814:37
openstackgerritLucas Alvares Gomes proposed openstack/ironic master: Ironic to use DevStack's neutron"-legacy" module  https://review.opendev.org/74125614:45
*** tkajinam has quit IRC14:47
dtantsurlucasagomes: FYI we also use "new" neutron functions in https://opendev.org/openstack/metalsmith/src/branch/master/.zuul.yaml#L6114:47
lucasagomesdtantsur, yeah I will send a patch to it as well14:47
dtantsurthanks man!14:47
lucasagomesdtantsur, I am doing one now for python-ironic-inspector-client14:48
dtantsurehhhm, why do we have explicit configuration there14:48
lucasagomesdtantsur, no problem!14:48
lucasagomesdtantsur, idk heh it does explicitly enable the neutron-* services and disable the q-* services14:48
lucasagomesdtantsur, maybe I can fix it let me see14:49
dtantsurlucasagomes: it inherits ironic-inspector-base, I think you can kill devstack_services with fire14:49
lucasagomesdtantsur, lol ok14:50
dtantsurmetalsmith is a different story, it builds its jobs from the ground up (because it doesn't use tempest)14:50
* lucasagomes looks at ironic-inspector-base first14:50
*** mkrai has joined #openstack-ironic14:51
openstackgerritLucas Alvares Gomes proposed openstack/python-ironic-inspector-client master: Tempest job to use the inherited devcstack_services value  https://review.opendev.org/74147414:55
lucasagomesdtantsur, let's see if it works ^14:55
*** Goneri has quit IRC14:55
*** derekh has quit IRC14:55
*** hoonetorg has quit IRC14:55
*** bdodd has quit IRC14:55
*** irclogbot_3 has quit IRC14:55
*** Goneri has joined #openstack-ironic14:57
*** derekh has joined #openstack-ironic14:57
*** hoonetorg has joined #openstack-ironic14:57
*** bdodd has joined #openstack-ironic14:57
*** irclogbot_3 has joined #openstack-ironic14:57
openstackgerritLucas Alvares Gomes proposed openstack/metalsmith master: Metalsmith to use DevStack's neutron"-legacy" module  https://review.opendev.org/74147614:59
openstackgerritLucas Alvares Gomes proposed openstack/python-ironic-inspector-client master: Tempest job to use the inherited devstack_services value  https://review.opendev.org/74147414:59
*** diurnalist has joined #openstack-ironic15:01
TheJuliaand so, it looks like there is no real escaping the inhouse werkerzug response handling at least with the integrated response class15:02
openstackgerritLucas Alvares Gomes proposed openstack/ironic-prometheus-exporter master: Prometheus-exporter to use DevStack's neutron"-legacy" module  https://review.opendev.org/74147715:02
lucasagomesI think those are all according to codesearch.openstack.org15:04
* dtantsur is on the verge of proposing reverting the agent token work....15:04
dtantsurthanks lucasagomes15:04
lucasagomesno problem o/ I will keep an eye on those patches15:05
dtantsurTheJulia: let me guess: we wipe driver_internal_info on driver updates?15:05
TheJuliaNot that I'm aware... but why? what is going on?15:06
iurygregorydtantsur, hey if image_ref is not found I can just raise an Exception ?15:07
dtantsurTheJulia: I have a node in fast-track with empty driver_internal_info15:07
dtantsurcompletely empty, so breaking both credential-less deploy and the agent token15:07
TheJuliaon driver update?15:07
dtantsurTheJulia: that's a guess, it's the only unusual thing that I've done15:08
dtantsuriurygregory: IMO yes15:08
* dtantsur will cry15:08
TheJuliaIt wouldn't surprise me if the driver update wiped the d_i_i15:08
iurygregorydtantsur, ok let's see what CI will say =)15:08
TheJuliaI don't think it has to, I think it was to keep cross-driver crud from getting mixed across drivers15:08
* TheJulia thinks this was a thing like 5 years ago15:09
dtantsurTheJulia: yup, we wipe driver_internal_info on driver updates \o/15:09
TheJuliakill that with fire?15:09
TheJuliaPreferably molten metal based fired15:09
TheJuliafire15:09
TheJuliaWorst comes to worst, I have a propane torch someplace15:09
dtantsurhttps://opendev.org/openstack/ironic/src/branch/master/ironic/objects/node.py#L44515:09
dtantsurkeeping driver_internal_info is potentially dangerous, but this is outright wrong15:10
dtantsurI honestly want to apply a propane torch to all our JSON fields15:11
TheJuliaheh15:12
TheJuliakeeping power on crossing does mean that we can't drop it all15:12
*** johnthetubaguy has quit IRC15:16
openstackgerritMerged openstack/ironic master: Add `get_node_network_data` to Neutron NetworkInterface  https://review.opendev.org/68889615:18
openstackgerritDmitry Tantsur proposed openstack/ironic master: Stop wiping driver_internal_info on node.driver updates  https://review.opendev.org/74148115:19
dtantsurTheJulia: ^^15:19
*** johnthetubaguy has joined #openstack-ironic15:20
openstackgerritMerged openstack/ironic master: Fixes to skip validation of in-band deploy steps before agent boot  https://review.opendev.org/74126715:20
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops  https://review.opendev.org/74144515:21
iurygregoryI forgot to push the change /me facepalm15:21
dtantsurheh15:21
TheJuliadtantsur: +215:21
TheJuliadtantsur: also left a comment with reasoning.15:21
dtantsuryep, thx15:22
*** gyee has joined #openstack-ironic15:25
*** ricolin_ is now known as ricolin15:30
openstackgerritVerification of a change to openstack/ironic failed: Decompose the core deploy step on iscsi and ansible deploy  https://review.opendev.org/73118015:31
dtantsurare my patches cursed or what?15:33
iurygregoryI can put a recheck for you dtantsur15:34
TheJuliadtantsur: not just yours15:35
TheJuliait is the freaking failure rate15:35
dtantsuriurygregory: done already, but thanks15:35
TheJuliahopefully the ironic-inspector CPU patch will merge which will make inspector jobs a little happier until we figure out the thing bad gateway15:35
dtantsur"GET /pxelinux.cfg/52-54-00-7e-b4-d2 HTTP/1.1" 40315:36
dtantsurHTTP 403 on generated netboot files \o/15:36
dtantsurdear bifrost, you're drunk, go home15:36
dtantsur(I know it's likely selinux, but omg)15:36
iurygregorywut?!15:37
dtantsuron the bright side: I've got credential-less deploy working \o/15:37
iurygregorybifrost decided to take PTO15:37
iurygregory"it's friday"15:38
iurygregory\o/15:38
TheJuliaI'm digging through flask and thinking friday sounds great15:38
dtantsur++15:38
dtantsurwell, at least I have ideas how to fix this problem (unlike the ones I'm having with metal3)15:39
TheJuliaoooooooohhhh ahhhh I might have figured out how to bend flask to my will15:46
TheJuliadtantsur: would you like to chat about metal3 things?15:46
dtantsurTheJulia: I'm already torturing stbenjam and shardy with my problems. or do you mean something else?15:47
TheJuliayour torturing them?!?15:48
TheJuliaimpressive!15:48
TheJuliaWell, a shoulder, maybe some ideas might bubble up?15:48
dtantsurThanks, I've switched to the 403 problem while the guys are staring at my logs15:49
dtantsurbut I may ping you later, especially if I keep having problems with tokens15:49
TheJulia++15:50
dtantsurtype=AVC msg=audit(1594913617.415:15520): avc:  denied  { read } for  pid=145600 comm="nginx" name="52-54-00-7e-b4-d2" dev="dm-0" ino=35352000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=lnk_file permissive=015:50
TheJuliaoh... wow15:51
dtantsurI wonder if ironic should try setting contexts on its files15:51
dtantsuralthough.. tripleo works fine without it15:51
TheJuliaso one thing I HAVE noticed is depending on your system partition layout, defaults kind of go sideways because the tftpboot selinux context constraints get copied15:51
TheJuliabecause the kernel and selinux is evil at times15:51
dtantsuraudit2allow suggests the problem is that the file is a link15:52
TheJuliainteresting15:53
TheJuliaso there is something in ironic's code that will link or copy15:53
TheJuliawhich is wehre that goes sideways15:53
*** uzumaki has quit IRC15:55
dtantsuryep, the problem was in the symlink15:55
TheJuliasign, looks like werkerzug forces a content type15:57
dtantsurugh15:59
openstackgerritDmitry Tantsur proposed openstack/bifrost master: selinux: allow nginx to read symbolic links  https://review.opendev.org/74149115:59
dtantsurokay, the selinux fix ^^15:59
*** lucasagomes has quit IRC16:00
dtantsurmm, I need a release note, right?16:01
rpittaudtantsur: yeah :)16:01
rpittau I was reading it now16:01
openstackgerritDmitry Tantsur proposed openstack/bifrost master: selinux: allow nginx to read symbolic links  https://review.opendev.org/74149116:02
dtantsuryay, logged in! a deployment completely without BMC credentials :)16:03
rpittaunice :)16:03
TheJuliaI'm kind of at the point of wondering if I can take their response object, and rip out the offending parts....16:03
dtantsurheh16:03
dkingWhen doing inspection, I am able to receive the LLDP information for the main interface, the one that PXE boots. However, even after setting add_ports=all, and the port is now updated, it still doesn't show the switch port information. I'm assuming that's because the interface isn't up. Is there a way to get that inspected?16:04
dtantsurdking: maybe try https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/config.py#L157 ?16:05
dtantsurand/or increase the timeout? https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/config.py#L10616:06
*** tkaprol has joined #openstack-ironic16:07
*** tkaprol has quit IRC16:15
TheJuliaWell, I'm getting a literal "Content-Type: None" now...16:16
* dtantsur facepalm16:16
TheJuliaand now ''16:17
*** johnthetubaguy has quit IRC16:17
dtantsurit DOES insist on having content-type, doesn't it?16:17
TheJuliayeah, workerzug does basically re-cast everything through its response object, even things populated using it16:18
TheJuliawell, flask and werkerzug16:18
TheJuliaso pretty much there is no escaping content-type16:18
TheJuliaclosest we seem to be able to get is to generate a respone class, change it before handing it back where it gets pushed in based on the make_response handling16:19
TheJuliathat _does_ allow us to tweek the default mimetype setting so it is "None" or ''16:19
TheJuliabut yeah, the code path is fairly explicit, it is going to try to add it16:20
dtantsurI should have raised it earlier.. but our current ideas don't explain why the only affected job is the one with inspector API/engine split?16:20
*** johnthetubaguy has joined #openstack-ironic16:20
TheJulianope.... but that launches differently doesn't it?16:20
dtantsurprobably.. yeah16:21
dtantsuryeah, right. the regular jobs should just lunch ironic-inspecor16:21
TheJuliayeah16:22
TheJuliastandalone service versus wsgi app in apache16:22
TheJuliaso what if just on 202, since user friendly messages are somewhat expected yet not required, we just go back to the empty json dict and see if we can let that roll forward?16:23
TheJuliawe could likely recheck that a number of times too to see if that truly does resolve our apache unhappiness16:24
*** ociuhandu_ has joined #openstack-ironic16:24
TheJuliaits technically a api change, but at the same time we don't actually cover that as a behavior or requirement in terms of the post response body contents16:24
dtantsurTheJulia: let's give it a try (with a few rechecks)16:25
* TheJulia goes and refills her coffee mug while she decides... to just rm -r -f her current ironic-inspector repo, or to just reset back to change set ?4?16:26
dtantsurwe may even consider a meaningful body16:26
dtantsurlike, start returning the introspection status when starting it16:26
*** ociuhandu has quit IRC16:27
*** ociuhandu_ has quit IRC16:28
*** sshnaidm is now known as sshnaidm|afk16:28
dtantsuroh, my metal3 troubles proved to be an actual breakage that now affects their CI! I just was the first to get hit by it16:30
TheJuliawheeeeeeeee16:36
dkingdtantsur: That sounds promising. Thank you. That looks like it's done when building the IPA image? So, I'm going to have to look back into that, as I think I'm doing that with DIB.16:39
dtantsurdking: you can provide these options via kernel parameters16:40
*** dtantsur is now known as dtantsur|afk16:40
dtantsur|afkhave a good rest of the day folks16:41
TheJuliaarne_wiebalck: you around?16:44
TheJuliadtantsur|afk: goodnight!16:44
openstackgerritJulia Kreger proposed openstack/ironic-inspector master: DNM: Respond so a ContentType is included  https://review.opendev.org/74034416:46
TheJuliadtantsur|afk: ^^^ will need later revision, but we can at least recheck it a half dozen times or so and see how it fares16:46
openstackgerritBob Fournier proposed openstack/ironic stable/train: Use min_command_interval when ironic does IPMI retries  https://review.opendev.org/74149616:55
*** uzumaki has joined #openstack-ironic16:56
*** Lucas_Gray has quit IRC16:58
*** derekh has quit IRC17:00
*** k_mouza has quit IRC17:16
openstackgerritBob Fournier proposed openstack/ironic stable/train: New configuration parameter to use ipmitool retries  https://review.opendev.org/74149617:18
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops  https://review.opendev.org/74144517:23
dkingI'm trying to understand how I can collect LLDP data for my second (and other interfaces that aren't the PXE interface). add_port=all in inspector.conf allows the port to be added during inspection. However, only the main interface has patches to add it's value. I don't see anything for that in my introspection rules, so I'm assuming it's done by the local_link_connection or lldp_basic processing hooks?17:24
*** k_mouza has joined #openstack-ironic17:25
openstackgerritMerged openstack/ironic master: Implement get_deploy_steps for AgentRAID  https://review.opendev.org/74122517:27
openstackgerritMerged openstack/ironic master: Wipe agent token during reboot or power off  https://review.opendev.org/73996417:27
TheJulianice17:27
TheJuliadking: yeah, it is done by the processing hooks and should do it for all interfaces added17:27
TheJulia_should_ being the operative keyword there.17:27
openstackgerritVerification of a change to openstack/ironic failed: Change non-tinyipa jobs to use multiple cores  https://review.opendev.org/74082017:27
dkingIt gets both ports added, but the second doesn't have any switch info in it. I added `ipa-inspection-dhcp-all-interfaces=1`. So, maybe I should modify ipa-inspection-dhcp-wait-timeout ?17:29
dking...or perhaps ipa-lldp-timeout ?17:30
*** k_mouza has quit IRC17:34
TheJuliaI seem to remember it tries to collect it regardless as long as line carrier is up17:42
TheJuliais the second interface up? is it dhcp'able, or not?17:42
*** mkrai has quit IRC18:13
openstackgerritJulia Kreger proposed openstack/ironic master: Guard conductor from consuming all of the ram  https://review.opendev.org/72648318:35
openstackgerritJulia Kreger proposed openstack/ironic master: DNM/Test Force Insufficent memory in CI  https://review.opendev.org/74151918:35
TheJulianot feeling great, going to lay down for a little bit18:38
dkingTheJulia: Sorry, just seeing that. Thank you, I hope that you feel better. I might try again tomorrow.18:53
zer0c00l_I am starting to write SPEC for the RFE https://storyboard.openstack.org/#!/story/2007839. I intend to allow operators to change kickstart files per node. So the operator would get the custom kickstart files into /etc/ironic/kickstart/ and set --driver-info inst.ks=<nameofkstemplate> is that acceptable?19:09
*** andrein has quit IRC19:14
*** Goneri has quit IRC19:14
*** andrein has joined #openstack-ironic19:15
openstackgerritVerification of a change to openstack/ironic-inspector failed: Make inspector base CI job use 2 CPUs  https://review.opendev.org/74074619:16
TheJuliazer0c00l_: instnace_info please, not driver_info.19:24
*** Goneri has joined #openstack-ironic19:24
TheJuliaspecifically because you can hae different access privs on fields19:24
TheJuliaso less privilged users  won't be abel to change the driver info, but will be able to setup an instance19:25
TheJuliaalso, I suspect anything file accessible to the conductor would be okay, unless you see a reason to make it downloadable. Locking it in to /etc won't work on systems with... I'm drawing a blank on the name but only certian paths are actually writable19:26
*** ricolin has quit IRC19:36
openstackgerritJulia Kreger proposed openstack/ironic-inspector master: DNM: Respond so a ContentType is included  https://review.opendev.org/74034419:38
openstackgerritJulia Kreger proposed openstack/ironic master: Remove old driver name from cross-gating job  https://review.opendev.org/74125419:44
openstackgerritJulia Kreger proposed openstack/ironic master: Follow-up on blocking port deletions  https://review.opendev.org/74103919:47
openstackgerritJulia Kreger proposed openstack/ironic master: Add ``ipxe_template`` override option  https://review.opendev.org/73984119:54
zer0c00l_TheJulia: ack20:16
zer0c00l_TheJulia: The kickstart file has to be copied into the httpboot directory because it needs to be transferred to the node20:16
zer0c00l_Basically the user has to work with operator to setup their custom kickstart file/template, then they can use instance_info to tell the node to use their custom template20:17
zer0c00l_Is that a good?20:17
*** vesper11 has quit IRC20:20
*** vesper11 has joined #openstack-ironic20:21
*** tzumainn has quit IRC20:32
TheJuliazer0c00l_: you could use the embeded iamge retrieval method which will do glance image/httpurl/swift/fileondisk transparently20:32
TheJuliaI guess20:33
openstackgerritMerged openstack/ironic master: Do not validate driver on changing non-driver fields  https://review.opendev.org/73969920:50
openstackgerritJulia Kreger proposed openstack/ironic master: DNM/Test Force Insufficent memory in CI  https://review.opendev.org/74151920:52
*** Lucas_Gray has joined #openstack-ironic21:01
openstackgerritMerged openstack/ironic master: Stop wiping driver_internal_info on node.driver updates  https://review.opendev.org/74148121:06
stevebakermorning21:09
openstackgerritVerification of a change to openstack/ironic failed: Decompose the core deploy step on iscsi and ansible deploy  https://review.opendev.org/73118021:12
zer0c00l_TheJulia: i see. So conductor will use the embedded image retrival method to get the kickstart into the httpboot directory. The installer will be told about that file using kernel cmdline inst.ks=http(s)://server/<uuid>/ks.cfg21:36
openstackgerritJulia Kreger proposed openstack/ironic master: iPXE ISO Ramdisk booting  https://review.opendev.org/72946421:37
zer0c00l_Basically https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/chap-anaconda-boot-options21:38
JayF+++1 re: TheJulia zer0c00l_: you could use the embeded iamge retrieval method which will do glance image/httpurl/swift/fileondisk transparently21:59
JayFthat's sorta what I was trying to suggest in slack, I didn't realize it had been made so easy21:59
JayFcan you upload arbitrary things to glance? like a txt file and just call it an image of type=None?21:59
openstackgerritMerged openstack/ironic stable/ussuri: Use min_command_interval when ironic does IPMI retries  https://review.opendev.org/74116922:07
*** uzumaki has quit IRC22:16
openstackgerritMerged openstack/ironic master: Change non-tinyipa jobs to use multiple cores  https://review.opendev.org/74082022:22
*** ociuhandu has joined #openstack-ironic22:25
*** ociuhandu has quit IRC22:29
*** aedc_ has joined #openstack-ironic22:36
*** aedc has quit IRC22:39
*** rcernin has joined #openstack-ironic22:43
*** rcernin has quit IRC22:47
*** aedc_ has quit IRC22:52
*** rcernin has joined #openstack-ironic22:54
*** aedc has joined #openstack-ironic22:56
*** aedc has quit IRC22:57
*** aedc has joined #openstack-ironic22:58
*** tkajinam has joined #openstack-ironic23:02
*** aedc has quit IRC23:10
*** gyee has quit IRC23:16
*** hoonetorg has quit IRC23:16
*** bdodd has quit IRC23:16
*** irclogbot_3 has quit IRC23:16
*** gyee has joined #openstack-ironic23:22
*** hoonetorg has joined #openstack-ironic23:22
*** bdodd has joined #openstack-ironic23:22
*** irclogbot_3 has joined #openstack-ironic23:22
*** rloo has quit IRC23:53
« Wednesday, 2020-07-15 Index Friday, 2020-07-17 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!