Tuesday, 2019-08-27

*** aaronsheffield has quit IRC00:16
*** bh526r has quit IRC00:49
*** kranthikirang has joined #airshipit00:54
*** michael-beaver has quit IRC00:55
*** kranthikirang has quit IRC01:00
openstackgerritRajeshwari Dharwadkar proposed airship/deckhand master: Fix for opensuse gate issue  https://review.opendev.org/67870601:32
openstackgerritRajeshwari Dharwadkar proposed airship/shipyard master: Fix for opensuse image build issue.  https://review.opendev.org/67869901:42
*** dukov has quit IRC01:43
*** Kuirong has joined #airshipit01:46
openstackgerritRajeshwari Dharwadkar proposed airship/deckhand master: Fix for opensuse image build issue.  https://review.opendev.org/67870601:50
*** mbeierl has quit IRC01:54
*** seaneagan has quit IRC03:11
*** nishantkr has quit IRC03:43
*** gkadam has joined #airshipit03:51
*** gkadam has quit IRC03:51
*** kskels has quit IRC05:19
*** jamesgu_ has quit IRC05:52
*** dukov has joined #airshipit06:02
*** dddpak has joined #airshipit06:05
openstackgerritDmitry Ukov proposed airship/airshipctl master: Add isogen subcommand for bootstrap  https://review.opendev.org/67612106:06
openstackgerritDmitry Ukov proposed airship/airshipctl master: [WIP] Introduce container interface  https://review.opendev.org/67585106:06
*** dddpak has left #airshipit06:08
*** kranthikirang has joined #airshipit06:19
*** kranthikirang has quit IRC06:24
*** dr_feelgood has joined #airshipit06:34
*** dr_feelgood has quit IRC06:52
*** georgk has quit IRC07:07
*** georgk has joined #airshipit07:08
*** kranthikirang has joined #airshipit07:14
*** kranthikirang has quit IRC07:18
openstackgerritDmitry Ukov proposed airship/airshipctl master: Add logic to isogen subcommand  https://review.opendev.org/67585107:23
openstackgerritDeepak proposed airship/airshipctl master: test  https://review.opendev.org/67874508:04
*** dddpak has joined #airshipit08:28
*** dddpak has quit IRC08:35
*** dddpak has joined #airshipit08:36
*** dddpak has quit IRC08:36
*** dddpak has joined #airshipit08:37
*** avolkov has joined #airshipit08:46
*** kranthikirang has joined #airshipit09:02
*** kranthikirang has quit IRC09:06
*** licanwei has joined #airshipit09:10
*** dddpak has quit IRC09:20
*** kranthikirang has joined #airshipit09:56
*** kranthikirang has quit IRC10:00
*** dpawlik has quit IRC10:04
*** LoicL35 has joined #airshipit10:13
*** kranthikirang has joined #airshipit10:50
*** kranthikirang has quit IRC10:54
*** rezroo has quit IRC10:56
*** dpawlik has joined #airshipit11:18
*** souradage has joined #airshipit11:20
*** kranthikirang has joined #airshipit11:44
*** souradage has quit IRC11:47
*** kranthikirang has quit IRC11:48
*** dddpak has joined #airshipit11:54
openstackgerritRoman Gorshunov proposed airship/armada master: Fix: Armada Exceptions docs rendering on RTD  https://review.opendev.org/67861812:25
*** kranthikirang has joined #airshipit12:38
*** mbeierl has joined #airshipit12:41
*** phasespace has joined #airshipit12:41
*** kranthikirang has quit IRC12:42
*** mbeierl has quit IRC12:45
*** bh526r has joined #airshipit12:49
*** aaronsheffield has joined #airshipit12:58
*** mbeierl has joined #airshipit13:00
*** mcriswel1 has joined #airshipit13:03
openstackgerritRoman Gorshunov proposed airship/armada master: Fix: Armada Exceptions docs rendering on RTD  https://review.opendev.org/67861813:09
openstackgerritRoman Gorshunov proposed airship/promenade master: Fix: Promenade Exceptions docs rendering on RTD  https://review.opendev.org/67862513:09
*** sthussey has joined #airshipit13:10
*** KeithMnemonic has joined #airshipit13:14
*** seaneagan has joined #airshipit13:30
*** kranthikirang has joined #airshipit13:32
*** ab2434_ has joined #airshipit13:33
*** dddpak has quit IRC13:33
*** kranthik_ has joined #airshipit13:37
*** kranthikirang has quit IRC13:38
*** ian-pittwood has joined #airshipit13:45
*** michael-beaver has joined #airshipit13:55
alexanderhugheshello!  meeting kicking off in a few minutes, agenda here https://etherpad.openstack.org/p/airship-meeting-2019-08-2713:56
*** nishantkr has joined #airshipit13:56
*** nmimi has quit IRC13:56
*** kskels has joined #airshipit13:59
mattmceuen#startmeeting airship14:00
openstackMeeting started Tue Aug 27 14:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: airship)"14:00
openstackThe meeting name has been set to 'airship'14:00
mattmceuen#topic Rollcall14:00
CobHeado/14:00
*** openstack changes topic to "Rollcall (Meeting topic: airship)"14:00
alexanderhugheso/14:00
nishantkro/14:00
kskels|o/14:00
howello/14:00
dukovo/14:00
mattmceuenty for sharing agenda alexanderhughes14:00
ian-pittwoodo/14:00
mattmceuen<alexanderhughes> Alexander Hughes hello!  meeting kicking off in a few minutes, agenda here https://etherpad.openstack.org/p/airship-meeting-2019-08-2714:00
seaneagano/14:00
sthusseyhere14:00
mattmceuenGM/GE all!14:00
alexanderhughesyw :)14:01
mattmceuenwe'll get going in just a minute14:01
mattmceuenfeel free to add any more topics to discuss to this week's agenda14:01
CobHeadWhat time is it over there?14:01
michael-beavero/14:01
mattmceuen9am here in Saint Louis! How about you CobHead?14:01
aaronsheffieldo/14:01
CobHead4pm in Norway14:02
*** jemangs has joined #airshipit14:03
mattmceuenthanks all for joining, and away we go:14:03
mattmceuen#topic Airship YouTube playlist14:03
*** openstack changes topic to "Airship YouTube playlist (Meeting topic: airship)"14:03
mattmceuengo for it alexanderhughes14:03
alexanderhughesJust a quick announcement, Chris and the rest of the awesome folks at OSF have set us up a playlist on their youtube channel14:04
alexanderhughes#link https://www.youtube.com/playlist?list=PLKqaoAnDyfgp8YjZbzjVrmZBJR9thV27y14:04
*** jamesgu_ has joined #airshipit14:04
alexanderhughesthese are a collection of interviews/presentations etc. regarding Airship.  if you have more content you'd like to see added please shoot me a message14:04
hogepodgeSunny and the rest of the marketing team deserve most of the credit.14:04
mattmceuenthat is awesome14:04
CobHeadNice :)14:05
alexanderhughesyes, huge thanks to Sunny for her hard work on the media channels14:05
mattmceuenty both for spearheading that (and Sunny for doing the hard work!! :)14:05
hogepodgeIt became a valuable exercise for all of the projects so thank you for the suggestion.14:05
mattmceuennice14:05
mattmceuenhopefully we get a few more on there after the Shanghai summit as well14:06
CobHeadAny presentations due for the summit?14:06
sthusseyMay be worthwhile to have someone demo some of the projects there14:07
mattmceuen++++ sthussey14:07
sthusseyrather than having only presentations and interviews14:07
CobHeadDemos are always nice14:07
alexanderhughesabsolutely, there's a lot more we can do with this.  I'd love to see more content on the channel, making it a one stop shop for anyone curious about airship and use cases, how certain things work etc14:07
mattmceuenalexanderhughes, were you already prepping a Shanghai presentation list or did I make that up?14:07
*** roman_g has quit IRC14:08
alexanderhughes#link https://www.airshipit.org/blog/airship-at-the-open-infrastructure-summit-shanghai-in-november.html14:08
alexanderhughesthere's a list of Airship presentations on the blog14:08
*** roman_g has joined #airshipit14:08
roman_go/14:09
mattmceuenah, ty for sharing that14:09
mattmceuenAlright, I think we can move on unless anything else on this topic?14:09
alexanderhughesnope :)14:10
mattmceuen#topic Airship GUI Project Name14:10
*** openstack changes topic to "Airship GUI Project Name (Meeting topic: airship)"14:10
mattmceuenSo last week we agreed to create a new project14:10
mattmceuenBut we didn't figure out anything awesome to call it14:10
mattmceuenI sent out an email on the ML to remedy that14:10
*** kkalina has joined #airshipit14:10
*** kkalina has left #airshipit14:10
*** kkalina has joined #airshipit14:10
mattmceuenDetails in that email - please read it and plan to share your opinions on the best names in next week's IRC meeting14:11
mattmceuenThat's all on that topic unless any questions14:11
*** pramchan has joined #airshipit14:11
mattmceuen#topic New Project Proposal:  airship/isogen14:12
*** openstack changes topic to "New Project Proposal: airship/isogen (Meeting topic: airship)"14:12
mattmceuenDmitry: the floor is yours, can you please give us an overview of the plan for this project?14:12
dukovThanks14:13
mattmceuenI know discussion has been going on in the Bootstrap SIG, so folks who are not regulars may need to be caught up just a bit14:13
dukovBasically as a part of airshipctl development we need Live CD iso to bootstrap ephemeral k8s cluster14:14
dukovthis cluster is going ot be used for new (Target) cluster deployment using metal314:14
dukovephemeral cluster get destroyed then14:15
dukovso to build an LiveCD/LiveUSB image it was decided to use container with set of scripts14:15
dukovso there is a question: where to keep Dockerfile and scripts14:16
dukovwe also decided to create separate repo for all such image builders (e.g for Debian SuSe etc)14:16
mattmceuenMakes sense to me.  So this new project will not contain the airshipctl module for generating the iso; that code would live in the airshipctl project, right?14:17
jemangso/14:17
dukovit turned out that we may need more the one type of images14:17
mattmceueno/ jemangs14:17
roman_gI support to have another repo, but for all images/Dockerfiles all together, which would be supplementary to the airshipctl.14:17
dukovso may be isogen is not good name for that repo14:17
roman_gairshipctl-images?14:18
dukov+1 roman_g14:18
pramchanyoue ,ean not using quay.io?14:18
CobHeadWouldn't it make more sense to keep it all time builders in the same repo? If the project ends up with builders for 6 distros, it might be difficult to keep track of them all?14:18
roman_gsimilar to openstack-helm-images14:18
mattmceuenCobHead: there's overhead involved in project creation/maintenance, so if it doesn't get out of hand I like starting with a single repo.  Agree roman_g openstack-helm-images is some good prior art there14:19
pramchancan we not add these standard specified images to quay.io14:20
roman_gCobHead: yes, that's what I would like to have - one repo for all images related to the airshipctl14:20
mattmceuenpramchan:  yep we can store build images in quay.io, agree14:20
roman_gpramchan: they would be published there, but we discuss where to host sources14:20
mattmceuenwe just need a project for the unbuilt images14:20
pramchanOK that makes sense14:20
pramchanfor that we can create patches under tools and add those sources14:21
CobHeadA question for a later date is whether or not the Airship project team should be responsible for all the builders, or if some should be sourced out to the community?14:21
pramchanwe need to maintain at-least two builds to ensure variety14:22
mattmceuenCobHead:  agree and I would frame it up straw-man style this way, an airship-images project would be only for image sources that need to be created or built by the airship team (potentially forked from a project) only when necessary, and would not take image build responsibility away from other projects that are a more natural fit (e.g. deckhand would continue to build its own image)14:23
mattmceuenAgree?14:23
*** souradage has joined #airshipit14:23
CobHeadSounds good.14:24
jamesgu+114:24
roman_gwe could welcome other container images as long as they have high quality code, well covered by CI tests, and there would be someone willing to support them14:24
pramchan+114:24
roman_g_integrated_ CI tests14:25
mattmceuenAgree roman_g14:25
mattmceuendo we like the airship-images convention?  Or do we want to come up with something more creative?14:25
mattmceuen(in terms of project name I mean)14:25
CobHeadKeep it nice and simple14:26
howell+114:26
roman_gairship-images or airshipctl-images?14:26
pramchan+114:26
alexanderhughesdescriptive is nice.  there's no question what airship-images is14:26
sthusseywhy wouldn't this go into airshipctl if this is only to support code being added there?14:26
pramchanairship-images as ui will also use it14:26
nishantkrairship-images +114:26
jamesgu+1 to airship-images14:27
mattmceuenI'd go for airship-images roman_g, I think it makes sense for other non-cli images we may need in the future14:27
pramchanlooks majority is airship-imaged go for it14:27
pramchanairship-images typo14:27
mattmceuenI think you are right pramchan, I will sip coffee for 30s in case anyone comes up with a counterpoint14:28
michael-beaver+1 sounds good to me14:28
pramchanwelcome to colambian cofee!!!14:28
roman_gsthussey: we were discussing on bootstrap call, that there are high chances that we would have more airshipctl modules implemented as containers.14:28
sthusseyright14:28
roman_gsthussey: so I see reasons to keep it with airshipctl14:29
CobHeadmodules as containers - I like it :)14:29
sthusseyIs the code in the airshipctl module going to be coupled with the container contents?14:29
dukovand not only airshipctl i guess ...14:29
mattmceuennope the airshipctl module code will live in airshipctl I believe14:29
mattmceuenfor isogen, etc14:29
sthusseycoupled as in co-dependent14:29
mattmceuenwhere etc = built-in modules14:29
pramchanI think given the trends it's a no brainer and best practice by definition will be use Docker conatainer or some container that fits your needs14:29
sthusseya change in module code requires a change in the container build - or vice versa14:30
roman_gsthussey: line 65 and onwards https://etherpad.openstack.org/p/Airship_bootstrap14:30
mattmceuenwhat do you think dukov in terms of coupling?  Do you expect a stable interface?14:30
sthusseyreally just an ecosystem difference. OSF projects seem to shy away from a repo having more than a single artifact produced from it14:32
roman_gprobably not super stable. but intention was to use Depends-On: xxx for cross-repo CI checks14:32
roman_gand to use cross-CI Zuul jobs14:32
dukovmattmceuen: well the idea is to define a contract of communication between container and airshipctl which means any image that implements interface can be used14:32
roman_g*cross-repo CI Zuul jobs14:32
dukovthat being said we still need a place where to keep couple default ISO builder container14:33
dukov*containers14:33
mattmceuenmakes sense dukov, it would have to be pretty stable or else it would wreak some havoc14:33
sthusseyjust seems odd to me that images built for this and the airshipui are considered coupled enough to be in the same repo, but putting this in the same repo as airshipctl isn't. anyway, carry-on.14:34
dukovtechnically  we can use airshipctl for that but eventually it will create a mess14:34
mattmceuenairshipui - I think container build code would live in-tree with application code, but I haven't been tied into those convos and aren't an authority14:35
*** dpawlik has quit IRC14:35
mattmceuenOk, I think we can move on unless anything else on this topic:14:35
roman_gsthussey: are you for separating supplementary images into one additional repo, or are you for keeping supplementary images (e.g. isogen) inside airshipctl?14:36
dukovso what is the conclusion? are we going to create separate repo ?14:36
sthusseyI was just bringing up a viewpoint. It sounds like the folks working on it are for this generic images repo14:37
sthusseyI tend to avoid repo sprawl14:37
sthusseybut airship is probably long past that point14:37
mattmceuenYep I think we still have consensus for airship-images (unless anyone de-consensizes).  Appreciate probing the pros/cons of this sthussey14:38
mattmceuenAgree on repo sprawl 100%14:38
mattmceuenI was leaning toward bundling this into airshipctl till learning more from dukov14:38
mattmceuenthanks for bringing this up and filling us in on progress dukov!14:39
dukovmattmceuen: you are welcome! I'll update CR accordingly then14:39
pramchanif we run into unanticipated issues we can always move it out later14:40
mattmceuenPerfect ty - then feel free to add that on to the agenda review list as well so we can put +1s on it14:40
mattmceuen#topic Has anyone been able to roll AIAB in the recent weeks14:40
*** openstack changes topic to "Has anyone been able to roll AIAB in the recent weeks (Meeting topic: airship)"14:40
mattmceuenCobHead I take it you're hitting some issues :(14:40
CobHeadYes, I can't for the life of me get past shipyard trying to build Armada. It's stuck there until it finally fails. Then it moves on to try submitting Horizon, but that inevitably fails too. Anyone else in here experienced this?14:40
CobHeadI initially suspected a timeout, so I tried bumping it, but to no avail.14:41
mattmceuenAnd just to clarify this is Treasuremap single-node AIAB, right?14:41
CobHeadYes.14:41
mattmceuenThat's what I thought - yeah, I tried a couple times a week and a half or so ago with two successes14:42
mattmceuenWe need to figure out your RC14:42
mattmceuenRemind me, are you running in a local VM or in a public cloud or?14:42
CobHeadLocal VM - however I've tried public cloud aswell.14:42
mattmceuensame behavior?14:42
CobHeadActually, it's different. Can't get past PostgreSQL in the cloud.14:43
CobHeadLocally it stops at Armada build.14:43
mattmceuenWell it oughta be working fine in both places.  Which one do you want to fix first14:43
CobHeadThere feels like there are some caveats here, and I'd like to resolve them on the top, so people who are unfamiliar with the project don't hit a dead end when they want to try AIAB.14:44
CobHeadI'd like to fix local first.14:44
mattmceuenAgree, may be some hidden assumption we need to ferret out14:44
mattmceuenOk - would you be free to power through this in the IRC channel after the meeting?14:44
pramchandoes AIAB stand for Airship in a Box ? or is it Airskiff my ignorance please clarify?14:44
CobHeadAirship in a Box ;)14:44
mattmceuenClose pramchan :)  Bottle14:44
pramchanok14:44
CobHeadmattmceuen: Absolutely. Can start deployment now.14:45
mattmceuenagain with the nautical theme14:45
CobHeadAfter we have gone through the changes for review, there is another thing I'd like to take up.14:45
pramchanIs it then etcd dependency that is fainling in local VM similar to PostgresSQL in cloud for AIAB?14:46
CobHeadAt this point, I have not a single clue.14:46
pramchanCan we not track the failure or error code?14:46
mattmceuenSince we only have a few mins left, let's save the troubleshooting for post-call -- but please stay tuned pramchan I want to get this fixed14:46
mattmceuen#topic Requests for Review14:46
*** openstack changes topic to "Requests for Review (Meeting topic: airship)"14:46
mattmceuen    These are copied from last week's; confirm whether more review is needed before meeting:14:47
mattmceuen    https://review.opendev.org/#/c/671575/ - Shipyard (one more +2 is needed)rt dependencies)14:47
mattmceuen    https://review.opendev.org/#/c/675440 (add airskiff suse cite and ci gate)14:47
mattmceuen    https://review.opendev.org/#/c/672688/ (Tungsten Fabric support in Treasuremap)14:47
mattmceuen    https://review.opendev.org/#/c/678311/ Airskiff dep fix in Treasuremap14:47
mattmceuen    https://review.opendev.org/#/c/671575/ Shipyard14:47
mattmceuen    https://review.opendev.org/#/c/677980/ Bump ucp keystone timeour14:47
mattmceuen    https://review.opendev.org/#/c/674409/ - shellcheck linting (this one - for divingbell) - need to evaluate if it's usefull to be implemented to other charts we host under airship/*.14:47
mattmceuen    https://review.opendev.org/#/c/678618/ - airship/armada - Fix: Armada Exceptions docs rendering on RTD14:47
mattmceuenLet's please get some eyeballs on these today, team14:47
CobHeadAnother thing, mattmceuen - which everyone should hear.14:47
CobHeadOr read, for that matter..14:47
mattmceuenYes please CobHead, go for it14:47
mattmceuen#topic Roundtable14:48
*** openstack changes topic to "Roundtable (Meeting topic: airship)"14:48
CobHeadhttps://review.opendev.org/#/c/672540/ Take a look at this commit. This commit removed the package dependencies for Airskiff, since they are being handled in OSH - but the OSH package install script is never handled by the airskiff script(s)14:48
mattmceuenahh14:48
CobHeadSo essentially, if you deploy Airskiff using the steps described in the docs - you will halt when Armada is built14:48
CobHeadSince that requires make.14:48
CobHeadMake is never installed.14:48
CobHeadLikewise with the Python client for Openstack14:49
CobHeadSince python-pip is never installed.14:49
CobHeadhttps://review.opendev.org/#/c/678302/14:49
CobHeadI've made some amends in that commit.14:49
mattmceuenYep - I hit the Python / pip issue yesterday14:49
CobHeadBut please - if there are dependencies removed - please also remove them at the gate. Zuul has no issues with this as those packages are listed in the gate.14:49
CobHeadBut not in the script - so the CI succeeds, while the developer fails.14:50
mattmceuenYeah14:50
mattmceuenGood catch, thanks for fixing that14:50
evgenylhttps://review.opendev.org/#/c/678311/ I think this should help with this.14:50
evgenylGate Ubuntu image has a lot of stuff pre-installed, this is the reason it does not fail.14:51
CobHeadIncluding make, I presume.14:51
CobHeadI think evgenyl's patch should be merged instead of mine, as it addresses both pip and the other dependencies.14:52
CobHeadI14:52
CobHeadI will abandon mine ;)14:52
mattmceuenok :)  great minds are thinking alike14:52
CobHeadThat was what I had to bring up :)14:53
mattmceuenI think we hit everything in the agenda, anything esle you'd like to discuss today team?14:53
mattmceuenI will be on vacation tomorrow through next Thursday, so don't expect much out of me :D14:54
pramchanCan i ask a question on baremetal?14:54
mattmceuensure pramchan14:54
roman_gsthussey: you wanted to review https://review.opendev.org/#/c/674409/ - shellcheck linting. I would appreciate a lot your opinion.14:55
roman_gThank you.14:55
pramchanI am trying to find a directory where I can submit the code, shall I call it baremetal as a new folder to start coding for Redfish boot for testing vBMS?14:56
pramchanvBMC14:56
roman_gstart anywhere, and we will find a right place :)14:56
mattmceuenpramchan: this code will be the Redfish equivalent of vBMC, right?14:57
pramchancool thx will do14:57
roman_gairship/airshipctl/tools/ <- probably somewhere here14:57
pramchanpartly yes will push to cmd later once I can test it14:57
roman_gas it would be part of gates, I think14:58
pramchanyes tools is where I will put the vBMC14:58
mattmceuenAgree with roman_g, get it out somewhere public and we can figure it out.  You can always make changes in a new github repo and then we can import it later when we know where it should go - we have done that many times14:58
mattmceuenthat's awesome14:58
pramchanthanks and I will try get it out the base by week end14:58
mattmceuenperfect - drop us a note in the IRC channel when its ready for a look!14:59
pramchanOK14:59
mattmceuenok I think we're out of time - great meeting all, appreciate your time & discussion14:59
mattmceuenhave a good week!14:59
mattmceuen#endmeeting14:59
*** openstack changes topic to "AIAB timeout (deafline) on tests is set too low by default for common hardware (i.e. desktops). Should ideally be higher by default (Meeting topic: airship)"14:59
openstackMeeting ended Tue Aug 27 14:59:43 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-08-27-14.00.html14:59
CobHeadThanks, and have a good week :]14:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-08-27-14.00.txt14:59
openstackLog:            http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-08-27-14.00.log.html14:59
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implement intermediary file validation  https://review.opendev.org/67864215:00
CobHeadThe AIAB deploy is running now.15:00
mattmceuenaaaaaand we're back to the stuck topic from last week's meeting as our channel topic, like sthussey pointed out yesterday15:00
mattmceuenif anyone is smarter than me at IRC and has the proper authority, should probably be something like:  #airshipit https://opendev.org/airship || https://wiki.openstack.org/wiki/Airship || https://review.opendev.org/#/q/projects:airship+status:open+NOT+label:Verified%253D-1+NOT+label:Workflow%253D-1+NOT+message:DNM+NOT+message:WIP15:01
CobHeadInteresting how it defaulted to my topic from last week though.15:02
mattmceuenCool CobHead.  Once the API server is up and running, something like `watch "kubectl get pods --all-namespaces | grep -v Completed" `  would give you some visibility into what's going on during the deployment15:02
sthusseyYou just need to run '/topic <msg>'15:02
sthusseybut you need to be a channel operator which is only 'openstack' and 'ChanServ'15:03
mattmceuenYeah, the part I forget is how to get yourself those privileges; I think there's a better way than "go ask openstack infra".  I.e. some kind of temporary op rights that you can grant yourself if you are correctly-permissioned15:04
mattmceuenbut I may be crazy15:04
CobHeadmattmceuen - AIAB prints out run status by default, so running that command yields only part of the picture (i.e what containers have a "Completed" status for the current run)15:07
CobHeadDoes AIAB produce any debug logs?15:08
openstackgerritKaspars Skels proposed airship/treasuremap master: Use MAAS VIP as DNS server for Seaworthy site  https://review.opendev.org/67865215:08
mattmceuenyep CobHead that's also good status.  Was the issue you were hitting that Armada was failing to deploy, or that Armada was failing to deploy other things?15:09
mattmceuenAlso - did you start with a fresh VM?  Sometimes cruft can accumulate if you run in a stale VM, so fresh VMs are especially good when troubleshooting issues15:10
CobHeadI think it was Armada deploying - but I'll send the output when it reaches that part :d15:10
CobHeadI always start afresh :)15:10
mattmceuen+115:10
CobHeadmattmceuen - I think etcd is failing. Let me send you some output.15:11
CobHeadDo you prefer paste.openstack or paste.ubuntu?15:12
CobHeadhttp://paste.openstack.org/show/KfhhfKrinHrIJppVNuHd/15:12
mattmceuenEither wfm :)15:12
mattmceuenplease try `kubectl logs airship-kubernetes-etcd-etcd-test --namespace kube-system`15:13
CobHeadNot much to go on there: http://paste.openstack.org/show/RKic1MgRs4OrXO1yjlfK/15:14
*** jeremy_moffitt has joined #airshipit15:14
mattmceuensure isn't15:14
CobHeadBut it continues15:14
CobHeadThe deploy, that is.15:14
mattmceuenAh, it recovered?15:14
CobHeadYeah, it does this on every run.15:15
mattmceuenI should have done this earlier, but I'll start up a fresh AIAB myself in a terminal so I can follow along with some lag, compare & contrast15:15
CobHeadSounds good :)15:16
roman_gmattmceuen:15:18
roman_gclarkb> roman_g: mattmceuen is an op15:18
CobHeadBehold the powers that be? o715:18
*** kranthik_ has quit IRC15:21
*** kranthikirang has joined #airshipit15:22
mattmceuenI think I'm a "potential op".  when I try to /topic I get "You're not a channel operator"15:24
mattmceuenSo I'm back to "how do I elevate my privs to op"15:25
sthussey@mattmceuen you need to op yourself w/ chanserv15:25
CobHeadPerhaps ask Chanserv ?15:25
CobHeadsthussey beat me to it15:26
sthussey'/msg chanserv op'15:26
*** ChanServ sets mode: +o mattmceuen15:26
CobHeadhttps://gist.github.com/xero/2d6e4b061b4ecbeb9f99 Excellent op reference.15:27
roman_gfungi> roman_g: mattmceuen should also be able to add additional channel access15:27
roman_gfungi> roman_g: something like `/msg chanserv access #airshipit add roman_g +Aeortv`15:27
mattmceuenwoo hoo that did it, thanks sthussey CobHead15:27
*** kranthikirang has quit IRC15:27
*** mattmceuen changes topic to "https://opendev.org/airship || https://wiki.openstack.org/wiki/Airship || https://review.opendev.org/#/q/projects:airship+status:open+NOT+label:Verified%253D-1+NOT+label:Workflow%253D-1+NOT+message:DNM+NOT+message:WIP"15:27
mattmceuenmission accomplished15:27
CobHead:D15:27
*** ChanServ sets mode: -o mattmceuen15:28
*** ian-pittwood has quit IRC15:31
CobHeadmattmceuen: CreateContainerConfigError with drydock-api container.15:36
mattmceuenany more detail in `kubectl describe pod` ?15:37
*** ian-pittwood has joined #airshipit15:38
CobHeadSeems to have recovered - but those are the logs from the container: http://paste.openstack.org/show/GrBrpHUZ6hX0Oz7PbUpm/15:38
sthusseythat is expected waiting for the MAAS chart deployment to finish and export a secret15:38
mattmceuenahh15:38
CobHeadAnd it seems to have recovered further.15:39
CobHeadOnly info messages now15:40
CobHeadPast genesis now15:43
mattmceuenawesome15:44
CobHeadSo it's now running the site deploy, which is where it stops after a while.15:46
CobHeadmattmceuen: Any containers to query while it is now running the site deploy?15:53
nishantkrCobHead: You could query shipyard to know what step is being run by the site deploy. command: `shipyard get actions`16:03
nishantkrfollowed by `shipyard describe <action-id>`16:03
CobHeadWhere is the shipyard bin located?16:04
nishantkrif it's running armada_build step, you could then look logs for armada-api pod as to what it is trying to do16:04
openstackgerritKaspars Skels proposed airship/treasuremap master: Updates on site authoring and deployment guide  https://review.opendev.org/67861716:07
CobHeadThis is the last line from the armada api pod (ucp tiller) http://paste.openstack.org/show/KEXD4wNuMYABryBEdpGV/16:09
CobHeadWhich is weird, as the keystone test passed long ago16:09
nishantkryou could try using this script - https://opendev.org/airship/airship-in-a-bottle/src/branch/master/tools/multi_nodes_gate/airship_gate/bin/shipyard.sh16:09
CobHeadWait - are you guys using https://opendev.org/airship/airship-in-a-bottle and not https://opendev.org/airship/treasuremap ?16:16
mattmceuenI am using the latter16:16
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implement intermediary file validation  https://review.opendev.org/67864216:17
mattmceuentreasuremap is the canonical place for AIAB single node.  Migrating the multi-node setup is still in flight though.16:17
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: [WIP] Implement intermediary validation  https://review.opendev.org/67864416:17
CobHeadWhy isn't shipyard.sh included in the treasuremap repo, or rather, cloned from the AIAB repo?16:17
michael-beaverThe treasuremap repository has a script under tools/airship that accomplishes that16:18
mattmceuenthat airship script is actually better, but I'm not sure whether it works with the aiab.sh-created shipyard, out of box?  Has anyone tried?16:18
*** dr_feelgood has joined #airshipit16:18
michael-beaverOnce my deployment is in a place where I have shipyard up I will try it and see if it works16:19
CobHeadIt works, alright.16:19
*** dr_feelgood has quit IRC16:19
CobHeadThe airship script, that is.16:19
mattmceuenfor my deployment, I'm on to the `armada_build` step -- same for you CobHead?16:20
CobHeadIndeed.16:20
CobHeadWhich it has been for the last 30 minutes or so16:20
CobHeadRough estimate16:20
evgenyltools/airship is used for aiab deployment scripts, so it should work https://opendev.org/airship/treasuremap/src/branch/master/tools/deployment/aiab/common/deploy-airship.sh#L7216:20
michael-beaverTo be clear that step does take quite a while16:20
mattmceuenThe best things to keep an eye on during this are probably 1) the armada pod logs, 2) watch for any crashlooping or otherwise unhealthy pods16:21
*** pramchan has quit IRC16:21
mattmceuennote:  those two things are what I look for when I am debugging chart deployment errors; for normal deployment status things the shipyard API (or airship script cli etc) provides a much friendlier interface...16:22
CobHeadArmada build failed ;\16:25
CobHeadI don't understand the shipyard logs command format. Giving it the action ID or step ID returns invalid namespace item.16:27
CobHeadOk, so the script ended now, after failing to publish Horizon.16:27
michael-beaveryou can run ./tools/airship shipyard logs step/<ID_NUMBER>/armada_build to get the logs for the armada_build step16:28
mattmceuendo you see any specifics in the armada logs, or any crashlooping pods?16:28
mattmceuenI need to run to an away from keyboard meeting, will be back in an hour16:28
CobHeadAha..16:30
CobHeadThere is a ton of logs here. Gonna use paste.ubuntu - https://paste.ubuntu.com/p/cwfGMvdJBk/16:31
CobHead..16:32
CobHeadIt seems like the OSH Keystone step is failing because of a deadline that is being exceeded :)16:32
CobHeadTime to bump more timeouts16:32
CobHeadSo, it seems like both keystone timeouts need to be set way higher. Will amend my commit to reflect that later.16:34
*** dr_feelgood has joined #airshipit16:37
CobHeadAnd I'm rerunning the deploy after setting both timeouts to 1600s16:44
*** kranthikirang has joined #airshipit16:45
*** kranthikirang has quit IRC16:51
kskelsI think it's the test timeout likely16:53
kskelskeystone itself usually deploys quick - but tests take really long time16:53
kskelsI think there was a patchet out there for it.. let me look16:53
kskelshttps://review.opendev.org/#/c/677980/16:53
kskelsI already gave +2 some time back.. I think this should solve the keystone test timeout issue16:53
CobHeadI've added test: \n ++yaml-compliant whitespace+timeout: 160016:53
CobHeadkskels: Yes, it is mine ;)16:54
CobHeadBut that is only for the keystone ucp test16:54
CobHeadNot for the Keystone test within Armada16:54
kskelsah - you mean for the openstack part16:54
kskelssure16:54
CobHeadYes16:54
kskelsyou would then need to bump that as well16:54
CobHeadYep. Deploy in progress.16:55
kskels(we are running 2 keystone intances - one for ucp, and one for openstack)16:55
CobHeadI'll also amend that patch16:55
*** arunkant_ has quit IRC16:59
openstackgerritRoman Gorshunov proposed airship/promenade master: Fix: Promenade Exceptions docs rendering on RTD  https://review.opendev.org/67862517:06
*** ian-pittwood has quit IRC17:17
openstackgerritRoman Gorshunov proposed airship/drydock master: Fix: Drydock Exceptions docs rendering on RTD  https://review.opendev.org/67862417:18
*** dr_feelgood has quit IRC17:18
openstackgerritRoman Gorshunov proposed airship/drydock master: Fix: Drydock Exceptions docs rendering on RTD  https://review.opendev.org/67862417:19
*** dr_feelgood has joined #airshipit17:23
*** dpawlik has joined #airshipit17:29
*** dpawlik has quit IRC17:34
mattmceuenmore keystones more problems17:36
mattmceuenfingers crossed CobHead17:36
* CobHead holds stones17:37
* mattmceuen ducks17:37
CobHeadCan't toss stones if I need them :d17:37
openstackgerritEvgeniy L proposed airship/shipyard master: [WIP] Fix opensuse gate by installing python3  https://review.opendev.org/67868417:47
openstackgerritScott Hussey proposed airship/promenade master: Configurable genesis package install timeout  https://review.opendev.org/67892117:47
openstackgerritEvgeniy L proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782317:49
openstackgerritEvgeniy L proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782317:54
*** bh526r has quit IRC17:58
*** ian-pittwood has joined #airshipit18:00
*** dr_feelgood has quit IRC18:02
*** arunkant has joined #airshipit18:19
*** kranthikirang has joined #airshipit18:35
openstackgerritEvgeniy L proposed airship/shipyard master: [WIP] Fix opensuse gate by installing python3  https://review.opendev.org/67868418:36
*** LoicL35 has quit IRC18:36
openstackgerritRoman Gorshunov proposed airship/drydock master: Fix: Drydock Exceptions docs rendering on RTD  https://review.opendev.org/67862418:38
*** kranthikirang has quit IRC18:40
*** mbeierl has left #airshipit18:46
*** michael-beaver has quit IRC19:04
openstackgerritAhmad Mahmoudi proposed airship/drydock master: [AICUNDRC-1985] - Allow Deploy Allocated Node  https://review.opendev.org/67894819:05
*** jamesgu_ has quit IRC19:08
*** jamesgu_ has joined #airshipit19:08
CobHeadmattmceuen: Did your deploy finish?19:14
openstackgerritEvgeniy L proposed airship/shipyard master: [WIP] Fix opensuse gate by installing python3  https://review.opendev.org/67868419:16
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (docs) Document manifest authoring  https://review.opendev.org/67567619:17
openstackgerritRoman Gorshunov proposed airship/drydock master: Fix: Drydock Exceptions docs rendering on RTD  https://review.opendev.org/67862419:22
openstackgerritEvgeniy L proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782319:22
CobHeadFATAL:  the database system is shutting down19:23
CobHeadWell. At least there's a different error now, I guess19:24
openstackgerritRoman Gorshunov proposed airship/armada master: Fix: Armada Exceptions docs rendering on RTD  https://review.opendev.org/67861819:26
openstackgerritRoman Gorshunov proposed airship/promenade master: Fix: Promenade Exceptions docs rendering on RTD  https://review.opendev.org/67862519:27
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (multinode) Make disk configuration configurable  https://review.opendev.org/67521319:28
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: [DNM] (multinode) Network configuration by JSON  https://review.opendev.org/67521419:28
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: [DNM] (multinode) Use network roles for selecting IPs  https://review.opendev.org/67521519:28
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (multinode) Update configs for new scripting  https://review.opendev.org/67521619:28
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (gating) Add shellcheck linter for multinode  https://review.opendev.org/67563619:28
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (docs) Document manifest authoring  https://review.opendev.org/67567619:28
openstackgerritAhmad Mahmoudi proposed airship/drydock master: Allow Deploy Allocated Node  https://review.opendev.org/67894819:52
*** mcriswel1 has quit IRC20:02
openstackgerritEvgeniy L proposed airship/shipyard master: [WIP] Fix opensuse gate by installing python3  https://review.opendev.org/67868420:02
*** dr_feelgood has joined #airshipit20:03
*** dr_feelgood has quit IRC20:08
*** dpawlik has joined #airshipit20:09
openstackgerritEvgeniy L proposed airship/shipyard master: [WIP] Fix opensuse gate by installing python3  https://review.opendev.org/67868420:12
openstackgerritRoman Gorshunov proposed airship/armada master: Fix: Armada Exceptions docs rendering on RTD  https://review.opendev.org/67861820:12
mattmceuenCobHead:  yep, mine completed successfully, all pods look happy!20:12
mattmceuenThat is indeed a different error -- did that come from the mariadb logs or elsewhere?20:13
*** dpawlik has quit IRC20:13
roman_gPlease, review:20:18
roman_ghttps://review.opendev.org/67862520:18
roman_ghttps://review.opendev.org/67862420:18
roman_ghttps://review.opendev.org/67861820:19
roman_gThis are read the docs renderring fixes and refactoring to use standardized zuul job for Promenade/Drydock/Armada20:19
roman_gWe will get Exceptions renderred properly for all 3 projects, and a few other bonuses.20:20
*** ian-pittwood has quit IRC20:21
roman_gCobHead: you were asking for Armada exceptions rendering ;)20:22
roman_g*renderring20:22
roman_gAll right. See you all tomorrow, need to go home. Local time is 22:23 :(20:23
*** kranthikirang has joined #airshipit20:23
*** souradage has quit IRC20:26
*** kranthikirang has quit IRC20:28
*** dr_feelgood has joined #airshipit20:37
*** dr_feelgood has quit IRC20:43
openstackgerritEvgeniy L proposed airship/shipyard master: Fix opensuse image build gate  https://review.opendev.org/67868420:54
evgenylHi everyone, can I please get reviews for Shipyard gate fix https://review.opendev.org/#/c/678684/ (opensuse image build), thanks!20:55
openstackgerritRoman Gorshunov proposed airship/drydock master: Fix: Drydock Exceptions docs rendering on RTD  https://review.opendev.org/67862420:55
openstackgerritRoman Gorshunov proposed airship/drydock master: Fix: Drydock Exceptions docs rendering on RTD  https://review.opendev.org/67862420:56
openstackgerritRoman Gorshunov proposed airship/promenade master: Fix: Promenade Exceptions docs rendering on RTD  https://review.opendev.org/67862520:57
openstackgerritEvgeniy L proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782320:57
openstackgerritKaspars Skels proposed airship/treasuremap master: Updates on site authoring and deployment guide  https://review.opendev.org/67861720:57
evgenylAlso here is a fix for Airskiff gate in Shipyard repo https://review.opendev.org/#/c/677823/20:58
openstackgerritRoman Gorshunov proposed airship/armada master: Fix: Armada Exceptions docs rendering on RTD  https://review.opendev.org/67861820:58
*** dddpak has joined #airshipit21:00
*** mixelplix has joined #airshipit21:01
openstackgerritTrung Thai proposed airship/porthole master: Initial commit of mysqlclient-utility container.  https://review.opendev.org/67488121:36
openstackgerritScott Hussey proposed airship/promenade master: [WIP] Make debug-report more fine grained  https://review.opendev.org/67896321:38
*** dpawlik has joined #airshipit22:10
*** kranthikirang has joined #airshipit22:12
*** dpawlik has quit IRC22:15
*** kranthikirang has quit IRC22:16
*** dddpak has quit IRC22:24
*** ab2434_ has quit IRC22:25
openstackgerritKaspars Skels proposed airship/treasuremap master: Updates on site authoring and deployment guide  https://review.opendev.org/67861722:31
*** mixelplix has quit IRC22:43
*** avolkov has quit IRC22:47
*** kkalina has quit IRC22:49
*** aaronsheffield has quit IRC23:03
*** kranthikirang has joined #airshipit23:12
*** kranthikirang has quit IRC23:14
*** kranthikirang has joined #airshipit23:16
*** kranthikirang has joined #airshipit23:16
openstackgerritKaspars Skels proposed airship/treasuremap master: Forceful deletion of Stateful set pods  https://review.opendev.org/67778923:25
*** kranthikirang has quit IRC23:26
*** kranthikirang has joined #airshipit23:26
*** kranthikirang has quit IRC23:31
*** kranthikirang has joined #airshipit23:36
*** kranthikirang has joined #airshipit23:37
*** kranthikirang has quit IRC23:41
*** AlexNoskov has quit IRC23:47
*** sthussey has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!