From 8035f874b1ab08d8d589e8f70164d0feff695793 Mon Sep 17 00:00:00 2001 From: gleb-lobov <126697986+gleb-lobov@users.noreply.github.com> Date: Fri, 13 Dec 2024 15:33:05 +0100 Subject: [PATCH] Add data products for snowplow-cli (#1101) * Add snowplow-cli documentation for data products * Add data products cli instructions * Apply suggestions from code review Co-authored-by: Costas Kotsokalis <55377146+cksnp@users.noreply.github.com> * One more missed spec -> specification * Remove github annnotate on publish * Even more renames * Amend to new sidebar * Add the manage data products in CLI section * Fix typo --------- Co-authored-by: Costas Kotsokalis <55377146+cksnp@users.noreply.github.com> --- .../data-products/cli/index.md | 100 ++++ .../data-product-studio/data-quality/index.md | 2 +- .../data-structures/manage/cli/index.md | 71 +-- .../data-structures/version-amend/index.md | 2 +- .../snowplow-cli/images/orgID.png | Bin 0 -> 19006 bytes .../data-product-studio/snowplow-cli/index.md | 84 +++ .../snowplow-cli/reference/index.md | 542 ++++++++++++++++++ docs/data-product-studio/snowtype/index.md | 2 +- .../recipe-data-structures-in-git/index.md | 205 ++++++- 9 files changed, 933 insertions(+), 75 deletions(-) create mode 100644 docs/data-product-studio/data-products/cli/index.md create mode 100644 docs/data-product-studio/snowplow-cli/images/orgID.png create mode 100644 docs/data-product-studio/snowplow-cli/index.md create mode 100644 docs/data-product-studio/snowplow-cli/reference/index.md diff --git a/docs/data-product-studio/data-products/cli/index.md b/docs/data-product-studio/data-products/cli/index.md new file mode 100644 index 0000000000..15d0ba91ea --- /dev/null +++ b/docs/data-product-studio/data-products/cli/index.md @@ -0,0 +1,100 @@ +--- +title: "Managing data products via the CLI" +description: "Use the 'snowplow-cli data-products' command to manage your data products." +sidebar_label: "Using the CLI" +sidebar_position: 999 +--- +```mdx-code-block +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +``` +The `data-products` subcommand of [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) provides a collection of functionality to ease the integration of custom development and publishing workflows. +## Snowplow CLI Prerequisites +Installed and configured [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) +## Available commands +### Creating data product +```bash +./snowplow-cli dp generate --data-product my-data-product +``` +This command creates a minimal data product template in a new file `./data-products/my-data-product.yaml`. +### Creating source application +```bash +./snowplow-cli dp generate --source-app my-source-app +``` +This command creates a minimal source application template in a new file `./data-products/source-apps/my-source-app.yaml`. +### Creating event specification +To create an event specification, you need to modify the existing data-product file and add an event specification object. Here's a minimal example: +```yaml title="./data-products/test-cli.yaml" +apiVersion: v1 +resourceType: data-product +resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0 +data: + name: test-cli + sourceApplications: [] + eventSpecifications: + - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca + name: test + event: + source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0 +``` +:::caution Warning +The `source` fields of events and entities must refer to a deployed data structure. Referring to a locally created data structure is not yet supported. +::: +### Linking data product to a source application +To link a data product to a source application, provide a list of references to the source application files in the `data.sourceApplications` field. Here's an example: +```yaml title="./data-products/test-cli.yaml" +apiVersion: v1 +resourceType: data-product +resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0 +data: + name: test-cli + sourceApplications: + - $ref: ./source-apps/my-source-app.yaml +``` +### Modifying the event specifications source applications +By default event specifications inherit all the source applications of the data product. If you want to customise it, you can use the `excludedSourceApplications` in the event specification description to remove a given source application from an event specification. +```yaml title="./data-products/test-cli.yaml" +apiVersion: v1 +resourceType: data-product +resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0 +data: + name: test-cli + sourceApplications: + - $ref: ./source-apps/generic.yaml + - $ref: ./source-apps/specific.yaml + eventSpecifications: + - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca + name: All source apps + event: + source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0 + - resourceName: b9c994a0-03b2-479c-b1cf-7d25c3adc572 + name: Not quite everything + excludedSourceApplications: + - $ref: ./source-apps/specific.yaml + event: + source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0 +``` +In this example event specification `All source apps` is related to both `generic` and `specific` source apps, but event specification `Not quite everything` is related only to the `generic` source application. +### Downloading data products, event specifications and source apps +```bash +./snowplow-cli dp download +``` +This command retrieves all organization data products, event specifications, and source applications. By default, it creates a folder named `data-products` in your current working directory. You can specify a different folder name as an argument if needed. +The command creates the following structure: +- A main `data-products` folder containing your data product files +- A `source-apps` subfolder containing source application definitions +- Event specifications embedded within their related data product files. +### Validating data products, event specifications and source applications +```bash +./snowplow-cli dp validate +``` +This command scans all files under `./data-products` and validates them using the BDP console. It checks: +1. Whether each file is in a valid format (YAML/JSON) with correctly formatted fields +2. Whether all source application references in the data product files are valid +3. Whether event specification rules are compatible with their schemas +If validation fails, the command displays the errors in the console and exits with status code 1. +### Publishing data products, event specifications and source applications +```bash +./snowplow-cli dp publish +``` +This command locates all files under `./data-products`, validates them, and publishes them to the BDP console. diff --git a/docs/data-product-studio/data-quality/index.md b/docs/data-product-studio/data-quality/index.md index 482fbcaa52..eae4ce8954 100644 --- a/docs/data-product-studio/data-quality/index.md +++ b/docs/data-product-studio/data-quality/index.md @@ -1,7 +1,7 @@ --- title: "Data quality" date: "2024-12-04" -sidebar_position: 7 +sidebar_position: 8 --- There are a number of ways you can test and QA your pipeline to follow good data practices. diff --git a/docs/data-product-studio/data-structures/manage/cli/index.md b/docs/data-product-studio/data-structures/manage/cli/index.md index aa0b432f1e..c3a91de85c 100644 --- a/docs/data-product-studio/data-structures/manage/cli/index.md +++ b/docs/data-product-studio/data-structures/manage/cli/index.md @@ -13,78 +13,11 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ``` -The `data-structures` subcommand of [Snowplow CLI](https://github.com/snowplow-product/snowplow-cli) provides a collection of functionality to ease the integration of custom development and publishing workflows. +The `data-structures` subcommand of [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) provides a collection of functionality to ease the integration of custom development and publishing workflows. ## Snowplow CLI Prerequisites -### Install - -Snowplow CLI can be installed with [homebrew](https://brew.sh/) -``` -brew install snowplow-product/taps/snowplow-cli -``` - -Verify the installation with -``` -snowplow-cli --help -``` - -For systems where homebrew is not available binaries for multiple platforms can be found in [releases](https://github.com/snowplow-product/snowplow-cli/releases). - -Example installation for `linux_x86_64` using `curl` - -```bash -curl -L -o snowplow-cli https://github.com/snowplow-product/snowplow-cli/releases/latest/download/snowplow-cli_linux_x86_64 -chmod u+x snowplow-cli -``` - -Verify the installation with -``` -./snowplow-cli --help -``` - -### Configure - -You will need three values. - -API Key Id and API Key Secret are generated from the [credentials section](https://console.snowplowanalytics.com/credentials) in BDP Console. - -Organization Id can be retrieved from the URL immediately following the .com when visiting BDP console: - -![](images/orgID.png) - -Snowplow CLI can take its configuration from a variety of sources. More details are available from `./snowplow-cli data-structures --help`. Variations on these three examples should serve most cases. - - - - - ```bash - SNOWPLOW_CONSOLE_API_KEY_ID=********-****-****-****-************ - SNOWPLOW_CONSOLE_API_KEY=********-****-****-****-************ - SNOWPLOW_CONSOLE_ORG_ID=********-****-****-****-************ - ``` - - - - - ```yaml - console: - api-key-id: ********-****-****-****-************ - api-key: ********-****-****-****-************ - org-id: ********-****-****-****-************ - ``` - - - - - ```bash - ./snowplow-cli data-structures --api-key-id ********-****-****-****-************ --api-key ********-****-****-****-************ --org-id ********-****-****-****-************ - ``` - - - - -Snowplow CLI defaults to yaml format. It can be changed to json by either providing a `--output-format json` flag or setting the `output-format: json` config value. It will work for all commands where it matters, not only for `generate`. +Installed and configured [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) ## Available commands diff --git a/docs/data-product-studio/data-structures/version-amend/index.md b/docs/data-product-studio/data-structures/version-amend/index.md index d6d3206a3e..41735edeec 100644 --- a/docs/data-product-studio/data-structures/version-amend/index.md +++ b/docs/data-product-studio/data-structures/version-amend/index.md @@ -2,7 +2,7 @@ title: "Versioning data structures" date: "2020-02-25" sidebar_position: 2 -sidebar_label: "Verson and amend" +sidebar_label: "Version and amend" --- Snowplow is designed to make it easy for you to change your tracking design in a safe and backwards-compatible way as your organisational data needs evolve. diff --git a/docs/data-product-studio/snowplow-cli/images/orgID.png b/docs/data-product-studio/snowplow-cli/images/orgID.png new file mode 100644 index 0000000000000000000000000000000000000000..51a470f5b5f376fc589eb6e89b81a586abddb12a GIT binary patch literal 19006 zcmZ^K1z4QTk}iY<4G`QRf#8F?2M88C1a}4rI=H*Ly9Njl+}&LUcOBf_b>Nco@7~=# zdoT4cU(MI`ws&=PS51h5oCGQ|Auy1L&_{3w_}9qC z%w#RCHT(~2a6h_Wauqx4&EJQAe5hg^C`nTVwE3qAj$^kJF`OIkJGsbQ(|FTU)^Za_ zzQAtOZ29J3!o&Arf7EPj`3=)^0#hXx>K*x}YS(*y|HBRZ2R@`O!8a;cm|O1+S7EB& z#j;`g7&EuQR5`>(_e{miE4T@J_xH~9OMIFh=o#g~_wRo(2mN^_ev2k8Ug?2Y`9%5tJ?<(#elj(APi#IjvjqsK z7u^u>K`EubqhqzJqvBXW*g%nd88&=XEp#e8;-dwr8$A4M^UI6Zte&1;)KlR#j2N9Qt^Yz`_?>xQSxX~({g2L;7FKq=&I08B zlHh&i|Gs7-|M)KvdvgJDHCcs^q99wNj~tBbjLhVM$R9s``jgT zKd`?k|G@s$u78T-|C<@FvYnBwD9F;%$jV;uzdD!yUnTvI#Q&=2KR^XjXCn)>ucog^ zyVo`ea!4DM;#=e-R5 z2fS>adhRJkvuvQzF5WLsFSRD^W_{Oe2k33oU!hXlzsz^=^M&5m`G8bRyN<|B0rTY; z&+6nJ?y|YYP53iLjFhoTZ2cFCo|P}*g`2wx)MUkl2pD2WUtcNw88g`>^Zu(t$bTf0|g_mQMca4spbCF>&6Omz-WbeG6imD>uk8ytT8Ul2eoP%X@p1$I~W;L-Xf#CHN z@~@%77yYcqNNWR)JanSIPU`%`;!%d0JiMDaS*mU}R_X4+7E8;9p@K=at(p}Hd{I;n zuiGqmrrK2c*HCrr(fe346xEWeS5v0*_g^FGf+Qgk>daU);rum?gKW*=*TY8#a>ef#)IzbuhjF21&KP1cc=ugmjTto#FUu-HiQ4p=<)ow@aUnka*i&FG{p9hq-6@Ly z#{Y5Yl8L>7?Sy-?K@3-?W8|h-eVHOk@PyQDgO;nRpx~1{ezIwwZj;AzOEKT`*3@#G zyil=}P^#V=uJ-UK9^ST3z8#moc=eAs4&;ecOA=wQYQ4eUu)?AV2 zhlW4iwhK)?wP!reS#5++nbCmB+8YibkJn85u@%!B?RMrRO|g&dSu*>aOU4qnWGQn# zroGNpTLxiGHedV3klJsBu7dmUpy6zz^HwqM@O}x3QHPj1o1phVE;*AlA2#mtT)ZLT$Q) zxkk1$L~~a&5ib>Gnj21GO{zL>D9>fypI@8udePiD#9MJ%^(atm)jkYAN%pUvJ?&K0b;}^CdSpDY+IPp(vM>; z-DVDAt%R9${iS`?<3pS9 z0?k2y!fn|y=g%jnHF>_%wK|CVW}q$Cifho4(EudmJ0w~SMTT#L{Um-fh=a0~rY~7} zZb4zwtA0S_q(?u9FCO2|%kzn;eo)2BzP$B3UyOD%j&|t2u^{6!^m$jxlM_1)RG;g1 z5|9+zph7mkM(jE8900Y7P*$<^T6Vppn(UTHKadlIT5vu6>9)_EjL(L{05jJ5^PMMC zJn0JKx{5lkOpw?;Av!HC>?-H9byZ0>>j#k??XkgIxYj{DRc~kJbekbVwOCgR=~^Fw zn7|O&v=!Mphz^%xPr_jKP=k)ydGB(T(}JQY#B)wSB|+1=ZM7-jYdbeI{W7kh&2_H^ z;_IWAL{jI_9hkL+Z|8x^shf+=aw58&+LR*A^3|Z?r$gzXIRxPiwB)+^@fY;Q5o@5Q zWfkJqVA)Iluo*(-7{*X#-G0W`tX5c2=O2xOm+!Pe5Z0H2!tjR=nwjcwRBvEvz|y+w zgTSXK_8Wszfd;mAdW0VsyL+bJyw=2Fwcz5LlF58-g zSliWFoREL}O>2a3T31Q&CQ{N;8Y6`;1qAG-Td1QnxbU0UhD88WTF6{AGwQ+f5^-8v4C$WGC&9f#rE% zQKnn~^m(X+={l-JM1)nSexx*M86(Hj8;9FG!3P0P9%xAnZF71uP2?L-XEHRvsv zF^>o=V(i466tVvvzUSCOhIvUji+>=^F+`QnGsxqNgR%r31F z@nD`>>4#-R7R1VssBwVL>$GyK|K!$a$J8(&@UjsWxvMnB4IbfqEKilLCtw?yR%%2U z9B4fWMH!O=AW@UJ{nm5|*p5FS;nF&`9vlemE^>IhR&5k;T3yg~g`}JmBAnvV z&rw=cM?6PcU&YZDjIYcBZy_h7O^3K#M-_e~5}q)#q`1k0R2iQKg&s@VCz-LzVc}wc zQ3zx=WXT1?GfA1JnO;w@PS6Cr!^*hrfP!2!f$6sShR+r zCg@7DtW^XrOxO?JG?CnMuwzSfWlX@a{Sn-|%Q>Jd}P(~ zz$!t~0e@Jw?OZ;9vlg7ee2Q9(IHpE(+H|jRyXEIpcx{%s`lNAb4BZ<1o#2|?+)G8& z1RfQVHgY@EEysRmrIdp+1R?T36#D?#AEOaf=g*yv4x(^{gCNE-TzBXH*5E|S-tvZkVz@Ik@PYnRqC>7mcx22@Ysw~Hi%Eg^%#;~pSNr3HjLdT@^M1+hL{1U2pVZBQtCiTCjcQ&Gm?6y4^KSE3 zcjRR5=60gaq-0ZGl9o*03jr^w3kVEobdK%o_Prb_Mn-1Tp7tYm9Sy)c7cIarcm6MD3z31VvQ#V~|rs9r| zM3!?^c3Ztae24K`2z8cYi|VqBAeLCvrqf!1!99AmWPUgt_Ol&ROHX!G1wM7 zjK{JUY(C64U&*fO>?3)n7uUDkIu^Y`Y#O!V6hzeY0ttzX$*WgjlyCbD>vG9@P2xH3 znHr?jo>qQdU`H5p-py-u&yd0qSW8MX#npAcIP`KDCvRWo`KtV;WpQtHmO(o)Q^D-m z*A5pO;$4IncTlH4o{qlhj5$I4~SsPdAtLfl%@bYv|O+FxH$b#@XErgc0;^gwjGVVH8 z&AuysQ!`9X=Pm+n=yBb53JFfroe{a8_a?Dz_K`RdANSlzylhb`?N#<{#mN?Z3q_@i z|3H=3eA28-^$?_NDrB7{5YO3PLQGHzL%_T@pKa5l?~QraqcpZBhqtOaR8egllPQD! z7M@`UGYmNtUc}hOcq<#uWcR;_9M4`n= z7{4AI<|DjvSl${~nX2c!j)F0DI833ZQ2sTRgRolO9*HxS;=yL04bvxCrly2^zQ8{$tM7CFGtww1BIUU zvMxvqrcEFtLch@~B*&N>*2#h}GXu#}YQ8ghD!Luj#dXtJZ z5_Mi}g?2sD^}DVSvqBtI)kE5Snm>S|hHovsoOJTNVwx7(DEg*zrls(zu|n zdV~?uW_p~w%V@5;tKcqa>KOG}7RN#6?4-fYxBgQMh7D>6V{8@KrQ-~+u=lCRiXXHe z5M<5oDY9aoBd+FqvR`#RvIXx3l7h9AyZ9P#Xkrfj=Wm(qE z|M1u|wZj;n3*Fh?H&jCMx9wsg)9XgUu0_oqc zE+~AL$dDfo)_mt`>fRH0zlOI0OEbwOiN0&zrG9o?4_PHt4^)_)TVFSfVW6&cT7C2; z)KEjYCD`B+7BeTX8=~zFYFr(FCL)k}PU}m4v1&f8@fZ-9s@`W6`>PGQSdDj5ft=c4 zms2s$aoi zv`AcSpue*>@sY%22)E7M9EpSx@zYf?BhOfAb)VTu>Y>#SLK<~7_ljT3Hgm03U8(XH z9A2%RsJPo7-a%l*9vbm$<^M?Xa73tAo>OfTO6rhS$8ZV zi79KIJvfPm-q_MJwue6)s0$ETHkQgc0`S$2lhn}aeJttm188haEtqQ=P#-lLuuOA}* zdV9obNanGFpNxtax|lGhRbX?Y^f?ZS##3V3dbdyN-oxxvO*5<2v=KYsyPIB?G}ZMS za|UV}U=D|Le@C%$Wn;@w zqcEmf>`-Gjp7#@}aOHKeU<_oGorxgQt+tmFHx4T`&C4wXBP~0Z80kV&fej~=M=Yg>vR3po z`6)a6K3cgT{49L!ittRAi}NOZq!rdkGMiM54b%A*t`}wbDK#e^4#`6X{I{XAsW~nw zPEo>PBh3*eIL2v0Vh`}2t9Q)P+!kb*K3x(NjFiBo#{($MYkUSd%gfuJhRj`?u6^<4 zX~Xg!PBhEnz{hqpAvv|-aa?jooGZo%t?Z-lw;B@x!9uQgA$=l@jFSYkUtctIId8Ol z-*H2MXroiD_Of_*pWkOKqv*@_f3WWtxNWLc(N#7*>u#1{EYgza-2ct0|ERf7yzTxZ z_53?c;m=*iuPSiN5vvbv59Tu6vfT^ZqV>?H_l}s=)9xybrBOx?ZA(2IA_on!fHr#t z7+v;ff{M6IyBqbD;HlB7KRpgU+Kh?9h8^UQ2B_BQ505iYN-9po4qZ@-dZl1k6ipnL zE-$mQZpq?J08CqWhrh+~=q4WI4^g{jv&~qk19p&@Va%o))mt5M@7jl?_Orw{)$q*o zQ$rlhY>_$Q#7^-#xFQp+Ya8I&jJl9c-(Y3r4!S7Y5sbsHQ&7pxi$#ycl-&bG%5Ut= zR)C5YxDu$0f*O4cTpAgin}ok2@2hST7omwm@=FRR{MJficeHmN)>~k|5dt+GnFr1> zEn~OV_xGf>4~4eRg(LwPAUMo7Y7wUa+}PIf8Agd(B65p9>u`zJW=gE6N;=LgCFZ@^$>g-j0NeP?Du^}-oYYlO7GkZ+! z7oo_rfgk$HuCL2KXV{FYez1xi!T<8|n=ExZv4J8|K!-EFx{0KI<=YE}iMKt_kf;6Hq1d)Gf?%tU?bio)c5sV%NDT zVjqe)24pH*f@BjP7`}@#%CFwfLgaafz8Wmz+%ZmJ4*<0GY#gw;R}2R{5_A91&Mz8@ z7mJ?~`nl;fLJJF1en#8{L3p2ldZ%lYt7yq@e9gC)tuWPogLyyRUg4-?w;y0NqO4-3 zP^Q>|4+69#1v9;F5USdXSG6$#NgBduTysx2Md4@;txU@gPm z0Mke9pgr2qiMP9h*OV>=Ms}aJm-0YBpK^PPc5v1;Sjgglk>~ckfQx8IDS1$p4QiCS zH%B7NAbw5UptT?>c$ms6M@5%9vcn`H`JMevi_?y`#Xc{Qo)f!ZNJ}bh z`;hx_H)4iach)YmO&HR0pL+e!$#=+$b~)IFKvHCGsV0F#vbAsWvZIouq{aAi!k=u%|Pa%9w_- zBfjIsiF1|lVZ7D5v!B$#851;CAG)2Z)Dcq;+uwNXOY5$Z$&AA5EBM8+e2o9L<)o|v zrV2}Q{}$X7Q8zzYMeQ6DurmkS89D3i6a2kbZz~VhB@32~(s!jEmLy=b|Bd0p ztJ*Oe=kML~<|Wcc({p>x`ztKJhGw_u7*&gF8@NZ3eH!oelJc60OAXbMNusj)NUe@k zO^;@-6+MsoBRbFSFZT-f-LVD=vsPQZA9oyY5LmObzh+qv5fo_w3&r<^_G{Q8(l^fH zRH(>``&l)(<*6oN>E@bah@+|%x|cR>F(Njhu7wxnXvs#$IsThR% zRhN23XDw19c|hRRl=&MYo^s;z7ISaUxOZZw6Ood%T#{Op-jJK^Q}Yv?Z-C99TphoX zb1_0(fL3e;P1XG3AxoN>GWAQgN|X_Kc(({Jz4&=VFJOv5R8{(Srx?9cZQr|++|FFI zw6QOh#&}i10@uT<_KMUnFBXgYTo#x*p4-Z`_QL%gk=_jHeN187phxXMNw+r1*b2E^ ztbV_oR+|NeYf`=_AGcSwyp0e&|BS}je*3?D3q?ZpQH8;zrdbcC> ztA7k+lLh_9gsk3tvZ(58D)P5BrA;SmtTD49-ZCN411fVeX)+m8=wDvN+q%o8L2wW4 zX6p1SjYPZL)-H@5;$p;XOIh)50-sl3p3!mEM>f!mF45BncBqvEA(**1(bNc&Lwy`B zlO4v0z_9DIs(bsRN_g<{WUhI`MAfzN+1f)Y(E;mf{2uGSR!v0E2d|q2lSW{Id!5{( z9e4ZIKGB|p%O%GpHI(R=d^JWrYOVOJ=g3iuy3QyB7i)fZYiHE2A zC&_`OgV(+c;CT7sNP1oPsP>dpF{IJICQU=Uuz~w#oaatuua@ghn@|Gj<>?OSkmQY< zWIk_7N>g8*b-#2OA{LN~Ssk34-i?5#H{|V;+5hTUQmZyM^j*up?7@=%JnP5ffLVtH zQw;YGy`$ATge+OEq^BV}8J((1L`vVL$JYDh@x<-80GaqXo`zpd&8l^iks<3A=zvrE z=xKv0oU(;@9he4EZkK~#IKbS|233R~!Hk@aTQwF$*}32uu@-8aE}EAV`K6buR7e^) z7~X2WOa817t-##y3!d0xUbu*EaMp<&hj|F`5Jx*i z!RuBK^lrHwz{qR8Y&!uDNghxOv`O|wym6w${T6xGRVlY>7|x z4+1ipIWrtQ6lB3JVw*QKiydFFWIvDTWcP~&Qph~uyC+|V$r2O*_EEyV{aQ0Yt-Q%y zJ8gXF91Ka%SQKVM$FHWDYk`)(m_5QtOFcE4vJYzz=imEzSn?6<3C2c-+ zbjn(8Ua)lo5ty(A9O56deO%ok0{yX5_*qLOaLj99WKlMpmSh>%P?lboqXpwg9en%U zsBFCp+Y{RG@Jp?rJy?-yuBx%3{z?|!E(B$p!PQ@%%Rj3}`_rrg z+b^S$0)2u>2hSJ>ys*y|cR48Zp28=p^K+8yIa~c#iRAZ*BVf*crG&{~ZOt8^(r;dia!?ox9=*~cgim%|^SzO*!)j+~u zn*R{d)6_#2BSmks-<5$3^ntN%5PI|tn~wNIV)BheLXBAZ`M449M+Z%ontopln!~*C zBVC5lD7U5~_vkvfSCON?^YOT{na=Hz#I}L4DQt+WXwA7Fpmql@igFm+$uYO`yWe|P z+C!#ItxIvI5%KB?g=t?Err|C~>_U?VQ%wi^TO_ zjPDjS4N5A<`|SfJYA9I)qB7BBYc*`X=^HXHJ2g)Brei&-vvSg7pM)-WE~H2X50Z1eK^^W9wdE!|$~^?vHBG z_vFY#0L}5Pri;u~ms5dbw#xy%{)Y(L=AwpRY}kHD^U}IdD`z>ZVq@+KEh<`n2_`$?qj=%F~`uUqds?To0xn%P6E|cGmPuJ7HN>`cF2cL~Gfn!{6 zDq?o7mv~dmIo4!fy}?4qSTJ8`+^X&3th9#6iFSwjpY>DpDO68&<$ z_m%s>@&&4xLUJ-+Ael!pVTxMZ8B5=I_v+K19Ib-u*A@5OWuir<$#SuN*--QrxE~k3 z&98S0e-(gho?@d;nmh_YsqZE z6NDXrRt)xfq;RC&KkY1?Sg6TY-;GQWR9|4K=9@(K&)Uy^6ck8(Hx{ZR?nTBQCX)(k zInQ4*yBt}5)nPk2nC24Hdhs%7#BnV8gv^RLdTPt4T&wxbFm2*zy{fuJwi%IpH~w+8 z+$1EsPJ)_&99tsyhFGh*ZyN|aIv}D~C}jSM+_!vp%%x=X1lI=d`h5LCvVSD2MFX-B zjJnvkclmFL*0^8dsXG+1^9rxyYiBSI#RMNSnrt4RS*7g zSJyKe*$|^$B4j(T{i&|yz11D`Na$q(_}#9Md2w8nVC7p0qr|I=r<4KhT1E@7CVIp3 zMO$kpUKWn=gE1|!9SHfwc{&LW}4WjN?TwoDpd>t6&{ z`?PXx4ZI1{<(3$+KvXzrJi88PQKgsmge#G1iK3;-0#riO78~)jb98;y`|gztvSo*) zg^Y6RddZv;!9oZJZw6Vgo?rN+v&7}ZP8A7V{yFf)d4$5!sPWcfWq4Q-p{GGcZy%~(sb7Pg01Ymvu<}J znSLjg7AL6<3eBODmL!?Ykzy0PeuQs*yOv!Bk>_Ng23ZE@?;?1&b2TH%c}Np8Qr?w%GXI`_AKe zf;8N+CaW#E0Awu^I~(*T&6X1ZL(XroHWcmR8DZm+G3|tM`*%#O_sBWFi@N|Shs~##tQSw6 zLJAw3cX{!^n-2{%aVlo(Y0~d%ioa2#?w>C&IWAFBKIdmoIm!Sye_ae&fA;FuWyGM|HG%@M_ua<{B_QwoSkUnih@ttb z^!Jw#;A@o54_(jgkOwMkQsYz;ooJESyQ&(xZ+}kNJ>E;cL5r6#+<0i7^oUH8z)Kz%;K2SV<9?$GD-6lf*|Ukff2Qj z^1Z&?x-|9G1V5NBP2!oIFHMW{2#Qo1-p}85mkN)2c)R-YtIFXa!GzR-ZG=OYbnvqv zNT=@ol~XwRu%GGP*QH7JsddP3fT@Oy{^9_k3{iC-@^cT2>W=_7l6 zkoeKLcLv_@@^!HA_j@&j!*J)Z;h#?o`&K;^wd?_!aFhC(_NODVl$F1%ujvv_hH~tm ztq?)nlsHFp>04^(;_S6?M`Ppi67;uBp#?Nd{eL}I`47(-fW#4iw@QqGEJ2^}tlD@@ z`fc*{=wlO8x4f%6#vJmvoE+nYw-gU(y@FK+lGUzWvW7D_y@#!DsNO?uv=20L2|XD2 zmJ2-||1D&_F@aX`#|~5NEzOEcsFPx&+;3%)C^3h1$KLif?-rMZ17xO;n(6CuH6uT_ z|8}km`_#FZWlf8bOHsJ_Pa6YR|AHw(#mB)kX$JWPcZ>TA>r~GackqtUl=j5|%BD>= zt#3`+#L_*lwyL78JeX3uP6qv_lP7H)C>6ac-c;;%C`VXpF5GTZ(@J-Yeg*yf?%Ra} zN~8*p$O)>oPx!l`<Z8B+QtFf430MOf;uss+_y#Feidyu(X}R*vtf2jn&Uw(XmCgLU5xsA?pPe!PJWNh zr`-y!B^*N7851M^hE|rm-RPQDcM&&Je53dVO7FuRbY0%^qH2Drv;9(B8g=|P{_=m) z-mkYgwkyK=;@|&5{yV)aVFnLfzuY`Vl}7z<5d39Q*nLl&CuyEE&0HB*{$?z5LY(}o z9{=6@ulUuw${0eWLHu*L{@oa#o(Abj@uc(I<;Sd=7lUYQvVS5k;alD|nO#;KaM(a6 zZxeLlVwnpSUoKhc)Z6bx4|C+_aGkfPd?~aL_Fa2IFufJ_gW&}k6QeG|pc2$%1o*&* zyUI#3n|}gRlRx@2c)kXtGR>Y^{n$3#r9!Tuo6V8X44NuPb%^l!ja;j*TnD_T5d!%n zjAA^Z6HT!=KQSYG{iieVl|Fx%J8keR)>+br<`LyQiP~|2YXsd$IyH;T`^>5}sPB#J z|F%oYYn<_OYfli%O8y2nEiT(>C#xT01F4Q1n=ULvDx~>O z=1(6!GWxwdvZ0S=-vcdh8Y;aod1DRxVsRp5cEW!UM99yPCTjGi38W z-+ZP@tSo^un5A1l4?>P2iO?$@&(i574TyqOX)A1P5zDC=k!Er2H!BEgRkyDu{46rq zWc@AJ&yvv`9d}(7?QN6sKN4a`_`Y>P`R&BRP$K@#O38QZn%>Hc{hZ7(61zHida zs6siS`vhk_tIYy`Sn4P8rXo_~5JXJCa@E_#4}qd<;O{3!*^|Faop0+aS-6{=V-6ou zZBM(SZBj*)w)D=~)oTDL-sPr6WixqW&gs9K0Hbfz@;V=>e!*4VHnC;XicqxE9G~*8 zd875i1733^?dRsZIhe619Pm!$!m6#a1vX~7Cv%sSdxR~o1&&5bfLla_F5=}fvRVqA zj5=v|>GYHiB%04WZ~cbG(G?e+2e3J?W!wPpBKu9$EWX&4r^b4xZRFdVu8{lP*G=4` zcviJo39Lv3je4)29#CD;qdy@F5P7{SnmUa(Cby7L(cpkLr5Mtu_A!;ikxMGC*tOA$ zx_xxnkDi@Z1b^da0&)gNI7M}u|X1Hl$6?HT? zh9VlfM$DsQRmJ4oS#KCtIQubY?|tvWO~NQ9rVwou5NBh*nYqUji|`WXUr(@^tGPkP zl1pnbgx|hc6i-N4B*mgOQrO{JZL#w)Xj8qv)I#`^tzU=iLM%)zW1e6|w#}6o-Vz{f z!yr0&o7XPO+xGe+C0$`Fzpofno05t(e6W67+yr~rt&nqn7?}@&ZW1ioi{PjmV-ltU zyQHA%J)_#3N{(%wp;z~`?flMfjzcd`H~QC*3dpjy%(zPfY9)^HDqSPvE|*g-GAD3^ zT@;N_sd9@Iv{$GEocu{V1Y$bT228Y?g5Vq$<}r-z5I8GNfa`6&sKnQ%%Em?~@ou68 z`?a%o3RWcUK=Ik+$_hQNZ#~XfVtm2qNetC8nn9BmU739-BNg~)K+-BXAS*s1eHAib z>8FpWK*!^dpMuDZ4(e{ge)}C+Cg0}A@{vT+jg}UnFD;cu%9;%%CG1_k;?qDV1?%F% zl?D)Kz{@`ALH#_j*;u9sS&Wq{$1XeX>It4Xe9SC&*F|@!X-n@@@#)oUpuo@hLijlU z0K&+<)>4rPrM_-Ks1mn+q*bagoa_BP_*BSBEdzEo`$?&JbUH?& zcPpXK<(|F`gTllR08m?rn~z2+G<-mrq>YV~kA2m9vuNZ(+eBjY>nx0)_$eFuq$hdx zN0-&&34nF_zKU-=7bTIqJW$==;U0W6CIR|cSPqd}n!3ynF?-4)_gNEaLPpp5%K&XN zsUqhEjq*Xo6P$9B+{Ns*a4K==*Zbv_MhR*S+UZx66ng$f;p}Q&hW{ zs)HDTQjM2x$H+;PCcqnqr07sfPS(Vgu4BDi{ms#?cMX41eGQ~Afk_c5v<^}t`=Y>+ z20It0KsnlPdGD{30$=s0T(JNA5BpjO==<{K;VD-zgz|Io?2i~hWaF7-ksyl%`f$1n z-Nx^#GHNR^YV#Ev&%Ztlm7fxrHZ9|;fXW#DOYy(c`s_ zX=n#jAKhqJ;2>uV$Cb@>sW-%3b52$*5(0@t6RCS5c;F2O<~eo%>0y*UB$gg}l=^ix zq(ypzQ&WDYR3twl{b}Q#5o$3tJbjYhefvqP=1EqRRBzpG0Hj)BM(GaUHqBr;eWa7F z0k9swL8;bVODp9x9R;KZgaFM-Hy@M)dWAa8Q^lK~Zu8SuWt@vm&Fv4CtC@4DTr}E7 zGF1fR87;vE1b}6OUTu21=yA4?XtsZrSaLeML5=Z3}mye9lu)WYPLT`sdTHTwmT zV?9ayF|XNmAzk;=@K&McnE?d!GBb!7Vc`Y7nMX)q| zL2BCT79OM9tU#YOZk8lRJruyY!2^xr1<0QsR%7v%8zTHJ4aGKQgb}eY%Fa8FMM%G9 z(oU@VNznU&`pAZ%6Km4<2{D>q&znW&rEu{j{9c2|JmS1kU*m;++^lBtZ*!$zaY+<0@CPrst`r@L#*0}#!XA5#r`TZM&2`U@_rI(}~J z4Hd20@UHw+ZwYwBRw8&a*yVzA8z~&m>lIVoDu1cpdL63IIoDfC3o58Bi-al(q*Et7 zhumJo$%8}EBzZa2);M}Oc3(cmO1T! znV|R+(fj4NE@xDB=+6jD^PX*~(lQ;X+XV)Oh_v7cc#M%_SvJcHHZkf9(m+wOS(L+z z%kQ;!6cSJWj&*KSd}Kw1|2IXLsHe(-DDHD!}#S1do_VBm-}7WBfg?I_|FzI7-a| zEkXkB=ZSN3-1cJ)P*Lq1c6h>wig_Z6XMZjyuDiSHkL+uG>fdKa*`2Z2_Xrlf=`#Ki z!c|w29@y=h*twbLURFvIw?T6c$fBb)_NeQ@9NvAh@3#nb8AxN%#z~r1W=I+G=08$O z^Qy)NjV9o1GBKlGT$oRod`O`XvNIXjpL<66z#@@7>IV{gu4GNcDF==zpwYWN4h$t)r4pSQfQlcgO8 ziZ$Zid?qx%Q87 z#?u9JNd_^r0q5zMjYYa5j<1IU_Eq{W)VWcDwin&%G>SHE%nFl+%-%U*>7q`i8z)q0 zNLKFx(v|CyY)o%7P&Kbr7zrw*{I{dW5Pqg@Q5Rcop>LKZ_iR=ddb#HoFK~2L8R-M+ zT#?Y5;BS`c>Mlrn_5EaFTyTC9jDqKgS96oVM^Pa5_T(MR7UX5Aig}e64QUng5@pgL zo+>lSvdWCJ3F2Z!=ndO9#<`1r=_}j#QW~S$LtjI$_dSd8$^(;=+!RdegzpwTx&7RE zi~jI}!Gay!bKEFzlVw9z|JV!;-wN52bX1@(l*?bKsP?sBR9}72OkHQT{3XBh7g z3%aI4vJWR8`*i6q*cu5q36J$2NOTWlWUZQVvHdMY2k_-VMG+|UR_o#=pf5QG$91!x zxuJSk-!OG^lhHKAa6#*W9c>ZUr6Qg<%t}eYAP<2fFhRFzx_qq??I)+cgmuS5$@;FY zh^rD@YYczlzHSKFV*MYg`&Uk^e zWpg#Ab^YtQL6!uo8m#xFobJvs#$wSs#`cS z2P`Rz*O<}A^tr=uZMGG@=Fya<+q$6bD={n!oV#=+_AOw<03NPgya& z;lN~*M+tRl`kW~)Qx!X0PBc-AWsO%0-I}Lg!Bp8(okQ5f`%5YgO45c#;P2BYR&XtQ ziHtab~%_A5Cs}e_@ zRy^<9Rzf}#MW&Mv;NRy7wI+>{+Gee!n(>G^A(62)LFCMZ$Y(UA;CUo#(RUN=7zRdO%eCk&D9oy#_ZN z+wOj^v(yxXTF5x2u23e-lwbY{OS=AJW|2?*%=b5Z>7v{Mu8zV#}{@8IU5+Wf{^YeLn$xn!m=(X8&@8mAr^g_V`h zvC)+#3$I(f3ZcF*#evQ1d4KTL3CXJjG)VX$IBh>3BhXuh+z?%nd?>~rXIngvUuBDX zqDqWBh0n6gO95b{YyY|r&X2j>Z#`Ph2)kCIi;K=uBedE4G2N(9tR&)Tt$1U~!#Gie zzaTdLL|6p@d7i)~#rC#2$akWKmHR`JOj;(>4U4x?mQ-G@bTZ3px6ueH8d&gvBe%LI zw23ITDq;FKVpXU%0pH~zx#LKiOCO1_k!aOXg>9Ro@KKXBx4sr4G6VMD`{V?|SkVV+ z+G4U^E)Xv0+tvpAk{twB5fx>B%9chW9uZ2l$S_Z5yPNtN!AO%xi5nHLSOC-nQEqAT zZwn2P&Cd@>)%6(t=}`Yq0AvT7_zLq6tmh)`?KK$A=iI2|=cvEvbRH`U9peKSuC7|eV#DIN;P6X{V+Vp8)Bc&$NJ~~{zZOl+`P*|e*eSr+M|Ckro6p=BA_K1 zEwkd*Pz$$%jYe5_vwSsijiQ=wMHe9t}J9W zPY9XOUjP6F@JU2LR8nCc%3C;1tM_3j!-au`v#{kU!^UkHE~C?!p5};DtD!MxetbG# zKZK2KtIEQ;SQ3%}C2vLtmdD);IQ_2*CQs@gZ<;^o;VoO9Bgvv*ST8Yh99oOd#nvFp zXW@RuRyjE9AIw;u&M=16)mvb*>5*1$arI@>oLM*v6X)N);pP~qlE-639d5m+*_+?i z2#X`8Bpbn_}%fN>8vrCayW|_?{*&nf?Y{EBX&+uS*I>1kwl!p0X^%t=88un|j zX$F!U#>=yYYgp%v)U%Qd%SZ*DhSvqCP?dTrZXn*w3$z9%utCWKpXS#<1Y!TkUlUA% z^JsJ~-fA?87S6)NrT!K&p{eiV zbDcEMsI`XabUHbT-zNTFrUY7DU5?{;aR*VWwJ=ilX)~+d8e`#3;IU=5{auZRq%YWu z&%$|?CWZ4dv=q;uRmQHBr9>0Q7gyYh4}tr+V0O4h*adulNDx z$q#?x`r>S@$l_hnNVd=JkwwF8V#>D&R-F|yb}T*&D{o{Z63D?#hND_O%kHhOEFv7e zH(LL_)tXUW^i>R_l*XQ|(d(Bs*a*twys@1~hZyg_*?2r}$MH7LhSaSV!+Eu(;~Q(w zeeLODsc;r2sVqq_U1O5ap$8-&fa^laI$c+~Yk7RmAMiIzZe?%Kb%i6>c#u7_~B7R$Ezmmh`DpUuUhb59e{xN)2UstEVUOYbt`fL*s_8DNDabtjJvW_I0JnWN?s*YAoSrf3o=ardCM=b*cW~A1uNb zX(~`7MZ5u|!?4BoA@)_$I@=su`>e%2+{{9i1~r$e8e_Fq#Z~F%8WZXAbk zS+!bv7S77w%Q$1JU2N4``J|mmgyq0;-3TbwSW*WS91}N|=v}Bs6l?127Z@bU!ldK4 zL1W8TH{9o6TeIC##X`EzTgH zjtaR?S7&+I2Q+YOfgJ1=>H4SnTR}qn`(#tMeRGZamIGaLqY+Cvjww#UOA`YxX8aGl%r?wwvQ&)EZsayTq zr`Y!;Talm1&2R^}t5xs&NFYCnXJbcSR>iuO&o>4p2;bcpOq@2 sVrgk?W&07sg0WAv7B0n-A+?$Oe{DakNnC#k@Bjb+07*qoM6N<$f<+^OoB#j- literal 0 HcmV?d00001 diff --git a/docs/data-product-studio/snowplow-cli/index.md b/docs/data-product-studio/snowplow-cli/index.md new file mode 100644 index 0000000000..30eda477d9 --- /dev/null +++ b/docs/data-product-studio/snowplow-cli/index.md @@ -0,0 +1,84 @@ +--- +title: Snowplow CLI +sidebar_label: Snowplow CLI +sidebar_position: 7 +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +`snowplow-cli` brings data management elements of Snowplow Console into the command line. It allows you to download your data structures and data products to yaml/json files and publish them back to console. This enables git-ops-like workflows, with reviews and brancing. + +# Install + +Snowplow CLI can be installed with [homebrew](https://brew.sh/): +``` +brew install snowplow-product/taps/snowplow-cli +``` + +Verify the installation with +``` +snowplow-cli --help +``` + +For systems where homebrew is not available binaries for multiple platforms can be found in [releases](https://github.com/snowplow-product/snowplow-cli/releases). + +Example installation for `linux_x86_64` using `curl` + +```bash +curl -L -o snowplow-cli https://github.com/snowplow-product/snowplow-cli/releases/latest/download/snowplow-cli_linux_x86_64 +chmod u+x snowplow-cli +``` + +Verify the installation with +``` +./snowplow-cli --help +``` + +# Configure + +You will need three values. + +An API Key Id and the corresponding API Key (secret), which are generated from the [credentials section](https://console.snowplowanalytics.com/credentials) in BDP Console. + +The organization ID, which can be retrieved from the URL immediately following the .com when visiting BDP console: + +![](./images/orgID.png) + +Snowplow CLI can take its configuration from a variety of sources. More details are available from `./snowplow-cli data-structures --help`. Variations on these three examples should serve most cases. + + + + + ```bash + SNOWPLOW_CONSOLE_API_KEY_ID=********-****-****-****-************ + SNOWPLOW_CONSOLE_API_KEY=********-****-****-****-************ + SNOWPLOW_CONSOLE_ORG_ID=********-****-****-****-************ + ``` + + + + + ```yaml + console: + api-key-id: ********-****-****-****-************ + api-key: ********-****-****-****-************ + org-id: ********-****-****-****-************ + ``` + + + + + ```bash + ./snowplow-cli data-structures --api-key-id ********-****-****-****-************ --api-key ********-****-****-****-************ --org-id ********-****-****-****-************ + ``` + + + + +Snowplow CLI defaults to yaml format. It can be changed to json by either providing a `--output-format json` flag or setting the `output-format: json` config value. It will work for all commands where it matters, not only for `generate`. + + +# Use cases + +- [Manage your data structures with snowplow-cli](/docs/data-product-studio/data-structures/manage/cli/index.md) +- [Set up a github CI/CD pipeline to manage data structures and data products](/docs/resources/recipes-tutorials/recipe-data-structures-in-git/index.md) diff --git a/docs/data-product-studio/snowplow-cli/reference/index.md b/docs/data-product-studio/snowplow-cli/reference/index.md new file mode 100644 index 0000000000..7b912230af --- /dev/null +++ b/docs/data-product-studio/snowplow-cli/reference/index.md @@ -0,0 +1,542 @@ +--- +title: Command Reference +date: 2024-11-26 +sidebar_label: Command Reference +sidebar_position: 1 +--- + +This page contains the complete reference for the Snowplow CLI commands. + +## Data-Products + + +Work with Snowplow data products + +### Examples + +``` + $ snowplow-cli data-products validate +``` + +### Options + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + -h, --help help for data-products + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id +``` + +### Options inherited from parent commands + +``` + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + --json-output Log output as json + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Products Download + + +Download all data products, event specs and source apps from BDP Console + +### Synopsis + +Downloads the latest versions of all data products, event specs and source apps from BDP Console. + +If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory + +``` +snowplow-cli data-products download {directory ./data-products} [flags] +``` + +### Examples + +``` + $ snowplow-cli dp download + $ snowplow-cli dp download ./my-data-products +``` + +### Options + +``` + -h, --help help for download + -f, --output-format string Format of the files to read/write. json or yaml are supported (default "yaml") +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Products Publish + + +Publish all data products, event specs and source apps to BDP Console + +### Synopsis + +Publish the local version versions of all data products, event specs and source apps from BDP Console. + +If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory + +``` +snowplow-cli data-products publish {directory ./data-products} [flags] +``` + +### Examples + +``` + $ snowplow-cli dp publish + $ snowplow-cli dp download ./my-data-products +``` + +### Options + +``` + -h, --help help for publish +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Products Validate + + +Validate data structures with BDP Console + +### Synopsis + +Sends all data products and source applications from \ for validation by BDP Console. + +``` +snowplow-cli data-products validate [paths...] [flags] +``` + +### Examples + +``` + $ snowplow-cli dp validate ./data-products ./source-applications + $ snowplow-cli dp validate ./src +``` + +### Options + +``` + --gh-annotate Output suitable for github workflow annotation (ignores -s) + -h, --help help for validate +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures + + +Work with Snowplow data structures + +### Examples + +``` + $ snowplow-cli data-structures generate my_new_data_structure + $ snowplow-cli ds validate + $ snowplow-cli ds publish dev +``` + +### Options + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + -h, --help help for data-structures + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id +``` + +### Options inherited from parent commands + +``` + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + --json-output Log output as json + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures Download + + +Download all data structures from BDP Console + +### Synopsis + +Downloads the latest versions of all data structures from BDP Console. + +Will retrieve schema contents from your development environment. +If no directory is provided then defaults to 'data-structures' in the current directory. + +``` +snowplow-cli data-structures download {directory ./data-structures} [flags] +``` + +### Examples + +``` + $ snowplow-cli ds download + $ snowplow-cli ds download --output-format json ./my-data-structures +``` + +### Options + +``` + -h, --help help for download + -f, --output-format string Format of the files to read/write. json or yaml are supported (default "yaml") +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures Generate + + +Generate a new data structure locally + +### Synopsis + +Will write a new data structure to file based on the arguments provided. + +Example: + $ snowplow-cli ds gen login_click --vendor com.example + Will result in a new data structure getting written to './data-structures/com.example/login_click.yaml' + The directory 'com.example' will be created automatically. + + $ snowplow-cli ds gen login_click + Will result in a new data structure getting written to './data-structures/login_click.yaml' with + an empty vendor field. Note that vendor is a required field and will cause a validation error if not completed. + +``` +snowplow-cli data-structures generate login_click {directory ./data-structures} [flags] +``` + +### Examples + +``` + $ snowplow-cli ds generate my-ds + $ snowplow-cli ds generate my-ds ./my-data-structures +``` + +### Options + +``` + --entity Generate data structure as an entity + --event Generate data structure as an event (default true) + -h, --help help for generate + --output-format string Format for the file (yaml|json) (default "yaml") + --vendor string A vendor for the data structure. + Must conform to the regex pattern [a-zA-Z0-9-_.]+ +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures Publish + + +Publishing commands for data structures + +### Synopsis + +Publishing commands for data structures + +Publish local data structures to BDP console. + + +### Options + +``` + -h, --help help for publish +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures Publish Dev + + +Publish data structures to your development environment + +### Synopsis + +Publish modified data structures to BDP Console and your development environment + +The 'meta' section of a data structure is not versioned within BDP Console. +Changes to it will be published by this command. + + +``` +snowplow-cli data-structures publish dev [paths...] default: [./data-structures] [flags] +``` + +### Examples + +``` + $ snowplow-cli ds publish dev + $ snowplow-cli ds publish dev --dry-run + $ snowplow-cli ds publish dev --dry-run ./my-data-structures ./my-other-data-structures +``` + +### Options + +``` + -d, --dry-run Only print planned changes without performing them + --gh-annotate Output suitable for github workflow annotation (ignores -s) + -h, --help help for dev +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures Publish Prod + + +Publish data structures to your production environment + +### Synopsis + +Publish data structures from your development to your production environment + +Data structures found on \ which are deployed to your development +environment will be published to your production environment. + + +``` +snowplow-cli data-structures publish prod [paths...] default: [./data-structures] [flags] +``` + +### Examples + +``` + + $ snowplow-cli ds publish prod + $ snowplow-cli ds publish prod --dry-run + $ snowplow-cli ds publish prod --dry-run ./my-data-structures ./my-other-data-structures + +``` + +### Options + +``` + -d, --dry-run Only print planned changes without performing them + -h, --help help for prod +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + +## Data-Structures Validate + + +Validate data structures with BDP Console + +### Synopsis + +Sends all data structures from \ for validation by BDP Console. + +``` +snowplow-cli data-structures validate [paths...] default: [./data-structures] [flags] +``` + +### Examples + +``` + $ snowplow-cli ds validate + $ snowplow-cli ds validate ./my-data-structures ./my-other-data-structures +``` + +### Options + +``` + --gh-annotate Output suitable for github workflow annotation (ignores -s) + -h, --help help for validate +``` + +### Options inherited from parent commands + +``` + -S, --api-key string BDP console api key + -a, --api-key-id string BDP console api key id + --config string Config file. Defaults to $HOME/.config/snowplow/snowplow.yml + Then on: + Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml + Darwin $HOME/Library/Application Support/snowplow/snowplow.yml + Windows %AppData%\snowplow\snowplow.yml + --debug Log output level to Debug + -H, --host string BDP console host (default "https://console.snowplowanalytics.com") + --json-output Log output as json + -m, --managed-from string Link to a github repo where the data structure is managed + -o, --org-id string Your organization id + -q, --quiet Log output level to Warn + -s, --silent Disable output +``` + + + diff --git a/docs/data-product-studio/snowtype/index.md b/docs/data-product-studio/snowtype/index.md index bc76f956e7..8bd820315a 100644 --- a/docs/data-product-studio/snowtype/index.md +++ b/docs/data-product-studio/snowtype/index.md @@ -1,7 +1,7 @@ --- title: "Code Generation - automatically generate code for Snowplow tracking SDKs" sidebar_position: 6 -sidebar_label: "Snowtype" +sidebar_label: "Code generation (Snowtype)" sidebar_custom_props: offerings: - bdp diff --git a/docs/resources/recipes-tutorials/recipe-data-structures-in-git/index.md b/docs/resources/recipes-tutorials/recipe-data-structures-in-git/index.md index 732d828966..57415e0a78 100644 --- a/docs/resources/recipes-tutorials/recipe-data-structures-in-git/index.md +++ b/docs/resources/recipes-tutorials/recipe-data-structures-in-git/index.md @@ -16,19 +16,19 @@ The Snowplow Console's UI offers excellent facilities to get started quickly wit A common solution when faced with these requirements is to move management to some form of version control platform (github/gitlab). This opens up an entire ecosystem of tools and patterns enabling all manner of custom workflows. -We have built [snowplow-cli](/docs/data-product-studio/data-structures/manage/cli/index.md) to help you bridge the gap between these repository based workflows and BDP Console. +We have built [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) to help you bridge the gap between these repository-based workflows and BDP Console. ## Prerequisites * A deployed Snowplow BDP pipeline -* [snowplow-cli](/docs/data-product-studio/data-structures/manage/cli/index.md#download) downloaded and configured +* [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md) installed and configured * A familiarity with [git](https://git-scm.com/) and an understanding of [github actions](https://docs.github.com/en/actions/writing-workflows) * A sensible [terminal emulator](https://en.wikipedia.org/wiki/Terminal_emulator) and shell ## What you'll be doing -This recipe will walk through creating and deploying a data structure from the command line using [snowplow-cli](https://github.com/snowplow-product/snowplow-cli). It will then show how it is possible to automate the validation and deployment process using [github actions](https://docs.github.com/en/actions/writing-workflows). +This recipe will walk through creating and deploying a data structure from the command line using [Snowplow CLI](/docs/data-product-studio/snowplow-cli/index.md). It will then show how it is possible to automate the validation and deployment process using [github actions](https://docs.github.com/en/actions/writing-workflows). ## Create a local data structure @@ -360,7 +360,206 @@ Validation has passed. Now our colleagues can feedback on our changes and if eve Finally, once we are convinced everything works we can open another pull request from `develop` to `main`, merge that and trigger our `publish-production.yml` workflow. +## Following up with data products + +Now that we have our data structures set up, we can define data products to organize and document how these structures are used across our applications. We'll walk through creating source applications, data products, and event specifications using the CLI, then integrate them into our automated workflows. + +### Create a source applications + +First, we'll create a source application to represent our website that will send the `login` event we defined earlier. + +```bash +snowplow-cli dp generate --source-app website +``` +:::note +`dp` is an alias for `data-products`. Source applications and event specifications are also managed by this command +::: + +This should provide the following output +``` +INFO generate wrote kind="source app" file=data-products/source-apps/website.yaml +``` + +The generated file is written to the default `data-products/source-apps` directory. Help for all the arguments available to `generate` is available by running `snowplow-cli dp generate --help`. + +Let's examine the generated file: + +```yml title="data-products/source-apps/website.yaml" +apiVersion: v1 +resourceType: source-application +resourceName: b8261a25-ee81-4c6a-a94c-7717ba835035 +data: + name: website + appIds: [] + entities: + tracked: [] + enriched: [] +``` + +* `apiVersion` should always be `v1` +* `resourceType` should remain `source-application` +* `resourceName` is a unique identifier of the source applications. It must be a valid uuid v4 +* `data` is the contents of the source app + +Now let's customize our source application. We'll configure it to handle events from our production website as well as staging and UAT environments. We'll also add an owner field and remove the unused entities section. + +```yml {6-7} title="data-products/source-apps/website.yaml" +apiVersion: v1 +resourceType: source-application +resourceName: b8261a25-ee81-4c6a-a94c-7717ba835035 +data: + name: website + appIds: ["website", "website-stage", "website-ua"] + owner: me@example.com +``` + +Before publishing, we can validate our changes and preview what will happen: + +```bash +snowplow-cli dp publish --dry-run +``` + +The command will show us the planned changes: +``` +publish will create source apps file=.../data-products/source-apps/website.yaml name=website resource name=b8261a25-ee81-4c6a-a94c-7717ba835035 +``` + +When we're happy with the proposed changes, we can publish by removing the `--dry-run` flag: + +```bash +snowplow-cli dp publish +``` + +After publishing, you'll be able to see your new source application in the BDP Console UI. + +### Create a data product and an event specification + +Let's now create a data product and an event specification by running the following command + +```bash +snowplow-cli dp generate --data-product Login +``` +This should provide the following output +``` +INFO generate wrote kind="data product" file=data-products/login.yaml +``` +Let's see what it has created for us + +```yml title="data-products/login.yaml" +apiVersion: v1 +resourceType: data-product +resourceName: 0edb4b95-3308-40c4-b266-eae2910d5d2a +data: + name: Login + sourceApplications: [] + eventSpecifications: [] +``` + +Let's amend it to add an event specification, and a reference to a source application: + +```yml {6,7,9,11-14} title="data-products/login.yaml" +apiVersion: v1 +resourceType: data-product +resourceName: 0edb4b95-3308-40c4-b266-eae2910d5d2a +data: + name: Login + owner: me@example.com + description: Login page + sourceApplications: + - $ref: ./source-apps/website.yaml + eventSpecifications: + - resourceName: cfb3a227-0482-4ea9-8b0d-f5a569e5d103 + name: Login success + event: + source: iglu:com.example/login/jsonschema/1-0-1 +``` + +:::note +You'll need to come up with a valid uuid V4 for the `resourceName` of an event specification. You can do so by using an (online generator)[https://www.uuidgenerator.net], or running the `uuidgen` command in your terminal +::: + +:::caution Warning + +The `iglu:com.example/login/jsonschema/1-0-1` data structure has to be deployed at least to a develop envinroment. Currently referencing local data structures is not supported + +::: + +We can run the same `publish --dry-run` command as before, to see if the output is as expected. The output should contain the following lines + +```bash +snowplow-cli dp publish --dry-run +``` + +``` +INFO publish will create data product file=.../data-products/login.yaml name=Login resource name=0edb4b95-3308-40c4-b266-eae2910d5d2a +INFO publish will update event specifications file=.../data-products/login.yaml name="Login success" resource name=cfb3a227-0482-4ea9-8b0d-f5a569e5d103 in data product=0edb4b95-3308-40c4-b266-eae29 +``` + +We can apply the changes by using the publish command without the `--dry-run` flag + +```bash +snowplow-cli dp publish +``` + +### Add data products validation and publishing in the github actions + +Now that we've modeled a source application, data product and event specification, let's see how we can add them to the existing github actions workflows for data structures. You can customize your setup, use a separate repository or a separate actions, but in this example we'll add the data products publishing into the existing workflows. + +Lets modify the PR example, and add the following line. This command will validate and print the changes to the github actions log. + +```yml {20} title=".github/workflows/validate-pull-request.yml" +on: + pull_request: + branches: [develop, main] + +jobs: + validate: + runs-on: ubuntu-latest + env: + SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }} + SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }} + SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }} + + steps: + - uses: actions/checkout@v4 + + - uses: snowplow-product/setup-snowplow-cli@v1 + + - run: snowplow-cli ds validate --gh-annotate + + - run: snowplow-cli dp publish --dry-run --gh-annotate +``` + +Data products, source applications and event specifications don't have the dev and prod environments, so it's enough to publish them once. +We can add the same command but without the `--dry-run` flag to the publish pipeline. + +```yml {20} title=".github/workflows/publish-develop.yml" +on: + push: + branches: [develop] + +jobs: + publish: + runs-on: ubuntu-latest + env: + SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }} + SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }} + SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }} + + steps: + - uses: actions/checkout@v4 + + - uses: snowplow-product/setup-snowplow-cli@v1 + + - run: snowplow-cli ds publish dev --managed-from $GITHUB_REPOSITORY + + - run: snowplow-cli dp publish +``` + +You might want to publish data products in the `.github/workflows/publish-production.yml` as well, or only there. It depends on your setup, but if you strictly follow the rules and always merge to `main` from `develop`, the setup above should be enough. + ## Let's break down what we've done * We have seen how snowplow-cli can be used to work with data structures from the command line * We have applied that knowledge to build github workflows which support automated validation and publication +* We have added source applications, data products and event specifications to use the same approach as data structures.