Big Industries Academy
HowTo? Series: Connect Git between Laptop & CDP through Azure Devops
With this How To? Series we want to #ShareOurWisdom not only internally with our Big Industries colleagues but also with everyone who is as passionate as us about new technologies. Francine Anestis dedicated this article to the generation of a Git repository on Cloudera Data Platform (CDP) and on your local machine, as well as the connection between the two through Azure DevOps portal. In that way, you will be able to locally implement your code, and pull it from your project’s dedicated user and folder in a CDP cluster.
Create a Git Repository on Azure DevOps
Firstly, you need to create your git repository on Azure DevOps.
Connect a Git Repository to a CDP cluster
To connect to a git repository on Azure DevOps from within a CDP cluster the steps are the following;
Firstly, open a terminal from within the CDP cluster and dedicated user of your project and type the command: $ ssh-keygen to generate 2 key-pairs for an SSH connection. When a “passphrase” is requested, simply press “Enter”.
Visual representation of the aforementioned steps
Afterwards, copy the content of the id_rsa.pub (public key) and paste it in the field “Public Key Data”, under the respective git repository in Azure DevOps, and save it, by giving a suitable, descriptive name (User Settings --> SSH public keys --> + New Key).
Finally, clone the respective git repository:
$ git clone git@ssh.dev.azure.com:<git_repo_url>
Those steps will lead to the creation and connection of your project’s dedicated user from within a CDP cluster to your git repository on Azure DevOps.
For more detailed information, follow the link: Azure DevOps - Use SSH key authentication
Connect a Git Repository to your Local Machine
Follow the same steps as were described above to repeat the process to connect your local/personal machine to the git repository on DevOps.
The undermentioned link will help you to push code from your local machine to your git repository by using IntelliJ as IDE: Azure DevOps Plugin for IntelliJ IDEA and Android Studio
Please, read carefully mostly when you work with multiple people on the same git repo. It’s not fun to mess up with other people’s branches.
Francine Anestis
My diploma thesis as well as my internship being on ETL, Analysis and Forecasting of Big Streaming Data, I am keen on learning more and immersing myself in Data Engineering and Data Space in general. Building data pipelines, using Kafka, databases and algorithms captivated me during my studies as Electrical and Computer Engineer and as a result I decided to dedicate myself on Data Engineering. I am very excited starting my learning and career path at Big Industries. Regarding my skills, if I had to choose one programming language and a platform, I would say that Python and Kafka are my strongest assets, but I am looking forward to extending that list.