Data Engineering
Qualification: National Certificate in IT (Systems Development)
Typical learning time: Full time: 9 - 12 months; Part time: 18 - 24 months
What does a Data Engineer do?
Data Engineering is the process by which companies gather and manage the large amounts of data they collect from their users and services. Data Engineers are the people responsible for gathering data from various sources and creating pipelines that transform the data to make it usable for others, such as Data Scientists, to generate business insights. Data Engineers are often the unsung heroes in “big data” applications.
Data Engineering is a specialist type of software engineering. One of their many sought-after skills is the ability to design, build, and maintain data warehouses, where raw data is transformed and stored in queryable forms.
On the Umuzi programme you will learn how to Extract data from various sources, Transform it so that it is clean and in the right format, and Load it into various databases and applications so that it can be useful.
Your learning journey with Umuzi:
Data Engineering
Step 1: Basic coding practices
Foundations of programming
Learn about object oriented programming and test driven development.
Tools of the trade & good practice
Basic code architecture and clean code, using Git and GitHub for code management and version control, incl. pull requests and code reviews.
Step 2: Intro to Data Engineering
FOUNDATIONAL
TOOLS OF DATA
ENGINEERING
Building blocks of data engineering
How does data work and how do we use it effectively?
Using Python for data engineering
Learn how to use Python for data engineering applications
Extract Transform & Load pipelines
Extract data from different sources and load into the data warehouse
DATABASES AND DATABASE MANIPULATION
Using relational databases
Structuring & accessing data in relational databases using PostgreSQL
Using non-relational databases
Structuring & accessing data using MongoDB
INTEGRATING
WEB
TECHNOLOGIES
Rest APIs
Learn RESTful API architectural style, which uses HTTP requests to access and use data
Working with JSON & XML
Learn how to use simple & clean data interchange formats to integrate applications
Step 3: Advanced Data engineering
Web framework and ORM
(i) SQL + Flask
Object relational mapping using SQLAlchemy and Flask web framework
OR
(ii) Django
Django web framework with integrated ORM
data engineering workflows and messaging
Advanced workflow management
Programmatically author, schedule and monitor data pipelines
Message-queueing & event streaming - RabbitMQ
Learn how to use the most widely deployed open source message broker.