Data Engineering module (CS32001)

Data-driven decision making is vital for businesses and organisations today. With the exponential growth of data available to us, it has become crucial to have professionals who can design, build, and maintain data warehouses that can store and process large amounts of data efficiently.

This enables faster and more efficient querying and reporting of data from multiple sources and systems, as well as better data quality and consistency.

Data engineers are responsible for creating the infrastructure and pipelines that enable data to be collected, transformed, and loaded into data warehouses. This data can then be used by analysts and data scientists to extract insights and make informed decisions.

A strong understanding of data warehousing methodology will enable you to develop professional, data-driven solutions. Together with modern developments, such as continuous deployment and microservice architecture, this forms a highly sought-after skillset.

What you will learn

In this module, you will:

• learn about the concept of data warehousing and its processes
• learn how to create physical models for data warehouse systems using the Star schema
• study the types of slowly changing dimensions, which are collections of related facts
• examine the Extract-Transform-Load (ETL) process
• gain experience using NoSQL systems
• study the principles of continuous deployment and immutable services
• learn methods for developing continuous deployment solutions
• examine the role of the Cloud in continuous deployment
• explore technologies used in microservices systems

By the end of this module, you will be able to:

• compare and contrast online transaction processing and data warehousing
• demonstrate a strong understanding of data warehousing development and operation processes
• understand the importance of dimensional modelling and explain the principles of dimensional design
• explain the role of ETL in data warehouse construction
• compare and contrast technologies used in microservices systems development
• demonstrate your understanding of continuous deployment
• design, implement, and operate a data warehouse
• develop a NoSQL system
• implement services to develop a cloud-based ETL microservices system

Assignments / assessment

• data warehouse development (30%)
• data warehouse operations (20%)
• cloud-based microservice system development (50%)

This module does not have a final exam.

Teaching methods / timetable

You will learn by taking a hands-on approach. This will involve taking part in tutorials and practical sessions.

Learning material is provided through videos, review notes, examples, and tutorial questions.