Description
Tech Stack Involved
Data Collection & Integration
- APIs: REST, GraphQL (to collect data from external sources)
- Data Connectors: AWS Glue, Talend, Stitch (for integrating multiple data sources)
- Data Streams: Apache Kafka, AWS Kinesis (for real-time data streams)
- ETL/ELT
- ETL Tools: Apache Airflow, dbt (data transformations in the cloud)
- Cloud ETL Services: AWS Glue, Azure Data Factory (for scalable ETL pipelines)
- Data Processing: AWS Lambda (for event-driven data processing)
Databases & Data Storage
- Relational Databases: PostgreSQL, MySQL (for structured data storage)
- Data Warehousing: Amazon Redshift, Snowflake (for centralized data storage and fast queries)
- NoSQL Databases: DynamoDB, MongoDB (for unstructured or semi-structured data)
- Cloud Storage: Amazon S3, Azure Blob Storage (for storing large datasets or flat files)
Data Analytics & Visualization
Business Intelligence (BI) Tools:
- Amazon QuickSight: Scalable cloud-native BI service
- Microsoft Power BI: Comprehensive analytics and interactive dashboards
- Tableau: Popular for creating highly visual dashboards
- Google Data Studio: Free and integrated with Google services for basic dashboards
- Data Querying: SQL, PostgreSQL (for querying data for dashboarding tools)
Data Preparation & Transformation
- Data Wrangling Tools: Pandas, PySpark (for handling complex data transformations before visualization)
- Data Cleansing: Trifacta, OpenRefine (for preparing clean datasets for dashboarding)
Cloud Infrastructure
- Cloud Compute: AWS EC2, Azure VMs (for hosting dashboards or running backend services)
- Containerization: Docker (for packaging and deploying dashboard applications)
- Serverless Options: AWS Lambda, Azure Functions (for lightweight, event-driven tasks)
Collaboration & Version Control
- Version Control: GitHub, GitLab (to track dashboard development)
- CI/CD: Jenkins, GitLab CI (to automate the deployment of dashboards)