Diego B.Data Engineer
Skills
An accomplished Data Engineering professional specializing in the execution of PySpark routines and management of Databricks environments. Proficient in leveraging critical technologies such as Azure Databricks, Azure Synapse, Azure Data Lake, and Azure Data Factory to develop optimized cloud-based solutions. Adept at designing ETL pipelines, writing data extraction scripts, and implementing business rules. Demonstrates a strong capability in solving complex problems through the integration of effective strategies and advanced technologies.
Data Engineer / Python Developer
11/1/2020 - Present
Developed proficiency in Python and SQL through multiple data migration projects, each involving complex data handling and advanced ETL processes. Demonstrated expertise with Azure Databricks, Azure Data Lake, Azure Synapse, Azure SQL Server, Azure Web App, and Azure DevOps, primarily using Azure technologies for sophisticated Asset Performance Management projects. Leveraged Databricks with PySpark for data processing and statistical modeling, ensuring optimal storage and insightful analytics.
In the IoT Hub 4.0 Project, implemented data migration from IoT devices to Azure cloud services, utilizing Azure IoT Hub (IoTEdge), Azure Functions, Stream Analytics, Log Analytics, and Azure Monitor to establish a comprehensive data pipeline. Built CI/CD pipelines using Azure DevOps, while employing Python for data ETL tasks and API communications between backend and frontend systems.
Executed a Data Factory pipeline project, collecting data through API calls, and using Databricks notebooks for data transformation and recording in SQL servers. Applied thorough knowledge in using Data Factory, Data Lake, and Databricks tools to maintain efficient data workflows, utilizing both SQL and Python for seamless data processes.
Conducted a web scraping project on Google Cloud Platform, gathering data from web sources and storing it in Cloud Storage. Developed fact and dimension tables using BigQuery for subsequent creation of interactive dashboards in Data Studio. Mastery of Compute Engine and Cloud Storage exhibited throughout the project, alongside adept use of SQL and Python.
Handled data migration with Apache NiFi, moving data from Azure Data Lake to MySQL databases, followed by creation of fact and dimension tables. These tables were subsequently used in Power BI to generate comprehensive analysis dashboards, showcasing the effective use of Apache NiFi, MySQL, and Power BI with a solid grounding in SQL to facilitate robust data analytics.
Analysis and Development of Systems at Paulista University
2022 - 2024
Programming with Databricks. at Databricks Academy
3/1/2021Cloud Architecture and System Integration Fundamentals. at Databricks Academy
3/1/2021Azure Databricks Cluster Usage Management at Databricks Academy
3/1/2021Foundations for Data Analysis. at Data Science Academy
12/1/2020HTML5 and CSS3 Part 4: Advancing in CSS at Alura Cursos Online
12/2/2019HTML5 and CSS3 Part 3: Working with Forms and Tables at Alura Cursos Online
12/2/2019HTML5 and CSS3 Part 1: The First Web Page at Alura Cursos Online
12/2/2019Fundamentals of Agility: Your First Steps Towards Agile Transformation at Alura Cursos Online
12/2/2019JavaScript: Programming in the Language of the Web at Alura Cursos Online
12/2/2019Flexbox: Position Elements on Screen at Alura Cursos Online
12/2/2019Introduction to Programming Logic at Impacta
11/2/2015
Diego is available for hire
Hire Diego B.
All Howdy Candidates are vetted for skills and english proficiency.
