Lead Data Engineer

JOB SUMMARY: Responsible for technical design, writing, support and operations of efficient, scalable, and extensible data integrations and ingestions. Implement effective solutions from the ground up and investigate anomalies in a mature running data platform and environment. Work with business, developers, analysts, architecture and product owners to develop data integration and ingestion solutions that are critical to the enterprise analytics, reporting and application environments leveraged by all areas of the business. Perform all data platform related configuration, software and server installations, upgrades and patching partnering with the necessary Infrastructure, IT Security and other teams as needed. MAJOR RESPONSIBILITIES: Implement enhancements for prioritized data integration and ingestions work efforts partnering with the business to ensure they meet the specified requirements. Implement critical and non-critical system data integration and ingestion fixes for the data platform and environment. Ensuring root cause resolution to identified problems. Monitor and support the campaign management environment and processes in order to meet the daily SLA. Recommend and code efficient and effective data integration and ingestion solutions for challenging problems for medium to large work efforts of medium to high complexity. Translate business requirements, data mappings into data integrations and ingestions code solutions. Support Release and Change Management processes along with other UPT Compliance initiatives. Work effectively with Technology Operations/Infrastructure teams to ensure continued operations and maintenance of the data integrations and ingestions as well as the data platform and environment. Understands and actively participates in Environmental, Health & Safety responsibilities by following established UO policy, procedures, training and team member involvement activities. Performs other duties as assigned. EDUCATION: Technical degree, Bachelor’s degree; or highly relevant equivalent experience is required. EXPERIENCE: 7+ years of demonstrated experience working as part of large Information Technology teams and/or consulting organizations partnering with clients/business groups in a role developing and supporting a complex data environment; or equivalent combination of education and experience. Strong understanding of software development lifecycle Experience in a transaction-based environment is a MUST Proficient with scripting languages, data ingestion tools and corresponding tools, preferably Python, Pip, Pipenv, Bash, Scala, Spark, PySpark ScalaSpark, NIFI, and Java Script, and one object-oriented language like Java/C# Able to work in and with Cloud and non-cloud technologies like AWS, Azure, Hadoop etc. Experience with one or more relational databases/data technologies like DB2, SQL Server, Snowflake. Able to write and ready complex SQL, stored procedures and leverage backend databases. Should be able to triage database performance issues with SQL being sent to the database. Able to read and explain plans, and understand the complexity. Experience working with scheduling, automation, file transfer, monitoring, alerting, notification and logging tools like Airflow, RunDeck, SFTP etc. Experience working in Linux environment Experience with source control management such as Git and/or SVN Strong problem solving skills, and the ability to communicate complex concepts clearly and accurately Familiarity with no-SQL databases like Couch, MongoDB, Hbase or Cassandra Knowledge of Data Science, Analytics and Business Intelligence/Reporting practices and tools as it pertains to data platforms and environments (R-Studio, Jupyter, Zeppelin, Tableau or DsX) Experience interacting with Kafka and API/Service programming and transactional databases Experience with principles of data governance and quality as well as tools used (i.e. Ranger, Atlas) Understanding of containers, resource management or YARN is desirable Able to perform software and server installat