Senior Data Engineer
Location:Wiltshire
Contract:2 months
Pay:Up to£62.50p/h Inside IR35 (DOE)
Due to the nature of the work, we require candidates who already hold active MOD DV clearance.
Responsibilities:
Design, build and maintain secure, scalable data solutions across the full lifecycle, including acquisition, integration, storage, processing and analysis.
Develop and maintain data ingestion and transformation pipelines using Apache NiFi, Python and SQL.
Design data models, schemas and data contracts / Interface Control Documents (ICDs) to support reliable integration between systems.
Integrate data from REST APIs and other source systems, working with Parquet and S3-compatible object storage.
Implement data validation, assurance, metadata management and traceability to ensure data is accurate, complete and fit for purpose.
Develop tools for data-driven insights and task automation, and research and adopt appropriate emerging technologies.
Collaborate with customer and internal stakeholders to define data, interface and tool requirements.
Work closely with DevOps Engineers, Data Scientists, Analysts and Cyber Consultants to deliver integrated solutions.
Take ownership of deliverables and ensure the quality, security, and maintainability of outputs.
Experience required
Proven experience delivering data engineering solutions in a project or production environment.
Strong Python and SQL skills.
Hands-on experience with Apache NiFi and data pipeline orchestration.
Experience with data modelling and schema design.
Experience with Parquet and S3-compatible object storage.
Experience with REST API integration.
Experience with OpenShift or Kubernetes.
Experience with GitLab CI/CD practices and pipelines.
Experience developing Data contracts or Interface Control Documents (ICDs).
Experience with data validation and assurance.
Experience with metadata management, lineage and traceability.
Disclaimer:
This vacancy is being advertised by either Advanced Resource Managers Limited, Advanced Resource Managers IT Limited or Advanced Resource Managers Engineering Limited ("ARM"). ARM is a specialist talent acquisition and management consultancy. We provide technical contingency recruitment and a portfolio of more complex resource solutions. Our specialist recruitment divisions cover the entire technical arena, including some of the most economically and strategically important industries in the UK and the world today. We will never send your CV without your permission. Where the role is marked as Outside IR35 in the advertisement this is subject to receipt of a final Status Determination Statement from the end Client and may be subject to change
