Engineer- Data Strategy Integration
Engineer - Data Strategy Integration
Description
In your role as an Integration Engineer, you are capable of doing the job of Web Data Collection
Engineer at a at least a strong Mid or Senior level. You are a technical wizard, willing to learn
and contribute in whatever ways needed to make sure our clients get the data they need. As a
member of the Web Data Integration team, you work on projects that serve the entire Web Data
Acquisition department. You work as an escalation point on the most advanced scraping
projects, you implement solutions outside the platform to perform tasks such as automated QA,
metrics reporting, data transformation, and connecting to outside platforms such as Snowflake
and Postgres databases, Jira.
Because of the expert-level knowledge of our platforms that is required, this role is almost
exclusively a role that is recruited for internally. External candidates with excellent technical
ability and willingness to jump quickly into the intricacies of our tech stack in a humble and
earnest way may also be considered.
As an Integration Engineer, you will work in languages such as Python, SQL, and JavaScript to
fulfill the tasks required. You might write a small web service or set up a small service in AWS
Lambda. You might create reporting that is accessed by the entire company. You might
implement a scraper using another framework or another platform. You will also likely
collaborate with the Mozenda development team or the Data Science team on different
initiatives, for example.
The Tech
At Engage3 we use our own Mozenda platform for the great majority of our web scraping work
rather than open source frameworks like Playwrite or Selenium. We do this because it is robust,
fast, and allows for relatively easy implementation of simple web scraping projects while still
maintaining the flexibility to use advanced techniques to optimize for high-volume collection and
avoiding and bypassing CAPTCHA services. It also provides a degree of consistency and
infrastructure that has allowed us to build and maintain thousands of web scrapers over time
with a relatively small team.
The data we collect for large retail clients feeds into our Snowflake database. Familiarity with
and/or willingness and ability to learn SQL well enough to query collected data and troubleshoot
problems in the collection process is highly encouraged and expected of more senior team
members.
General Expectations
● B2 or above English proficiency
● In-office, Monday-Friday during general business hours (generally 08:00 - 16:30 or 09:00 - 17:30)
● Respect and follow the directions of your team leader
● Respect and develop good working relationships with other team members and with
colleagues in other departments
● A strong ability and willingness to learn and improve in a collaborative and self-directed
way
Skill and Education Requirements
● Provisional -
○ A strong mid to senior WDC engineer may be brought to the team on a
temporary basis with the understanding that they rise to Associate level in 3-5
months
● Associate level - Senior Web Data Engineer requirements plus:
○ Strong SQL query skills and familiarity with the relevant Engage3 Snowflake
databases
○ Some experience working in Python (most likely including courses and simple
personal projects) and willingness to quickly learn more
○ Excellent interpersonal skills
● Mid level - Trainee level plus:
○ Strong intermediate skills in Python and Pandas
○ Ability to read API documentation and implement integrations with other systems
○ Ability to contribute in a real way to department initiatives planned by
management and Senior staff
● Senior level - Mid level requirements plus:
○ Ability to plan and architect new initiatives in collaboration with Team and
department management and Solution Engineers
○ A proven track record of enthusiastic and creative contribution to team objectives