Professional Machine Learning Engineer

Framing ML problems

Translating business challenges into ML use cases. Considerations include:

- Choosing the best solution (ML vs. non-ML, custom vs. pre-packaged [e.g., AutoML, Vision API]) based on the business requirements

- Defining how the model output should be used to solve the business problem

- Deciding how incorrect results should be handled

- Identifying data sources (available vs. ideal)

Defining ML problems. Considerations include:

- Problem type (e.g., classification, regression, clustering)

- Outcome of model predictions

- Input (features) and predicted output format

Defining business success criteria. Considerations include:

- Alignment of ML success metrics to the business problem

- Key results

- Determining when a model is deemed unsuccessful

Identifying risks to feasibility of ML solutions. Considerations include:

- Assessing and communicating business impact

- Assessing ML solution readiness

- Assessing data readiness and potential limitations

- Aligning with Google's Responsible AI practices (e.g., different biases)

Architecting ML solutions

Designing reliable, scalable, and highly available ML solutions. Considerations include:

- Choosing appropriate ML services for the use case (e.g., Cloud Build, Kubeflow)

- Component types (e.g., data collection, data management)

- Exploration/analysis

- Feature engineering

- Logging/management

- Automation

- Orchestration

- Monitoring

- Serving

Choosing appropriate Google Cloud hardware components. Considerations include:

- Evaluation of compute and accelerator options (e.g., CPU, GPU, TPU, edge devices)

Designing architecture that complies with security concerns across sectors/industries. Considerations include:

- Building secure ML systems (e.g., protecting against unintentional exploitation of data/model, hacking)

- Privacy implications of data usage and/or collection (e.g., handling sensitive data such as Personally Identifiable Information [PII] and Protected Health Information [PHI])

Designing data preparation and processing systems

- Visualization

- Statistical fundamentals at scale

- Evaluation of data quality and feasibility

- Establishing data constraints (e.g., TFDV)

Building data pipelines. Considerations include:

- Organizing and optimizing training datasets

- Data validation

- Handling missing data

- Handling outliers

- Data leakage

Creating input features (feature engineering). Considerations include:

- Ensuring consistent data pre-processing between training and serving

- Encoding structured data types

- Feature selection

- Class imbalance

- Feature crosses

- Transformations (TensorFlow Transform)

Developing ML models

Building models. Considerations include:

- Choice of framework and model

- Modeling techniques given interpretability requirements

- Transfer learning

- Data augmentation

- Semi-supervised learning

- Model generalization and strategies to handle overfitting and underfitting

Training models. Considerations include:

- Ingestion of various file types into training (e.g., CSV, JSON, IMG, parquet or databases, Hadoop/Spark)

- Training a model as a job in different environments

- Hyperparameter tuning

- Tracking metrics during training

- Retraining/redeployment evaluation

Testing models. Considerations include:

- Unit tests for model training and serving

- Model performance against baselines, simpler models, and across the time dimension

- Model explainability on Vertex AI

Scaling model training and serving. Considerations include:

- Distributed training

- Scaling prediction service (e.g., Vertex AI Prediction, containerized serving)

Automating and orchestrating ML pipelines

- Identification of components, parameters, triggers, and compute needs (e.g., Cloud Build, Cloud Run)

- Orchestration framework (e.g., Kubeflow Pipelines/Vertex AI Pipelines, Cloud Composer/Apache Airflow)

- Hybrid or multicloud strategies

- System design with TFX components/Kubeflow DSL

Implementing serving pipelines. Considerations include:

- Serving (online, batch, caching)

- Google Cloud serving options

- Testing for target performance

- Configuring trigger and pipeline schedules

Tracking and auditing metadata. Considerations include:

- Organizing and tracking experiments and pipeline runs

- Hooking into model and dataset versioning

- Model/dataset lineage

Monitoring, optimizing, and maintaining ML solutions

Monitoring and troubleshooting ML solutions. Considerations include:

- Performance and business quality of ML model predictions

- Logging strategies

- Establishing continuous evaluation metrics (e.g., evaluation of drift or bias)

- Understanding Google Cloud permissions model

- Identification of appropriate retraining policy

- Common training and serving errors (TensorFlow)

- ML model failure and resulting biases

Tuning performance of ML solutions for training and serving in production. Considerations include:

- Optimization and simplification of input pipeline for training

- Simplification techniques