Every Data Science project starts with … ETL

• Have you ever written an R script to be run in a non-interactive way?
• Did it work?
• Have you ever scheduled an R script to run without human intervention?
• Do you have any R script running in production?
• … with central logging, proper monitoring and alerting?
• … run on cheap spot-instances with real-time performance metrics collected and feeding an AI picking the optimal instance type for the next run?

Every BI consulting firm have developed its own …

• job scheduler
• data repository and metadata documentation tool
• central logging
• and a few other things …

Every BI consulting firm have developed its own …

• job scheduler
• data repository and metadata documentation tool
• central logging
• and a few other things …

Today we can just pick the right open-source tool, such as

• Jenkins
• Airflow or Luigi
• cronR and taskscheduleR R packages
• cloud services (eg AWS Batch, Cloudwatch, Lambda)

Source: When my co-worker wants to simplify code …

… use version control!

Source: A junior vs a senior during a huge system failure

git 101

Start from scratch:

git init .
git add .
git commit -m "hello world"
git push origin master

Contribute to an existing project:

git clone https://github.com/foo/bar
git branch -b feature-branch
git add .; git add -u .
git commit -m "add my new cool feature"
git push origin feature-branch

git 102

Version control all your scripts!

• R packages
• R scripts
• Other scripts
• Configuration files
• Rmd reports
• Ad-hoc queries

git 201

Read more at https://happygitwithr.com