Cloud composer orchestration via cloud build Google cloud composer is a managed apache airflow service that helps create, schedule, monitor and manage workflows.Cloud Composer automation helps you create Airflow environments quickly and use Airflow-native tools, such as the powerful Airflow web interface and command line tools, so you can focus on your workflows and not your infrastructure. In this article I will descibe how an engineering team can manage, develop and publish DAGS after running a full CI/CD build pipeline using google cloud build .

Timeseries financial forecasting Recently , I have been looking into various ways to forecast a time series dataset. This is an old pursuit in the field of statistics and there are many well known ways to achieve this. In this post I will demonstrate a very basic (Naive) approach of forecasting a quarterly dataset of sales figure, by using previous 4 years (16 quarters) and forecasting/predicting the next 1 year of sales aggregate.

Selenium is one of my favourite tool for automation. In this post, I will demonstrate some basic code to download a file from a website in a headless mode , and also provide a docker file to make things simpler. Python Code Here is some basic code which will make an attempt to download a **7zip exe. ** from pyvirtualdisplay import Display from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.

Scrapy is a very popular web scraping/crawling framework, I have been using it for quite some time now. In this post, I will demonstrate creating a very basic web crawler. Install Scrapy Installation is via pip pip install scrapy Minimalistic Code A very simple scraper is created like this To Run , simply type scrapy runspider scraper.py Running, above code will output something like below 2018-12-02 14:01:18 [scrapy.utils.log] INFO: Scrapy 1.