https://bravenewgeek.com/tag/analytics-pipeline/Recent content in Analytics Pipeline on Brave New GeekHugoen-usThu, 15 Oct 2020 10:51:25 -0500https://bravenewgeek.com/continuous-deployment-for-aws-glue/Thu, 15 Oct 2020 10:51:25 -0500https://bravenewgeek.com/continuous-deployment-for-aws-glue/<p><a href="https://aws.amazon.com/glue">AWS Glue</a> is a managed service for building ETL (Extract-Transform-Load) jobs. It’s a useful tool for implementing analytics pipelines in AWS without having to manage server infrastructure. Jobs are implemented using Apache Spark and, with the help of <a href="https://docs.aws.amazon.com/glue/latest/dg/dev-endpoints.html">Development Endpoints</a>, can be built using Jupyter notebooks. This makes it reasonably easy to write ETL processes in an interactive, iterative fashion. Once finished, the Jupyter notebook is converted into a Python script, uploaded to S3, and then run as a Glue job.</p>https://bravenewgeek.com/implementing-etl-on-gcp/Wed, 15 Jul 2020 15:53:17 -0500https://bravenewgeek.com/implementing-etl-on-gcp/<p>ETL (Extract-Transform-Load) processes are an essential component of any data analytics program. This typically involves loading data from disparate sources, transforming or enriching it, and storing the curated data in a data warehouse for consumption by different users or systems. An example of this would be taking customer data from operational databases, joining it with data from Salesforce and Google Analytics, and writing it to an OLAP database or BI engine.</p>