Empty Space (Page 12)

Email Service Verification

This configuration is for mailgun [https://www.mailgun.com/] which I am using to enable email notifications on this blog, I also have used SMTP2GO [https://www.smtp2go.com/] API as another recommended solution for enabling email on domain. This is for the SMTP verification model using Swaks [http://www.

code

Publishing Docker images to Docker Hub

Linking to GitHub repository with documentation https://github.com/Siliconrob/dockerfiles

code

.NET Core 2.1 Docker Framework issue

Found a longstanding docker launch configuration issue with an application when deploying to AWS ECS [https://aws.amazon.com/ecs/] This was the configuration which compiles and runs locally without issue. However every time it was deployed to ECS it would compile->build->deploy. When it came to

code

Docker Configuration files + Environment Variables

When testing locally a configuration file will allow you to quickly set all the launch environment variables easily https://docs.docker.com/engine/reference/commandline/cli/#configuration-files Example file test.env REDIS_HOST=host.docker.internal SPECIAL_KEY=here_it_is Then run the docker container as such docker -it

code

Using CsvHelper to read large CSV files

Working through a CSV file like this. This CSV file is produced automatically and contains hundreds of thousands of records, but the file itself is a collection of several CSV files appended together. Confusing yes, but CsvHelper [https://joshclose.github.io/CsvHelper/] can handle this at do it fast. C,

code

Creating AMI images with Packer

Hashicorp has a large suite of open source excellent utilities and programs that work with a variety of cloud providers and local system management tools. In particular I have been working with creating AMI EC2 templates and the packer [https://www.packer.io/intro/index.html] tools for creating this

code

Add script as cronjob

When you want to add a bash shell script to the local cronjob scheduler to run every minute #!/bin/bash THE_SCRIPT=<path_to_shell_script> sudo chmod -R 755 $THE_SCRIPT croncmd="$THE_SCRIPT > /tmp/the_script.log 2>&1" cronjob="

code

AWS CodeDeploy - GitLab

Integrating with GitLab CI runners you should setup and add the file .gitlab-ci.yml to your repository with the following components image: docker_hub_image before_script: - git submodule sync --recursive - git submodule update --init --recursive - sudo apt-get install zip unzip rsync -y - chmod +x step1_

code

Configure a git submodule

When you have a dependency to a external library which in most cases is because you had to fork the repository to make a change for yourself that the community/author will not/can not accept Add a file in the root of the repository called .gitmodules [submodule "module_

code

CodeDeploy 18.04 LTS

Currently AWS Codedeploy agent on Ubuntu 18.04 LTS does not install properly due to mismatch on ruby dependencies under 18.04 (installs 2.5, but codeploy only works with 2.3) Run the following script as a workaround # Ruby 2.5 sudo apt -y install ruby # Ruby 2.3

code

Supporting realtime data feeds

Currently I spend a non significant amount of time supporting 3rd party public API data feeds that supply satellite imagery through various formats and delivery mechanisms of the following * FTP [ftp://ftp.bom.gov.au/anon/gen/gms/] * SFTP [sftp-reg.cloud.bom.gov.au] * HTTPS [https://satepsanone.nesdis.noaa.gov/

code

LFTP

lftp [https://linux.die.net/man/1/lftp] is a much better version of ftp that can do parallel downloads and automatic recovery along with an interactive batch terminal that can be detached. When having to download over 40 TB of data from this allowed me to write a bash

code

Add header lines to gzipped files

Had to loop through numerous gzipped files that required adding a header line to them. This script uses zcat [https://linux.die.net/man/1/zcat] to write in a compressed .gz file #!/bin/bash fname=$1 echo $fname file_list=`find $fname -type f` for current_file in $file_

code

Postgres script to identify and change ownership

Running into an issue where the current Postgres database continues to get the wrong owner placed on the new/modified table. This handy script identifies the user that should own tables (you know that beforehand) and gives the appropriate statements to correct table/view ownership select t.table_name, t.

code

Bash: AWS SQS CLI + jq

With the almost ubiquitous adoption of JSON as the message format for Restful API calls now when working with bash scripts JSON parsing command line tool jq [https://stedolan.github.io/jq/] is invaluable Make sure to download the jq library for your distribution sudo apt-get install jq -y Start

code

Ngrok

The superior proxy that exposes localhosts, etc to the internet in a secured way. Explanation here [https://ngrok.com/product], but I have used ngrok in the following cases * Exposed my workstation publicly to demonstrate an internal feature directly to users/customers * Configured a temporary SSL certified server to receive

code

API Monitoring

Runscope [https://www.runscope.com/] - The do it all of API monitoring and with a plethora of scripting options and pipeline hooks. I met the founder at MonkeySpace Conference in Chicago right before he launched it and it was I could tell a seminal breakthrough for SaaS companies as

code

Basecamp versus Slack

Communication formats Slack [https://slack.com/] * Immediacy of the chat format * Terrible search * Organization is hard to enforce, groups fracture, distraction galore * Messages are short and context is lost all the time * Integrations can be fun like giphy * Planning is impossible except in tiny cases * Overall distracting, but popular Basecamp

code

Luigi (not Mario's brother)

Python processing framework [https://github.com/spotify/luigi] built by Spotify [https://www.spotify.com/us/] that I found while trying figure out a reliable distributed task orchestrator [https://luigi.readthedocs.io/en/stable/] system. Once you understand that the central scheduler is where the various agents dump their work

code

GitLab runners

Notes on public versus private gitlab runners. GitLab has a common set of public CI/CD build runners that are shared throughout the entire world. While this doesn't cost anything, it can be limiting in what they are capable of doing. For the majority of the time this

code

QuartzNet

QuartzNet is a port of the popular Java library [http://www.quartz-scheduler.org/] to C# that allows for durable background scheduling of jobs with a sophisticated triggering system ranging from immediate to repeatable options. There are other options in the .NET world such as HangFire [https://www.hangfire.io/], QuartzNet

code

Wrestling with Python

In my attempt to write idiomatically correct python code. I laid out the following structure. Define classes using args and kwargs class SyncFileMetadata: file_type = "" def __init__(self, *args, **kwargs): if 'file_type' in kwargs: self.file_type = kwargs.get('file_type') from datetime

code

Autoscaling Groups and Launch Configurations

AWS provides the ability to manage scaling manually or through programmatic API calls. This allows you to save a lot of money if you know what your usual web traffic will look like i.e. you could use spot requests which normally allow you to pay 1/4 the price

code

Awesome Postgres Features

Update 2019-09-10 This tutorial Postgres guide [https://www.guru99.com/postgresql-tutorial.html] by Tushar Korat covers Postgres and is comprehensive and gets you up and going with details and excellent examples. -------------------------------------------------------------------------------- https://pgdash.io/blog/postgres-features.html This goes through some of the great capabilities of Postgres. Key points

code

TravisCI + AppVeyor

.NET Core project Continuous Integration setups for public build SaaS services The following is a working appveyor.yml [https://github.com/Siliconrob/solcast_net/blob/master/appveyor.yml] file for a client library version: '0.0.{build}' configuration: - Release platform: Any CPU # Build worker image (VM template)