Hashicorp has a large suite of open source excellent utilities and programs that work with a variety of cloud providers and local system management tools. In particular I have been working with creating AMI EC2 templates and the packer [https://www.packer.io/intro/index.html] tools for creating this

When you want to add a bash shell script to the local cronjob scheduler to run every minute #!/bin/bash THE_SCRIPT=<path_to_shell_script> sudo chmod -R 755 $THE_SCRIPT croncmd="$THE_SCRIPT > /tmp/the_script.log 2>&1" cronjob="

Integrating with GitLab CI runners you should setup and add the file .gitlab-ci.yml to your repository with the following components image: docker_hub_image before_script: - git submodule sync --recursive - git submodule update --init --recursive - sudo apt-get install zip unzip rsync -y - chmod +x step1_

When you have a dependency to a external library which in most cases is because you had to fork the repository to make a change for yourself that the community/author will not/can not accept Add a file in the root of the repository called .gitmodules [submodule "module_

Currently AWS Codedeploy agent on Ubuntu 18.04 LTS does not install properly due to mismatch on ruby dependencies under 18.04 (installs 2.5, but codeploy only works with 2.3) Run the following script as a workaround # Ruby 2.5 sudo apt -y install ruby # Ruby 2.3

Currently I spend a non significant amount of time supporting 3rd party public API data feeds that supply satellite imagery through various formats and delivery mechanisms of the following * FTP [ftp://ftp.bom.gov.au/anon/gen/gms/] * SFTP [sftp-reg.cloud.bom.gov.au] * HTTPS [https://satepsanone.nesdis.noaa.gov/

lftp [https://linux.die.net/man/1/lftp] is a much better version of ftp that can do parallel downloads and automatic recovery along with an interactive batch terminal that can be detached. When having to download over 40 TB of data from this allowed me to write a bash

Had to loop through numerous gzipped files that required adding a header line to them.  This script uses zcat [https://linux.die.net/man/1/zcat] to write in a compressed .gz file #!/bin/bash fname=$1 echo $fname file_list=`find $fname -type f` for current_file in $file_

Running into an issue where the current Postgres database continues to get the wrong owner placed on the new/modified table.  This handy script identifies the user that should own tables (you know that beforehand) and gives the appropriate statements to correct table/view ownership select t.table_name, t.

With the almost ubiquitous adoption of JSON as the message format for Restful API calls now when working with bash scripts JSON parsing command line tool jq [https://stedolan.github.io/jq/] is invaluable Make sure to download the jq library for your distribution sudo apt-get install jq -y Start

The superior proxy that exposes localhosts, etc to the internet in a secured way.  Explanation here [https://ngrok.com/product], but I have used ngrok in the following cases * Exposed my workstation publicly to demonstrate an internal feature directly to users/customers * Configured a temporary SSL certified server to receive

Runscope [https://www.runscope.com/] - The do it all of API monitoring and with a plethora of scripting options and pipeline hooks.  I met the founder at MonkeySpace Conference in Chicago right before he launched it and it was I could tell a seminal breakthrough for SaaS companies as

Communication formats Slack [https://slack.com/] * Immediacy of the chat format * Terrible search * Organization is hard to enforce, groups fracture, distraction galore * Messages are short and context is lost all the time * Integrations can be fun like giphy * Planning is impossible except in tiny cases * Overall distracting, but popular Basecamp

Python processing framework [https://github.com/spotify/luigi] built by Spotify [https://www.spotify.com/us/] that I found while trying figure out a reliable distributed task orchestrator [https://luigi.readthedocs.io/en/stable/] system.  Once you understand that the central scheduler is where the various agents dump their work

Notes on public versus private gitlab runners.  GitLab has a common set of public CI/CD build runners that are shared throughout the entire world.  While this doesn't cost anything, it can be limiting in what they are capable of doing. For the majority of the time this

QuartzNet is a port of the popular Java library [http://www.quartz-scheduler.org/] to C# that allows for durable background scheduling of jobs with a sophisticated triggering system ranging from immediate to repeatable options.  There are other options in the .NET world such as HangFire [https://www.hangfire.io/], QuartzNet

In my attempt to write idiomatically correct python code.  I laid out the following structure. Define classes using args and kwargs class SyncFileMetadata: file_type = "" def __init__(self, *args, **kwargs): if 'file_type' in kwargs: self.file_type = kwargs.get('file_type') from datetime

AWS provides the ability to manage scaling manually or through programmatic API calls.  This allows you to save a lot of money if you know what your usual web traffic will look like i.e. you could use spot requests which normally allow you to pay 1/4 the price

Update 2019-09-10 This tutorial Postgres guide [https://www.guru99.com/postgresql-tutorial.html] by Tushar Korat covers Postgres and is comprehensive and gets you up and going with details and excellent examples. -------------------------------------------------------------------------------- https://pgdash.io/blog/postgres-features.html This goes through some of the great capabilities of Postgres. Key points

.NET Core project Continuous Integration setups for public build SaaS services The following is a working appveyor.yml [https://github.com/Siliconrob/solcast_net/blob/master/appveyor.yml] file for a client library version: '0.0.{build}' configuration: - Release platform: Any CPU # Build worker image (VM template)

Telnet Is a friendly app that is on almost all Linux machines and many windows machines that you can use for some basic connectivity testing. telnet ip port Examples - 10.0.0.1 suppose is the server Test for RDP enabled telnet 10.0.0.1 3389 If telnet

Postgres is a powerful database that can run anything it seems, but to kepp it healthy may not be as obvious and that is where PgHero can step in and give you a nice dashboard [https://pghero.dokkuapp.com/datakick] to help review and monitor your database instance. This should

This is a visualization tool [https://www.elastic.co/products/kibana] that can be run as a docker container and hooked to your preferred datasource for increased visibility and information.  When running a SaaS it is incredibly useful for visualizing time interval windows easily.

When you need to find the appropriate Ubuntu base image for AWS EC2 machines https://cloud-images.ubuntu.com/locator/ec2/