Robin Michael

Statements are legal defense in 194 countries

code

GeoJSON formatting with jq

Having to format some GeoJSON data like so { "type": "FeatureCollection", "name": "My Places", "crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } }, "features": [{ "

code

Bash loop cURL+ wget API extraction

Script to automate a set of sequential downloads from bounded list of Ids. Use cURL to get the JSON data from the API and then extract the relevant fields to create a nice CSV download and finally remove the unnecessary quote characters in the output. Linux commands used * cURL [https:

code

Bash Add extension to all files

Having to add an extension to a series of files as they were produced by a splitting function and not in expected format Here all files in the directory are read in as loop and then moved with the new file name by using the linux command basename [http://man7.

code

Web Scraping with lynx + cURL

Wanted to grab data from a webpage and process the results.  In this case the results are a csv file dump Get the utilities to be used here sudo apt-get install curl lynx cargo cargo install xsv; Setup XSV with Cargo export PATH="$HOME/.cargo/bin:$PATH" Copy

code

Bash script to unzip files in directory

This link [https://stackoverflow.com/a/29248777/194333]for a script to unzip all files within a directory works flawlessly in bash with only the use of zip [https://linux.die.net/man/1/zip] and sed [https://www.gnu.org/software/sed/manual/sed.html] In case link ever

code

Bash Rename Specific File to Directory Name

This is helpful when you have many different directories say something like -dir1 --metafile.txt -dir2 --metafile.txt In this case the directory names are different, but the underlying filenames are the same because in actuality you want to join them all together most likely or they were produced by

code

Bash Rename Spaces in Files

Spaces in files are not easy to work with on linux/unix system architectures.  Easiest solution is to replace the space character with underscores. for f in *\ *; do mv "$f" "${f// /_}"; done

code

Georeference an image

A popular format for storing geographic information onto a raster image is the TIF [https://en.wikipedia.org/wiki/TIFF] format which allows the raster image to be tagged with the boundaries it represents. This makes the file able to reference and modify coordinate systems/transform.  However from time to

code

PostGIS - Temp table for query comparison

Had to do a ETL [https://en.wikipedia.org/wiki/Extract,_transform,_load] task of creating a set of points in 4326 lat/lng projection to compare against the current table data in the same 4326 lat/lng projection and find all points that are unique and not within a

PostGIS - Buffer a Polygon

Needed to buffer a polygon (rectangle) that was listed in lat/lng (4326) projection to world mercator (3857) by a defined limit 10KM.  Problem is that 4326 is expecting decimal degrees 3857 is expecting meters.  However PostGIS understands both and has functions to convert between the two. In this case

code

Bash sort CSV file contents line by line

Suppose you have a CSV file like this pepperoni feta cheese mozzarella cheese beef,bacon,mozzarella cheese,onions,pineapple sausage pepperoni,artichokes four cheese And you want to sort each lines contents alphabetically to this pepperoni feta cheese mozzarella cheese bacon,beef,mozzarella cheese,onions,pineapple sausage artichokes,pepperoni four

code

Curl and jq with pipes

Bash Pipe Commands explained sudo apt-get install curl jq -y && curl -s "http://files.olo.com/pizzas.json" | jq -r '.[].toppings | @csv' | sort | uniq -c | sort -bgr | head -20 | cat Install cURL and jq if not on the debian Linux variant sudo apt-get install

Remove Carriage Returns

Simple command to remove carriage returns from files saved with windows line endings sed -i 's/\r//g' *.*

code

Syncing Fork from original

This is the situation where you fork the original repository and now want to update your fork to reflect all the new commits and changes present in the original You will need to add the remote origin as a available git remote -v This will list your current remotes which

code

Python UTC to Local DateTime

Example code to extract a serialized from JSON response with a property named period_end expressed as an ISO8601 [https://en.wikipedia.org/wiki/ISO_8601] timestamp string value to a localized python DateTime JSON data response data is of this format { "forecasts": [ { "pv_estimate": 0,

code

Systemd - Assign AWS reserved IP

For Ubuntu 16.04 and higher Systemd Configuration [Unit] Description=<Name of Service> After=network.target After=systemd-user-sessions.service After=network-online.target [Service] User=ubuntu ExecStart=path_to_bash_script.sh [Install] WantedBy=multi-user.target Then the script can be a standard bash shell script like so #!/bin/

code

HtmlAgilityPack

Sometimes you need to parse out raw HTML (scraping because there is no public API).  In python there is the excellent library BeautifulSoup [https://www.crummy.com/software/BeautifulSoup/bs4/doc/] that does this all of this nicely, but in .NET world I have found that HtmlAgilityPack [https://html-agility-pack.net/

code

PostGIS - Adjacent polygons

Trying to figure out the following situations for geographic data.  Here is the problem as you can see the point lies within chunk 64, but chunk 60 should be queried as well in this case because within the chunks are data points that contain the resource data and in this

personal

Hire me in 2019

I love to turn ideas into reality through software and want to make a difference in the world and your organization. I have been working remotely with a team based in Australia [https://solcast.com.au/]making SaaS [https://en.wikipedia.org/wiki/Software_as_a_service] solar energy available

personal

GitLab Activity Overview

GitLab activity [https://gitlab.com/siliconrob] - As GitLab doesn't have the same option as GitHub to anonymize contributions my GitLab profile looks empty, but here is my overview contribution chart that is not publicly viewable.

code

Email Service Verification

This configuration is for mailgun [https://www.mailgun.com/] which I am using to enable email notifications on this blog, I also have used SMTP2GO [https://www.smtp2go.com/] API as another recommended solution for enabling email on domain. This is for the SMTP verification model using Swaks [http://www.

code

Publishing Docker images to Docker Hub

Linking to GitHub repository with documentation https://github.com/Siliconrob/dockerfiles

code

.NET Core 2.1 Docker Framework issue

Found a longstanding docker launch configuration issue with an application when deploying to AWS ECS [https://aws.amazon.com/ecs/] This was the configuration which compiles and runs locally without issue.  However every time it was deployed to ECS it would compile->build->deploy.  When it came to

code

Docker Configuration files + Environment Variables

When testing locally a configuration file will allow you to quickly set all the launch environment variables easily https://docs.docker.com/engine/reference/commandline/cli/#configuration-files Example file test.env REDIS_HOST=host.docker.internal SPECIAL_KEY=here_it_is Then run the docker container as such docker -it

code

Using CsvHelper to read large CSV files

Working through a CSV file like this.  This CSV file is produced automatically and contains hundreds of thousands of records, but the file itself is a collection of several CSV files appended together. Confusing yes, but CsvHelper [https://joshclose.github.io/CsvHelper/] can handle this at do it fast. C,