Coding Blocks

We are committed to continuing our deep dive into Git from the Bottom Up by John Wiegley, while Allen puts too much thought into onions, Michael still doesn't understand proper nouns, and Joe is out hat shopping. The full show notes for this episode are available at https://www.codingblocks.net/episode192. News Thanks for the review Alex Woodhead! Want to help out the show? Leave us a review! Ludum Dare is a bi-annual game jam that's been running for over 20 years now. Jam #51 is coming up Sept 30th to October 3rd. (ldjam.com) We previously talked about Ludum Dare in episode 146. Commitment Issues Commits A commit can have one or more parents. Those commits can have one more parents. It's for this reason that commits can be treated like branches, because they know their entire lineage. You can examine top level referenced commits with the following command: git branch -v. A branch is just a named reference to a commit! A branch and a tag both name a commit, with the exception that a tag can have a description, similar to a commit. Branches are just names that point to a commit. Tags have descriptions and point to a commit. Knowing the above two points, you actually don't technically need branches or tags. You could do everything pointing to the commit hash id's if you were insane enough to do so. Here's a dangerous command: git reset --hard commitHash – This is dangerous. --hard says to erase all changes in the working tree, whether they were registered for a check-in or not and reset HEAD to point to the commitHash. Here's a safer command: git checkout commitHash – This is a safer option, because files changed in the working tree are preserved. However, adding the -f parameter acts similar as the previous command, except that it doesn't change the branch's HEAD, and instead only changes the working tree. Some simple concepts to grasp: If a commit has multiple parents, it's a merge commit. If a commit has multiple children, it represents the ancestor of a branch. Simply put, Git is a collection of commits, each of which holds a tree which reference other trees and blobs, which store data. All other things in Git are named concepts but they all boil down to the above statement. A commit by any other name The key to knowing Git is to truly understand commits. Learning to name your commits is the way to mastering Git. branchname – The name of a branch is an alias to the most recent commit on that branch. tagname – Similar to the branch name in that the name points to a specific commit but the difference is a tag can never change the commit id it points to. HEAD – The currently checked out commit. Checking out a specific commit takes you out of a "branch" and you are then in a "detached HEAD" state. The 40 character hash id – A commit can always be referenced by the full SHA1 hash. You can refer to a commit by a shorter version of the hash id, enough characters to make it unique, usually 6 or 7 characters is enough. name^ – Using the caret tells Git to go to the parent of the provided commit. If a commit has more than one parent, the first one is chosen. name^^ – Carets can be stacked, so doing two carets will give the parent of the parent of the provided commit. name^2 – If a commit has multiple parents, you can choose which one to retrieve by using the caret followed by the number of the parent to retrieve. This is useful for things like merge commits. name~10 – Same thing as using the commit plus 10 carets. It refers to the named commit's 10th generation ancestor. name:path – Used to reference a specific file in the commit's content tree, excellent when you need to do things like compare file diffs in a merge, like: git diff HEAD^1:somefile HEAD^2:somefile. name^{tree} – Reference the tree held by a commit rather than the commit itself. name1..name2 – Get a range of commits reachable from name2 all the way back to, but not including, name1. Omitting name1 or name2 will substitute HEAD in the place. name1…name2 – For commands like log, gets the unique commits that are referenced by name1 or name2. For commands like diff, the range is is between name2 and the common ancestor of name1 and name2. main.. – Equivalent to main..HEAD and useful when comparing changes made in the current branch to the branch named main. ..main – Equivalent to HEAD..main and useful for comparing changes since the last rebase or merge with the branch main, after fetching it. -since="2 weeks ago" – All commits from a certain relative date. –until="1 week ago" – All commits before a certain relative date. –grep=pattern – All commits where the message meets a certain regex pattern. –committer=pattern — Find all the commits where the committer matches a regex pattern. –author=pattern – All commits whose author matches the pattern. So how's that different than the committer? "The author of a commit is the one who created the changes it represents. For local development this is always the same as the committer, but when patches are being sen

It's surprising how little we know about Git as we continue to dive into Git from the Bottom Up, while Michael confuses himself, Joe has low standards, and Allen tells a joke. The full show notes for this episode are available at https://www.codingblocks.net/episode191. News Thanks for all the great feedback on the last episode and for sticking with us! Directory Content Tracking Put simply, Git just keeps a snapshot of a directory's contents. Git represents your file contents in blobs (binary large object), in a structure similar to a Unix directory, called a tree. A blob is named by a SHA1 hashing of the size and contents of the file. This verifies that the blob contents will never change (given the same ID). The same contents will ALWAYS be represented by the same blob no matter where it appears, be it across commits, repositories, or even the Internet. If multiple trees reference the same blob, it's simply a hard link to the blob. As long as there's one link to a blob, it will continue to exist in the repository. A blob stores no metadata about its content. This is kept in the tree that contains the blob. Interesting tidbit about this: you could have any number of files that are all named differently but have the same content and size and they'd all point to the same blob. For example, even if one file were named abc.txt and another was named passwords.bin in separate directories, they'd point to the same blob. This allows for compact storage. Introducing the Blob This is worth following along and trying out. The author creates a file and then calculates the ID of the file using git hash-object filename. If you were to do the same thing on your system, assuming you used the same content as the author, you'd get the same hash ID, even if you name the file different than what they did. git cat-file -t hashID will show you the Git type of the object, which should be blob. git cat-file blob hashID will show you the contents of the file. The commands above are looking at the data at the blob level, not even taking into account which commit contained it, or which tree it was in. Git is all about blob management, as the blob is the fundamental data unit in Git. Blobs are Stored in Trees Remember there's no metadata in the blobs, and instead the blobs are just about the file's contents. Git maintains the structure of the files within the repository in a tree by attaching blobs as leaf nodes within a tree. git ls-tree HEAD will show the tree of the latest commit in the current directory. git rev-parse HEAD decodes the HEAD into the commit ID it references. git cat-file -t HEAD verifies the type for the alias HEAD (should be commit). git cat-file commit HEAD will show metadata about the commit including the hash ID of the tree, as well as author info, commit message, etc. To see that Git is maintaining its own set of information about the trees, commits and blobs, etc., use find .git/objects -type f and you'll see the same IDs that were shown in the output from the previous Git commands. How Trees are Made There's a notion of an index, which is what you use to initially create blobs out of files. If you just do a git add without a commit, assuming you are following along here (jwiegly.github.io), git log will fail because nothing has been committed to the repository. git ls-files --stage will show your blob being referenced by the index. At this point the file is not referenced by a tree or a commit, it's only in the .git/index file. git write-tree will take the contents of the index and write it to a tree, and the tree will have it's own hash ID. If you followed along with the link above, you'd have the same hash from the write-tree that we get. A tree containing the same blob and sub-trees will always have the same hash. The low-level write-tree command is used to take the contents of the index and write them into a new tree in preparation for a commit. git commit-tree takes a tree's hash ID and makes a commit that holds it. If you wanted that commit to reference a parent, you'd have to manually pass in the parent's commit ID with the -p argument. This commit ID will be different for everyone because it uses the name of the creator of the commit as well as the date when the commit is created to generate the hash ID. Now you have to overwrite the contents of .git/refs/heads/master with the latest commit hash ID. This tells Git that the branch named master should now reference the new commit. A safer way to do this, if you were doing this low-level stuff, is to use git update-ref refs/heads/master hashID. git symbolic-ref HEAD refs/heads/master then associates the working tree with the HEAD of master. What Have We Learned? Blobs are unique! Blobs are held by Trees, Trees are held by Commits. HEAD is a pointer to a particular commit. Commits usually have a parent, i.e. previous, commit. We've got a better understanding of the detached HEAD state. What a lot of those files mean in the .git directory. Resources We L

After working with Git for over a decade, we decide to take a deep dive into how it works, while Michael, Allen, and Joe apparently still don't understand Git. The full show notes for this episode are available at https://www.codingblocks.net/episode190. News Thank you for the new reviews DisturbedMime and Gjuko! Want to help out the show? Leave us a review! Survey Says How do you feel about Git? Take the survey at: https://www.codingblocks.net/episode190. Git Under the Covers This is the book Outlaw was trying to remember … we think! How to approach Git; general strategy This episode was inspired by an article written by Mark Dominus. Git commits are immutable snapshots of the repository. Branches are named sequences of commits. Every object gets a unique id based on its content. The author is not a fan of how the command set has evolved over time. With Git, you need to think about what state your repository is in, and what state you would like to be in. There are likely a number of ways to achieve that desired state. If you try to understand the commands without understanding the model, you can get lost. For example: git reset does three different things depending on the flags used, git checkout even worse (per the author), and The opposite of git-push is not git-pull, it's git-fetch. Possibly the worst part of the above is if you don't understand the model and what's happening to the model, you won't know the right questions to ask to get back into a good state. Mark said the thing that saved him from frustration with Git is the book Git from the Bottom Up by John Wiegley (jwiegley.github.io) Mark doesn't love Git, but he uses it by choice and he uses it effectively. He said that reading Wiegley's book is what changed everything for him. He could now "see" what was happening with the model even when things went wrong. It is very hard to permanently lose work. If something seems to have gone wrong, don't panic. Remain calm and ask an expert. Mark Dominus Git from the Bottom Up A repository – "is a collection of commits, each of which is an archive of what the project's working tree looked like at a past date, whether on your machine or someone else's." It defines HEAD, which identifies the branch or commit the current tree started from, and contains a set of branches or tags that allow you to identify commits by a name. The index is what will be committed on the next commit. Git does not commit changes from the working tree into the repository directly so instead, the changes are registered into the index, which is also referred to as a staging area, before committing the actual changes. A working tree is any directory on your system that is associated with a Git repository and typically has a .git folder inside it. Why typically? Thanks to the git-worktree command, one .git directory can be used to support multiple working trees, as previously discussed in episode 128. A commit is a snapshot of your working tree at some point in time. "The state of HEAD (see below) at the time your commit is made becomes that commit's parent. This is what creates the notion of a 'revision history'." A branch is a name for a commit, also called a reference. This stores the history of commits, the lineage and is typically referred to as the "branch of development" A tag is also a name for a commit, except that it always points to the same commit unlike a branch which doesn't have to follow this rule as new commits can be made to the branch. A tag can also have its own description text. master was typically, maybe not so much now, the default branch name where development is done in a repository. Any branch name can be configured as the default branch. Currently, popular default branch names include main, trunk, and dev. HEAD is an alias that lets the repository identify what's currently checked out. If you checkout a branch, HEAD now symbolically points to that branch. If you checkout a tag, HEAD now refers only to that commit and this state is referred to as a "detached HEAD". The typical work flow goes something like: Create a repository, Do some work in your working tree, Once you've achieved a good "stopping point", you add your changes to the index via git add, and then Once your changes are in the state you want them and in your index, you are ready to put your changes into the actual repository, so you commit them using git commit. Resources We Like Things I wish everyone knew about Git (Part 1) (blog.plover.com) Git from the Bottom Up by John Wiegley (jwiegley.github.io) Orthogonality (Wikipedia) Version Control with Git: Powerful tools and techniques for collaborative software development (Amazon) Comparing Git Workflows (episode 90) git-worktree (git-scm.com) Designing Data-Intensive Applications – SSTables and LSM-Trees (episode 128) Celeste is quite a game, it's challenging but rewarding…or accessible and rewarding. Just go play it already! Tip of the Week Celeste is a tough, but forgiving game that is on all m

Once again, Stack Overflow takes the pulse of the developer community where we have all collectively decided to switch to Clojure, while Michael is changing things up, Joe is a future predicting trailblazer, and Allen is "up in the books". The full show notes for this episode are available at https://www.codingblocks.net/episode189. News Thank you for the new review fizzybuzzybeezy! Want to help out the show? Leave us a review! Joe's going to be speaking at the Orlando Elastic Meetup about running Elasticsearch in Kubernetes on July 27th 2022 (Meetup) Recommendation, keep your API interface in separate modules from your implementation! That makes it easier to re-use that code in new ways without having to refactor first. Survey Says How well does the Stack Overflow Survey results align with your reality? Take the survey at: https://www.codingblocks.net/episode189. Stack Overflow 2022 Survey Follow along with the show on the website (survey.stackoverflow.co) Resources we Like 2022 Developer Survey (survey.stackoverflow.co) Asked and answered: the results for the 2022 Developer survey are here! (stackoverflow.blog) Stack Overflow Annual Developer Survey (insights.stackoverflow.com) Is Docker the new Git? (codingblocks.net) Tip of the Week Do you worry about talking too much in virtual meetings? This app monitors your mic and lets you know when you're waffling on. (unblah.me) Did you know you can do a regex search in grep? Example: grep -Pzo "(?s)^(\s)\Nmain.?{.?^\1}" *.c (Stack Overflow) But what if you want to do that in vim? By default vim treats characters literally, but you can turn on "magic" characters with :set magic and then you're off to the races! (Stack Overflow) Looking for some great IntelliJ Code Completion Tips? Check out this video! (YouTube) Did you know yum install will not return an error code when installing multiple packages at the same time if one succeeds and another fails. Yikes! So be sure to install your dependencies independent of their dependents. (Stack Overflow)

We're going back in time, or is it forward?, as we continue learning about Google's automation evolution, while Allen doesn't like certain beers, Joe is a Zacker™, and Michael poorly assumes that UPSes work best when plugged in. The full show notes for this episode are available at https://www.codingblocks.net/episode188. Survey Says For your day job, are you primarily working ... Take the survey at: https://www.codingblocks.net/episode188. The famous "SRE Book" from Google Automation Begets Reliability Automating Yourself Out of a Job A cautionary, err, educational tale of automating MySQL for Ads and automating replica replacements. Migrating MySQL to Borg (Google Cluster Manager) Large-scale cluster management at Google with Borg (research.google) Desired goals of the project: Eliminate machine/replica maintenance, Ability to run multiple instances on same machine. Came with additional complications – Borg task moving caused problems for master database servers. Manual failovers took a long time. Human involvement in the failovers would take longer than the required 30 seconds or less downtime. Led to automating failover and the birth of MoB (MySQL on Borg). Again, more problems because now application code needed to become much more failure tolerant. After all this, mundane tasks dropped by 95%, and with that they were able to optimize and automate other things causing total operational costs to drop by 95% as well. Automating Cluster Delivery Story about a particular setup of Bigtable that didn't use the first disk of a 12 disk cluster. Some automation thought that if the first disk wasn't being utilized, then none of the disks weren't configured and were safe to be wiped. Automation should be careful about implicit "safety" signals. Cluster delivery automation depended on a lot of bespoke shell scripts which turned out to be problematic over time. Detecting Inconsistencies with ProdTest ProdTest contained chained "unit" tests that would figure out where problems began Cluster automations required custom flags, which led to constant problems / misconfigurations. Shell scripts became brittle over time. Were all the services available and configured properly? Were the packages and configurations consistent with other deployments? Could configuration exceptions be verified? For this, ProdTest was created. Tests could be chained to other tests and failures in one would abort causing subsequent tests to not run. The tests would show where something failed and with a detailed report of why. If something new failed, they could be added as new tests to help quickly identify them in the future. These tools gave visibility into what was causing problems with cluster deployments. While the finding of things quicker was nice, that didn't mean faster fixes. Dozens of teams with many shell scripts meant that fixing these things could be a problem. The solution was to pair misconfigurations with automated fixes that were idempotent This sounded good but in reality some fixes were flaky and not truly idempotent and would cause the state to be "off" and other tests would now start failing. There was also too much latency between a failure, the fix, and another run. Specializing Automation processes can vary in one of three ways: Competence, Latency, Relevance: the proportion of real world processes covered by automation. They attempted to use "turnup" teams that would focus on automation tasks, i.e. teams of people in the same room. This would help get things done quicker. This was short-lived. Could have been over a thousand changes a day to running systems! When the automation code wasn't staying in sync with the code it was covering, that would cause even more problems. This is the real world. Underlying systems change quickly and if the automation handling those systems isn't kept up, then more problems crop up. This created some ugly side effects by relieving teams who ran services of the responsibility to maintain and run their automation code, which created ugly organizational incentives: A team whose primary task is to speed up the current turnup has no incentive to reduce the technical debt of the service-owning team running the service in production later. A team not running automation has no incentive to build systems that are easy to automate. A product manager whose schedule is not affected by low-quality automation will always prioritize new features over simplicity and automation. Turnups became inaccurate, high-latency, and incompetent. They were saved by security by the removal of SSH approaches to more auditable / less-privileged approaches. Service Oriented Cluster Turnup Changed from writing shell scripts to RPC servers with fine-grained ACL (access control lists). Service owners would then create / own the admin servers that would know how their services operated and when they were ready. These RPC's would send more RPC's to admin server's when their ready state was reached. This resulted in low-la

We explore the evolution of automation as we continue studying Google's Site Reliability Engineering, while Michael, ah, forget it, Joe almost said it correctly, and Allen fell for it. The full show notes for this episode are available at https://www.codingblocks.net/episode187. News Thank you for the new reviews: ASobering, rupeshbende, Mnmbrane, angry_little_hamster, jonsmith1982 Want to help out the show? Leave us a review! rupeshbende asks: How do you find time to do this along with your day job and hobbies as this involves so much studying on your part? Survey Says What's your favorite Tom Cruise movie? Take the survey at: https://www.codingblocks.net/episode187. Automation Why Do We Automate Things? The famous "SRE Book" from Google Consistency: Humans make mistakes, even on simple tasks. Machines are much more reliable. Besides, tasks like creating accounts, resetting passwords, applying updates aren't exactly fun. Platform: Automation begets automation, smaller tasks can be tweaked or combined into bigger ones. Pays dividends, providing value every time it's used as opposed to toil which is essentially a tax. Platforms centralize logic too, making it easier to organize, find, and fix issues. Automation can provide metrics, measurements that can be used to make better decisions. Faster Repairs: The more often automation runs, it hits the same problems and solutions which brings down the average time to fix. The more often the process runs, the cheaper it becomes to repair. Faster Actions: Automations are faster than humans. Many automations would be prohibitively expensive for humans to do, Time Saving: It's faster in terms of actions, and anybody can run it. If we are engineering processes and solutions that are not automatable, we continue having to staff humans to maintain the system. If we have to staff humans to do the work, we are feeding the machines with the blood, sweat, and tears of human beings. Think The Matrix with less special effects and more pissed off System Administrators. Joseph Bironas The Value of SRE at Google Google has a strong bias for automation because of their scale. Google's core is software, and they don't want to use software where they don't own the code and they don't want processes in place that aren't automated. You can't scale tribal knowledge. They invest in platforms, i.e. systems that can be improved and extended over time. Google's Use Cases for Automation Much of Google's automation is around managing the lifecycle of systems, not their data. They use tools such as chef, puppet, cfengine, and PERL(!?). The trick is getting the right level of abstraction. Higher level abstractions are easier to work with and reason about, but are "leaky". Hard to account for things like partial failures, partial rollbacks, timeouts, etc. The more generic a solution, the easier it is to apply more generally and tend to be more reusable, but the downside is that you lose flexibility and resolution. The Use Cases for Automation Google's broad definition of automation is "meta-software": software that controls software. Examples: Account creation, termination, Cluster setup, shutdown, Software install and removal, Software upgrades, Configuration changes, and Dependency changes A Hierarchy of Automation Classes Ideally you wouldn't need to stitch systems together to get them to work together. Systems that are separate, and glue code can suffer from "bit rot", i.e. changes to either system can work poorly with each other or with the havoc. Glue code is some of the hardest to test and maintain. There are levels of maturity in a system. The more rare and risky a task is, the less likely it is to be fully automated. Maturity Model When your levels of abstraction get to be very sophisticated, you can lose the ability to work effectively at a lower level. Kind of like trying to make your own toaster today (Gizmodo). No automation: database failover to a new location manually. Externally maintained system-specific automations: SRE has a couple commands they run in their notes. Externally maintained generic system-specific automation: SRE adds a script to a playbook. Internally maintained system-specific automation: the database ships with a script. System doesn't need automation: Database notices and automatically fails over. Can you automate so much that developers are unable to manually support systems when a (very rare) need occurs? Resources we Like Links to Google's free books on Site Reliability Engineering (sre.google) Site Reliability Engineering book (sre.google) Chapter 7: The Evolution of Automation at Google (sre.google) Ultimate List of Programmer Jokes, Puns, and other Funnies (Medium) Shared success in building a safer open source community (blog.google) One Man's Nearly Impossible Quest to Make a Toaster From Scratch (Gizmodo) The Man Who Spent 17 Years Building The Ultimate Lamborghini Replica In His Basement Wants to Sell It (Jalopnik) Tip of the Week There's an easy way

We finished. A chapter, that is, of the Site Reliability Engineering book as Allen asks to make it weird, Joe has his own pronunciation, and Michael follows through on his promise. The full show notes for this episode are available at https://www.codingblocks.net/episode186. Sponsors Retool – Stop wrestling with UI libraries, hacking together data sources, and figuring out access controls, and instead start shipping apps that move your business forward. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. News Thank you for the review, Lospas! Want to help out the show? Leave us a review! Another great post from @msuriar, this time about the value of hiring junior developers. (suriar.net) Survey Says Did you intern or co-op while you were in school? Take the survey at: https://www.codingblocks.net/episode186. More about Monitoring Less The famous "SRE Book" from Google Instrumentation and Performance Need to be careful and not just track times, such as latencies, on medians or means. A better way is to bucketize the data as a histogram, meaning to count how many instances of a request occurred in the given bucket, such as the example latency buckets in the book of 0ms – 10ms, 10ms – 30ms, 30ms-100ms, etc. Choosing the Appropriate Resolution for Measurements The gist is that you should measure at intervals that support the SLO's and SLA's. For example, if you're targeting a 99.9% uptime, there's no reason to check for hard-drive fullness more than once or twice a minute. Collecting measurements can be expensive, for both storage and analysis. Best to take an approach like the histogram and keep counts in buckets and aggregate the findings, maybe per minute. As Simple as Possible, No Simpler It's easy for monitoring to become very complex: Alerting on varying thresholds and measurements, Code to detect possible causes, Dashboards, etc. Monitoring can become so complex that it becomes difficult to change, maintain, and it becomes fragile. Some guidelines to follow to keep your monitoring useful and simple include: Rules that find incidents should be simple, predictable and reliable, Data collection, aggregation and alerting that is infrequently used (the book said less than once a quarter) should be a candidate for the chopping block, and Data that is collected but not used in any dashboards or alerting should be considered for deletion. Avoid attempting to pair simple monitoring with other things such as crash detection, log analysis, etc. as this makes for overly complex systems. Tying these Principles Together Google's monitoring philosophy is admittedly maybe hard to attain but a good foundation for goals. Ask the following questions to avoid pager duty burnout and false alerts: Does the rule detect something that is urgent, actionable and visible by a user? Will I ever be able to ignore this alert and how can I avoid ignoring the alert? Does this alert definitely indicate negatively impacted users and are there cases that should be filtered out due to any number of circumstances? Can I take action on the alert and does it need to be done now and can the action be automated? Will the action be a short-term or long-term fix? Are other people getting paged about this same incident, meaning this is redundant and unnecessary? Those questions reflect these notions on pages and pagers: Pages are extremely fatiguing and people can only handle a few a day, so they need to be urgent. Every page should be actionable. If a page doesn't require human interaction or thought, it shouldn't be a page. Pages should be about novel events that have never occurred before. It's not important whether the alert came from white-box or black-box monitoring. It's more important to spend effort on catching the symptoms over the causes and only detect imminent causes. Monitoring for the Long Term Monitoring systems are tracking ever-changing software systems, so decisions about it need to be made with long term in mind. Sometimes, short-term fixes are important to get past acute problems and buy you time to put together a long term fix. Two case studies that demonstrate the tension between short and long term fixes Bigtable SRE Originally Bigtable's SLO was based on an artificial, good client's mean performance. Bigtable had some low level problems in storage that caused the worst 5% of requests to be significantly slower than the rest. These slow requests would trip alerts but ultimately the problems were transient and unactionable. People learned to de-prioritize these alerts, which sometimes were masking legitimate problems. Google SRE's temporarily dialed back the SLO to the 75th percentile to trigger fewer alerts and disabled email alerts, while working on the root cause, fixing the storage problems. By slowing the alerts it gave engineers the breathing room they needed to deep dive the problem. Gmail Gmail was originally built on a distributed process mana

We haven't finished the Site Reliability Engineering book yet as we learn how to monitor our system while the deals at Costco as so good, Allen thinks they're fake, Joe hasn't attended a math class in a while, and Michael never had AOL. The full show notes for this episode are available at https://www.codingblocks.net/episode185. Sponsors Retool – Stop wrestling with UI libraries, hacking together data sources, and figuring out access controls, and instead start shipping apps that move your business forward. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. News Thank you for the reviews! just_Bri, 1234556677888999900000, Mannc, good beer hunter Want to help out the show? Leave us a review! Great post from @msuriar who was an actual SRE at Google and had great feedback on our episode on toil and lots of other stuff! (suriar.net) Some other great recommendations from @msuriar: Slack's Outage on January 4th 2021 (slack.engineering) Post-Incident Review on the Atlassian April 2022 outage (Atlassian) Great episode on All The Code featuring Brandon Lyons and his journey to Microsoft. (ListenNotes.com) Couldn't resist posting this: Survey Says Which book should we finish? Take the survey at: https://www.codingblocks.net/episode185. Monitor Some of the Things Terminology Monitoring – Collecting, processing, and aggregating quantitative information about a system. White-box monitoring – Monitoring based on metrics exposed by a system, i.e. logs, JVM profiling, etc. Black-box monitoring – Monitoring a system as a user would see it. Dashboard – Provides a summary view of the most important service metrics. May display team information, ticket queue size, high priority bugs, current on call engineer, recent pushes, etc. Alert – Notification intended to be read by a human, such as tickets, email alerts, pages, etc. Root cause – A defect, that if corrected, creates a high confidence level that the same issue won't be seen again. There can be multiple root causes for a particular incident (including a lack of testing!) Node and machine – A single instance of a running kernel. Kernel – The core of the operating system. Generally controls everything on the system, always resident in memory, and facilitates interactions between the system hardware and software. (Wikipedia) There could be multiple services worth monitoring on the same node that could be either related or unrelated. Push – Any change to a running service or it's configuration. Why Monitor? The famous "SRE Book" from Google Some of the main reasons include: To analyze trends, To compare changes over time, and Alerting when there's a problem. To build dashboards to answer basic questions. Ad hoc analysis when things change to identify what may have caused it. Monitoring lets you know when the system is broken or may be about to break. You should never alert just if something seems off. Paging a human is an expensive use of time. Too many pages may be seen as noise and reduce the likelihood of thorough investigation. Effective alerting systems have good signal and very low noise. Setting Reasonable Expectations for Monitoring Monitoring complex systems is a major undertaking. The book mentions that Google SRE teams with 10-12 members have one or two people focused on building and maintaining their monitoring systems for their service. They've reduced the headcount needed for maintaining these systems as they've centralized and generalized their monitoring systems, but there's still at least one human dedicated to the monitoring system. They also ensure that it's not a requirement that an SRE stare at the screen to identify when a problem comes up. Google has since moved to simpler and faster monitoring systems that provide better tools for ad hoc analysis and avoid systems that try to determine causality This doesn't mean they don't monitor for major changes in common trends. SRE's at Google seldom use tiered rule triggering. Why? Because they're constantly changing their service and/or infrastructure. When they do alert on these dependent types of rules, it's when there's a common task that's carried out that is relatively simple. It is critical that from the instant a production issue arises, that the monitoring system alert a human quickly, and provide an easy to follow process that people can use to find the root cause quickly. Alerts need to be simple to understand and represent the failure clearly. Symptoms vs Causes A monitoring system should answer these two questions: What is broken? This is the symptom. Why is it broken? This is the cause. The book says that drawing the line between the what and why is one of the most important ways to make a good monitoring system with high quality signals and low noise. An example might be: Symptom: The web server is returning 500s or 404s, Cause: The database server ran out of hard-drive space. Black-Box vs White-Box Google SRE's use white-b

We say "toil" a lot this episode while Joe saw a movie, Michael says something controversial, and Allen's tip is to figure it out yourself, all while learning how to eliminate toil. The full show notes for this episode are available at https://www.codingblocks.net/episode184. Sponsors Retool – Stop wrestling with UI libraries, hacking together data sources, and figuring out access controls, and instead start shipping apps that move your business forward. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Reviews Thank you for the reviews! AA, Franklin MacDunnaduex, BillyVL, DOM3ag3 Want to help out the show? Leave us a review! Survey Says Does your job include any toil? Take the survey at: https://www.codingblocks.net/episode184. The famous "SRE Book" from Google Chapter 5: Eliminating Toil Toil is not just work you don't wanna do, nor is it just administrative work or tedious tasks. Toil is different for every individual. Some administrative work has to be done and is not considered toil but rather it's overhead. HR needs, trainings, meetings, etc. Even some tedious tasks that pay long term dividends cannot be considered toil. Cleaning up service configurations was an example of this. Toil further defined is work that is often times manual, repetitive, can be automated, has no real value, and/or grows as the service does. Manual – Something a human has to do. Repetitive – Running something once or twice isn't toil. Having to do it frequently is. Automatable – If a machine can do it, then it should be done by the machine. If the task needs human judgement, it's likely not toil. Tactical – Interrupt driven rather than strategy driven. May never be able to eliminate completely but the goal is to minimize this type of work. No enduring value – If your service didn't change state after the task was completed, it was likely toil. If there was a permanent improvement in the state of the service then it likely wasn't toil. O(n) with service growth – If the amount of work grows with the growth of your service usage, then it's likely toil. Why is Less Toil Better? At Google, the goal is to keep each SRE's toil at less than 50%. The other 50% should be developing solutions to reduce toil further, or make new features for a service. Where features mean improving reliability, performance, or utilization. The goal is set at 50% because it can easily grow to 100% of an SRE's time if not addressed. The time spent reducing toil is the "engineering" in the SRE title. This engineering time is what allows the service to scale with less time required by an SRE to keep it running properly and efficiently. When Google hires an SRE, they promise that they don't run a typical ops organization and mention the 50% rule. This is done to help ensure the group doesn't turn into a full time ops team. Calculating Toil The book gave the example of a 6 person team and a 6 week cycle: Assuming 1 week of primary on-call time and 1 week of secondary on-call time, that means an SRE has 2 of 6 weeks with "interrupt" type of work, or toil, meaning 33% is the lower bound of toil. With an 8 person team, you move to an 8 week cycle, so 2 weeks on call out of 8 weeks mean a 25% toil lower bound. At Google, SRE's report their toil is spent most on interrupts (non-urgent, service related messages), then on-call urgent responses, then releases and pushes. Surveys at Google with SRE's indicate that the average time spent in toil is closer to 33%. Like all averages, it leaves out outliers, such as people who spend 0 time toiling, and others who spend as much as 80% of their time on toil. If there is someone taking on too much toil, it's up to the manage to spread that out better. What Qualifies as Engineering? Work that requires human judgement, Produces permanent improvements in a service and requires strategy, Design driven approach, and The more generic or general, the better as it may be applied to multiple services to get even greater gains in efficiency and reliability. Typical SRE Activities Software engineering – Involves writing or modifying code. Systems engineering – Configuring systems, modifying configurations, or documenting systems that provide long term improvements. Toil – Work that is necessary to run a service but is manual, repetitive, etc. Overhead – Administrative work not directly tied to a service such as hiring, HR paperwork, meetings, peer-reviews, training, etc. The 50% goal is over a few quarters or year. There may be some quarters where toil goes above 50%, but that should not be sustained. If it is, management needs to step in and figure out how to bring that back into the goal range. "Let's invent more, and toil less" Site Reliability Engineering: How Google Runs Production Systems Is Toil Always Bad? The fact that some amount of toil is predictable and repeatable makes some individuals feel like they're accomplishing something, i.e. quick wins that

Welcome to the morning edition of Coding Blocks as we dive into what service level indicators, objectives, and agreements are while Michael clearly needs more sleep, Allen doesn't know how web pages work anymore, and Joe isn't allowed to beg. The full show notes for this episode are available at https://www.codingblocks.net/episode183. Sponsors Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Survey Says SLIs and SLOs sounds awesome ...but does your team use them? Take the survey at: https://www.codingblocks.net/episode183. News Monolithic repos … meh. But monolithic builds … oh noes. Chapter 4: Service Level Objectives The famous "SRE Book" from Google Service Level Indicators A very well and carefully defined metric of some aspect of the service or system. Response latency, error rate, system throughput are common SLIs. SLIs are typically aggregated over some predefined period of time. Usually, SLIs directly measure some aspect of a system but it's not always possible, as with client side latency. Availability is one of the most important SLIs often expressed as a ratio of the number of requests that succeed, sometimes called yield. For storage purposes, durability, i.e. the retention of the data over time, is important. Service Level Objectives The SLO is the range of values that you want to achieve with your SLIs. Choosing SLOs can be difficult. For one, you may not have any say in it! An example of an SLO would be for response latency to be less than 250ms. Often one SLI can impact another. For instance, if your number of requests per second rises sharply, so might your latency. It is important to define SLOs so that users of the system have a realistic understanding of what the availability or reliability of the system is. This eliminates arbitrary "the system is slow" or the "system is unreliable" comments. Google provided an example of a system called Chubby that is used extensively within Google where teams built systems on top of Chubby assuming that it was highly available, but no claim was made to that end. Sort of crazy, but to ensure service owners didn't have unrealistic expectations on the Chubby's up-time, they actually force downtime through the quarter. Service Level Agreements These are the agreements of what is to happen if/when the SLOs aren't met. If there is no consequence, then you're likely talking about an SLO and not an SLA. Typically, SLA's consequences are monetary, i.e. there will be a credit to your bill if some service doesn't meet it's SLO. SLAs are typically decided by the business, but SREs help in making sure SLO consequences don't get triggered. SREs also help come up with objective ways to measure the SLOs. Google search doesn't have an SLA, even though Google has a very large stake in ensuring search is always working. However, Google for Work does have SLAs with its business customers. What Should You Care About? You should not use every metric you can find as SLIs. Too many and it's just noisy and hard to know what's important to look at. Too few and you may have gaps in understanding the system reliability. A handful of carefully selected metrics should be enough for your SLIs. Some Examples User facing services: Availability – could the request be serviced, Latency – how long did it take the request to be serviced, and Throughput – how many requests were able to be serviced. Storage systems: Latency – how long did it take to read/write, Availability – was it available when it was requested, and Durability – is the data still there when needed. Big data systems: Throughput – how much data is being processed, and End to end latency – how long from ingestion to completion of processing. Everything should care about correctness. Collecting Indicators Many metrics come from the server side. Some metrics can be scraped from logs. Don't forget about client-side metric gathering as there might be some things that expose bad user experiences. Example Google used is knowing what the latency before a page can be used is as it could be bad due to some JavaScript on the page. Aggregation Typically aggregate raw numbers/metrics but you have to be careful. Aggregations can hide true system behavior. Example given averaging requests per second: if odd seconds have 200 requests per second and even seconds have 0, then your average is 100 but what's being hidden is your true burst rate of 200 requests. Same thing with latencies, averaging latencies may paint a pretty picture but the long tail of latencies may be terrible for a handful of users. Using distributions may be more effective at seeing the true story behind metrics. In Prometheus, using a Summary metric uses quantiles so that you can see typical and worst case scenarios. Quantile of 50% would show you the average request, while Quantile of 99.99% would show you the worst request durations. A really interesting takeaway here is that studies have s

We learn how to embrace risk as we continue our learning about Site Reliability Engineering while Johnny Underwood talked too much, Joe shares a (scary) journey through his mind, and Michael, Reader of Names, ends the show on a dark note. The full show notes for this episode are available at https://www.codingblocks.net/episode182. Sponsors Retool – Stop wrestling with UI libraries, hacking together data sources, and figuring out access controls, and instead start shipping apps that move your business forward. Survey Says How do we feel about DevOps? Take the survey at: https://www.codingblocks.net/episode182. Reviews Thanks for the help Richard Hopkins and JR! Want to help out the show? Leave us a review! News Great notes in the Slack from @msuriar! Sadly, O'Reilly is ending their partnership with ACM, so you'll no longer get access to their Learning Platform if you're a member. (news.ycombinator.com) Chapter 3: Embracing Risk Google aims for 100% reliability right? Wrong… Increasing reliability is always better for the service, right? Not necessarily. It's very expensive to add another 9 of reliability, and Can't iterate on features as you spend more time and resources making the service more stable. Users don't typically notice the difference between very reliable and extremely reliable services. The systems using these services usually aren't 100% reliable, so the chances of noticing are very low. SRE's try to balance the risk of unavailability with innovation, new features, and efficient service operations by optimizing for the right balance of all. Managing Risk Unstable systems diminish user confidence. We want to avoid that. Cost does not scale with improvements to reliability. As you improve reliability the cost can actually increase many times over. Two dimensions of cost: Cost of redundancy in compute resources, and The opportunity cost of trading features for reliability focused time. SREs try to balance business goals in reliability with the risk of service reliability. If the business goal is 99.99% reliable, then that's exactly what the SRE will aim for, with maybe just a touch more. They treat the target like a minimum and a maximum The famous "SRE Book" from Google Measuring Service Risk Identify an objective metric for a property of the system to optimize. Only by doing this can you measure improvements or degradation over time. At Google, they focus on unplanned downtime. Unplanned downtime is measured in relation to service availability. Availability = Uptime / (Uptime + Downtime). A 99.99% target means a maximum of 52.56 minutes downtime in a year. At Google, they don't use uptime as the metric as their services are globally distributed and may be up in many regions while being down in another. Rather, they use the successful request rate. Success rate = total successful requests / total requests. A 99.99% target here would mean you could have 250 failures out of 2.5M requests in a day. NOTE: not all services are the same. A new user signup is likely way more important than a polling service for checking for new emails for a user. At Google they also use this success rate for non-customer facing systems. Google often sets quarterly availability targets and may track those targets weekly or even daily. Doing so allows for fixing any issues as quickly as possible. Risk Tolerance Services SRE's should work directly with the business to define goals that can be engineered. Sometimes this can be difficult because measuring consumer services is clearly definable whereas infrastructure services may not have a direct owner. Identifying the Risk Tolerance of Consumer Services Often a service will have its own dedicated team and that team will best know the reliability requirements of that service. If there is no owning team, often times the engineers will assume the role of defining the reliability requirements. Factors in assessing the risk tolerance of a service What level of availability is needed? Do different failures have different effects on the service? Use the service cost to help identify where on the risk continuum it belongs. What are the important metrics to track? Target level of availability What do the users expect? Is the service linked directly to revenue, either for Google or for a customer? Is it a free or paid service? If there's a competing service, what is their level of service? What's the target market? Consumers or enterprises? Consider Google Apps that drive businesses, externally they may have a 99.9% reliability because downtime really impacts the end businesses ability to do critical business processes. Internally they may have a higher targeted reliability to ensure the enterprises are getting the best level of customer service. When Google purchased YouTube, their reliability was lower because Google was more focused on introducing features for the consumer. Types of failures Know the shapes of errors. Which is worse, a constant trickle of errors throughout the da

It's finally time to learn what Site Reliability Engineering is all about, while Jer can't speak nor type, Merkle got one (!!!), and Mr. Wunderwood is wrong. The full show notes for this episode are available at https://www.codingblocks.net/episode181. Survey Says So, DevOps is a culture, but SRE is a job title? Take the survey at: https://www.codingblocks.net/episode181. Reviews Thanks for the review "Amazon Customer"! (You, er, we know who you are.) Site Reliability Engineering Site Reliability Engineering: How Google Runs Production Systems is a collections of essays, from Google's perspective, released in 2016 … and it's free. (sre.google) There's a free workbook to go along with it too. (sre.google) But how is SRE as a career? (GlobalDots.com) Career Advancement Score (out of 10): 9 Median Base Salary: $200,000 Job Openings (YoY growth): 1,400+ (72%) These essays are what one company did, that company being Google. The book is told from the perspective of people within the company. It is about scaling a business process, rather than just the machinery. Site Reliability Engineering: How Google Runs Production Systems Their tale should be used for emulating, not copying. 40-90% of your effort is after you have deployed a system. The notion that once your software is "stable", the easy part starts is just plain wrong. Yeah, but what is a Site Reliability Engineering role? It's engineers who apply the principles of computer science and engineering to the design and development of computing systems, usually large distributed ones. It includes writing software for those systems. Including building all the additional pieces those systems need, i.e. backups, load balancers, etc. Reliability … the most fundamental feature of any product? Software doesn't matter much if it can't be used. Software need only to be reliable "enough". Once you've accomplished this, you spend time building more features or new products. SRE's also focus on operating services on top of the distributed computing systems. Examples include: Storage, Email, and Search. Reliability is regarded as the primary focus of the SRE. The book was largely written to help the community as a whole by exposing what Google did to solve the post deploy problems as well as to help define what they believe the role and function is for an SRE. They also call out in the book that they hope the information in the book will work for small to large businesses. Even though they know small businesses don't have the budget and manpower of larger businesses, the concepts here should help any software development shop. However, we acknowledge that smaller organizations may be wondering how they can best use the experience represented here: much like security, the earlier you care about reliability, the better. Site Reliability Engineering: How Google Runs Production Systems It's less costly to implement the beginnings of lightweight reliability support early in the software process rather than introduce something later that's not present at all or has no foundation. Who was the first SRE? Maybe Margaret Hamilton? (Wikipedia) The SRE way: Thoroughness, Dedication, Belief in the value of preparation and documentation, and Awareness of what could go wrong, and the strong desire to prevent it. Hope is not a strategy. Site Reliability Engineering: How Google Runs Production Systems Chapter 1 – Introduction The famous "SRE Book" from Google Consider the sysadmin approach to system management: The sysadmins run services and respond to events and updates as they happen. Teams typically grow as the capacity is needed. Usually the skills for a product developer and a sysadmin are different, therefore they end up on different teams, i.e. a development team and an operations team (i.e. the sysadmins). This approach is easy to implement. Disadvantages of the sysadmin approach: Direct costs that are not subtle and are easy to see. As the size and complexity of the services managed by the operations team grows, so does the operations team. Doesn't scale well because manual intervention with regards to change management and process updates requires more manpower. Indirect costs that are subtle and often more costly than the direct costs. Both teams speak about things with different vocabularies (i.e. no ubiquitous language from back in the DDD days). Each team has different assumptions about risk and possibilities for technical solutions. Each team has different assumptions about target level of product stability. Due to these differences, these teams usually end up in conflict. How quickly should software be released to production? Developers want their features out as soon as possible for their customers. Operations teams want to make sure the software won't break and be a pain to manage in production. A developer always wants their software released as fast as possible. An ops person would want to minimize the amount of changes to ensure the system is as stable as possible. Thi

We're living through the tail end, maybe?, of the Great Resignation, so we dig into how that might impact software engineering careers while Allen is very somber, Joe's years are … different, and Michael pronounces each hump. The full show notes for this episode are available at https://www.codingblocks.net/episode180. Sponsors Mergify – Save time by automating your pull requests and securing the code merge using a merge queue. Survey Says What's most important to you when you're looking for another job? Take the survey at: https://www.codingblocks.net/episode180. Reviews Thanks for the review Chuck Rugged (or is it Rugged?). What is "the great resignation"? The Great Resignation is an ongoing economic trend where a lot of people started quitting their jobs in 2021 and peaked at 3% unemployment (up roughly 50% from the pre-COVID unemployment average). Primarily, but not exclusively, in the US, but also trended in Europe, China, India, Australia as well. Some interesting factors: High worker demand and labor shortages. High unemployment. Employees between 30 and 45 years old have had the greatest increase in resignation rates, with an average increase of more than 20% between 2020 and 2021. Resignation rates actually dropped for people in their 20s. Tech and healthcare led the trend, 4.5% for US, 3.6% for healthcare. Reasons cited included stagnant wages and working conditions. Why is this a big deal? Hiring is expensive! Think of thinks like referral fees, recruiter's percentage, takes a while for people to become productive, onboarding, etc. What does this mean for working conditions? More remote, better compensation, more flexibility, etc.? Why do people change jobs? Promotion, Work life balance, Compensation, Flexibility, Leaving a bad environment, and/or Better company What can you gain? Salary bands, FAANG vs local vs remote vs startup The "TC" (total compensation) Trap Restricted Stock Units vs Options Top paying companies, by level (levels.fyi) Comparing levels across orgs (levels.fyi) About those levels Senior engineers are senior developers who may specialize in a specific area, oversee projects, and manage junior developers. Principal Engineer is a highly experienced engineer who oversees a variety of projects from start to finish. Staff engineer is a senior, individual contributor role in a software engineering organization. There is no "one" kind of staff engineer and many fall into one of four archetypes: Tech Lead, Architect, Solver, and Right Hand. (staffeng.com) Is there a hiring level cap? What does that mean? What can you lose? The people, The grass isn't always greener, Seniority (don't be the "At X we …" person), and/or Comfort Resources we Like Great Resignation (Wikipedia) The Great Resignation: Data and analysis show it's not as great as screaming headlines suggest (NevadaCurrent.com) The Great Resignation is here. What does that mean for developers? (stackoverflow.blog) Who Is Driving the Great Resignation? (hbr.org) What Do Software Developers Want Out of Their Next Job? (insights.dice.com) How the Government Measures Unemployment (bls.gov) Salary comparisons (levels.fyi) Tip of the Week Did you know you can expand or collapse all the files in a pull request on GitHub? Press Alt + Click on any file chevron in the pull request to collapse or expand them all! (github.blog) Kotlin code for using Google Cloud! (cloud.google.com) Thanks to Dave Follett for sharing How to securely erase your hard drive or SSD! (pcworld.com) Thanks to Fuzzy Muffin for sharing Nvchad, a nice face for Neovim (Nvim) that adds some nice features, like directory access and tabs. (nvchad.github.io) How do you merge two Git repositories? (Stack Overflow) Use git-sizer to get various statistics about your repository. (GitHub) How to find/identify large commits in git history? (Stack Overflow) Then forget about BFG and filter-branch, git filter-repo is the way to remove large files from your Git repo (GitHub) Use --shallow-exclude to exclude commits found in the supplied ref in either (or both) your git clone (git-scm.com) or git fetch operations. (git-scm.com) Limit your git push operation "up to" a commit by using the format git push :refs/heads/. (If the already exists on the , you can leave off the refs/heads/ portion. (git-scm.com)

We dive into what it takes to adhere to minimum viable continuous delivery while Michael isn't going to quit his day job, Allen catches the earworm, and Joe is experiencing full-on Stockholm syndrome. The full show notes for this episode are available at https://www.codingblocks.net/episode179. Sponsors Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Survey Says How mature is your CI/CD pipeline? Take the survey at: https://www.codingblocks.net/episode179. Sidebar Revisiting unit testing private methods in 2022, what would you do? Minimum Viable Continuous Integration CD is the engineering discipline of delivering all changes in a standard way safely. minimumcd.org The belief is that you must at least put a certain core set of pieces in play to reap the benefits of Continuous Delivery. The outcome that they're looking for is the improved speed, quality, and safety of the deployment pipeline. Minimum requirements: Use continuous integration, continuously integrating work into the trunk of your version control and ensuring, as much as possible, that the product is releasable. The application pipeline is the ONLY way to deploy to an environment. The pipeline decides if the work is releasable. The artifacts created by the pipeline meet the organization's requirement for being deployable. The artifacts are considered immutable, nobody may change them after they were created by the pipeline. All feature work stops if the pipeline status is red. Must have a production like test environment. Must have rollback on demand capability. Application configuration is deployed with the artifacts. If the pipeline says everything looks good, that should be enough – it forces the focus on what 'releasable' means. Dave Farley Continuous Integration Use trunk based development. Integrated daily at a minimum. Automated testing before merging work code to the trunk. Work is tested with other work automatically during a merge. All feature work stops when the build is red. New work does not break already delivered work. Trunk Based development What is trunk based development? Developers collaborate on a single branch, usually named trunk, main, or something similar. You must resist any pressure to create other long-lived development branches. The argument is that the simplicity of this structure is more than worth anything you might gain by any other structure. For small teams, this is easy, each committer commits straight to trunk, after a build/test gate. For larger teams, you use short-lived feature branches that might live for a couple days max and end with a PR review and build/test gate. What does this buy us? The codebase is always releasable on demand. Google, Facebook, authors of Continuous Delivery and The DevOps Handbook advocate for it. But how do we … Big feature? Feature flag it off. Hot fix? Fix forward. But … What if you need multiple CONSECUTIVE releases? i.e. think of the Kubernetes release cycle. What if you need multiple CONCURRENT releases? i.e. think of Microsoft support for multiple versions of Windows. Resources we Like Minimum Viable CD (MinimumCD.org) Beyond the Minimums, aka references (MinimumCD.org) Frequent Questions (MinimumCD.org) What Is a Deployment Pipeline? (Informit) Trunk Based Development: Introduction (TrunkBasedDevelopment.com) Real Example of a Deployment Pipeline in the Fintech Industry (YouTube) The Twelve-Factor App, III. Config (12Factor.net) Kubernetes Best Practices: Blueprints for Building Successful Applications on Kubernetes (Amazon) Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Amazon) Comparing Git Workflows (episode 90) Our discussions of The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (Coding Blocks) Tip of the Week Did you know you cannot set environment variables in Java? Terms & Conditions Apply is a game where you have to avoid giving up all your juicy data to Evil Corp by carefully avoiding accepting the terms and conditions. Good luck. Thanks Lars Onselius! (TermsAndConditions.game) Test Containers is a Java library that gives you a way to interact with and automate containers for testing purposes. Thanks Rionmonster! (TestContainers.org) Maybe it's time for JSON to die? YAML is finicky, but it's easier to read and it allows comments. YamlDotNet is a library that makes this easy in C#. (YamlDotNet.org) PowerToys are a collection of utilities from Microsoft that extend Window with some really powerful and easy to use features. Thanks Scott Harden! (Microsoft) Did you know you can now include diagrams inside your markdown files on Github? Mermaid is the name and you can create the diagrams directly in your files and keep it versioned along with your code. Thanks Murali and Scott Harden! (github.blog)

We have a retrospective about our recent Game Ja Ja Ja Jam, while Michael doesn't know his A from his CNAME, Allen could be a nun, and Joe still wants to be a game developer. The full show notes for this episode are available at https://www.codingblocks.net/episode178. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Survey Says What percentage of time does your team devote to technical debt per release cycle? Take the survey at: https://www.codingblocks.net/episode178. News Thanks for the review! iTunes: €l£M£N$! All CBJam Games are playable now! (itch.io) You can see a review of the top games we'll be talking about as well as a montage of all the games on YouTube. Check out all of the games! Lessons Learned itch.io is a social network(!) where you can follow other developers, learn about other game jams, etc. Most games are playable in a browser. Writing copy is hard! Playtesting – lots of games are hard! Game Duration – aim for a really good, well paced 1-5 minutes. Time zones – Start, end, defaulting to a little extra Teams – submissions, collaborators, voting How much is too much? Does theme skew voting? What do we get from competition? Durations Would you think making the event longer would get more submissions? How long should you get to vote? Breakdowns Platform Editor Rendering Submitted By CBJam 22 Stats Top 5 Overall Followers 5: Followers (itch.io) Tomaz Saraiva (itch.io) (youtube.com) Followers is a clever 2D platformer puzzle game where you play as 2 different characters in 2 different levels at the same time. You hit the jump button, they both jump. You move left or right, they both attempt to move left or right. The task is to safely collect all the fruit in the level but you "safely" is the keyword here because you have to be really thoughtful about your actions. The trick is utilize the various obstacles in the different levels to position your characters just right to get through. Doing this is a real mind bender, but it's really satisfying when you figure out how to get through. The art and music fit perfectly and it's such a cool riff on the the theme since both characters literally follow along with whatever you tell them…which usually leads to their demise! Knock 'Em 4: Knock 'Em (itch.io) Eike Kriesel Leon Arndt (itch.io) Knock 'Em is a 3D arcade style action game where you play as a bowling ball tasked with knocking down bowling pins…that sounds pretty normal, but once you start the game you realize that you are in for a treat. You have full reign of the bowling alley, and the characters in this game are all bowling pins. That's right, you are a bowling ball blasting around in an alley full of bowling pins that are…bowling! You get a point for every pin and are chased by alley staff pins and police pins that will ultimately wear down your health and end the game. The action is frantic and fun and funny, with a lot of attention to details. Just Down the Hall 3: Just Down the Hall (itch.io) Michael M. – Programming Lead, Game Design, Level Design, UI Design Cheyenne M. – Art, Programming, Game Design, Sound Design Alex M. – Game Testing, Special Thanks Just Down the Hall is a spooky 2D action platformer. In the game a deadly shadow is following behind you, jumping when you jump. If you clear an obstacle the shadow will plow into it, slowing it down. If you hit an obstacle, you slow down and the shadow catches up. The game has a really cool split screen feature where you can watch the shadow following you and smacking into obstacles. This is really cool to watch and it leads to some really tense and rewarding moments as it gets closer and closer to you. The art is beautiful as well so it actually feels good when it inevitably ends. Live & Evil 2: Live & Evil (itch.io) Zaksley – Developer and Game Designer (itch.io) Drenaya – Developer and Pixel Artist (itch.io) Thalia Music (itch.io) Live & Evil is a 2D puzzle platformer game about a robot named "Live" and their show "Evil"…which is also the word "live" spelled backwards and the the logo reflects that in a cool way. This is important to note because in this game you swap between the two characters by hitting the Q button. Live walks on top of the platforms, Evil on the bottom. Items like collectables, switches or platforms may only be visible or usable by one of the characters so you'll have to use both of them to win. This plays out in increasingly interesting ways as the game progresses…and again, you'll have to see it to believe it. Light of the World 1: Light of the World (itch.io) Nelson L: Lead Programmer (itch.io) Michael P: Lead Designer (itch.io) Light of the World is a 2D puzzle platformer with dark atmospheric and spooky levels, you are charged with rushing between beautiful beacons of light before the your enemy can attack. I

We wrap up our discussion of PagerDuty's Security Training, while Joe declares this year is already a loss, Michael can't even, and Allen says doody, err, duty. The full show notes for this episode are available at https://www.codingblocks.net/episode177. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Survey Says How awesome was Game Ja-Ja-Ja-Jamuary?! Take the survey at: https://www.codingblocks.net/episode177. News Ja Ja Ja Jamuary is complete and there are 46 new games in the world. Go play! (itch.io) Session Management Session management is the ability to identify a user over multiple requests. HTTP is stateless, so there needs to be a way to maintain state. Cookies are commonly used to store information on the client to be sent back to the server on subsequent requests. They usually contains a session token of some sort, which should be a random unique string. Do NOT store sensitive information in the cookie, such as no usernames, passwords, etc. Besides tampering, it can be difficult to revoke the cookies. Session Hijacking Session hijacking is stealing a user's session, possibly by: Guessing or stealing the session identifiers, or Taking over cookies that weren't properly locked down. Session Fixation Session fixation is when a bad actor creates a session that you will unknowingly take over, thus giving the bad actor access to the data in the user's session. This used to be more of an issue when session tokens were passed around in the URL (remember CFID and CFTOKEN?!). Always treat cookies like any other user input, don't implicitly trust it, because it can be manipulated on the client. How to Secure / Verify Sessions Add extra pieces of data to the session you can verify when requests are made. Ensure you actually created the session. Make sure it hasn't expired and ensure you set expirations for sessions. All of this just catches the easy stuff. Session ID's should be unique and random. Ensure the following when sending cookies to the client: Secure flag is set, httpOnly flag is set, and The domain is set on the cookie so it can only be used by your application. To avoid the session fixation we mentioned earlier, ALWAYS make sure to send a new session ID when privileges are elevated, i.e. a login. Always keep information stored on the server side, not on the client. Make sure you have an expiration that is set on the server side session. This should be completely independent of the cookie because the cookie values can be manipulated. When a user logs out or the session expires, ensure you fully destroy all session information. NEVER TRUST USER INPUT! Permissions Try to avoid using sudo in any shell scripts if you can. If you can't avoid it, use it with care. The the principle of least privilege, i.e. more restrictive permissions, as in, can you live with read-only perms? Revoke permissions you don't need. Create separate users for separate needs. If you need to delete files from a storage bucket, have a service account or user set up with just that permission. Same for managing compute instances. Use the least permissive approach you can as it greatly reduces risks. Other Classic Vulnerabilities Buffer overflow: This is when a piece of data is stored somewhere it shouldn't be able to access. From Wikipedia, a buffer overflow _"is an anomaly where a program, while writing data to a buffer, overruns the buffer's boundary and overwrites adjacent memory locations."_ Typically these are used to execute malicious code by putting instructions in a piece of memory that is to be executed after a previous statement completes. One malicious use of a buffer overflow is using a NOP sled (no-operation sled) to fill up the buffer with a lot of NOPs with your malicious code at the end of the ride. Apparently you can use this method to easily get a root shell – article linked in the resources Metasploit (YouTube) Path Traversal: This is when you "break out" of the web server's directory and are able to access, or serve up, content from elsewhere on the server Remember, your dependencies may also have vulnerabilities such as this. You need to run scans on your apps, code, and infrastructure. Side Channel Attacks: This is when the attacker is using information that's not necessarily part of a process to get information about that process. Examples include: Timing attack: Understanding how long certain processes take can allow you to infer information about the process. For example, multiplication takes longer than addition so you might be able to determine that there's multiplication happening. Power analysis: This is when you can actually figure out what a processor is doing by analyzing the electrical po

We're pretty sure we're almost done and we're definitely all present for the recording as we continue discussing PagerDuty's Security Training, while Allen won't fall for it, Joe takes the show to a dark place, and Michael knows obscure, um, stuff. The full show notes for this episode are available at https://www.codingblocks.net/episode176. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Survey Says For this year's Game Jam, you are ... Take the survey at: https://www.codingblocks.net/episode176. News Thanks for the reviews! iTunes: YouCanSayThisNickname Game Ja Ja Ja Jam is coming up! Just a few days away! (itch.io) XSS – Cross Site Scripting Q: What is XSS? A: XSS is injecting snippets of code onto webpages that will be viewed by others. This can allow the attacker to basically have access to everything a user does or types on a page. Consider something like a comment on a forum, or blog that allows one to save malicious code. The attacker could potentially access cookies and session information, As well as gain access to keyboard entry on the page. You can sanitize the inputs, but that's not good enough. You can't check for everything in the world. You really need to be encoding the stored information before you present it back to any users. This allows things to be displayed as they were entered, but not executed by the browser. Different languages, frameworks, libraries, etc., have their own ways of encoding information before it's rendered by the browser. Get familiar with your library's specific ways. User supplied data should ALWAYS be encoded before being rendered by the browser. ALWAYS. This goes for HTML, JS, CSS, etc. Use a library for encoding because the chances are they've been vetted. Just like we mentioned before, you still have to be diligent about using 3rd party libraries. Using a 3rd party library doesn't mean you can wash your hands of it. Content Security Policy (CSP) is another way to handle this. (Wikipedia) OWASP considers XSS a type of Injection attack in 2021. CSRF – Cross Site Request Forgery Q: What is CSRF? A: CSRF is tricking someone into doing something they didn't want to do, or didn't know they were doing. A couple of examples were given: For example, set the img src to the logout for the site so that when someone visits the page, they're automatically logged out. Just imagine if the image source pointed to something a little more nefarious. Another example is a button that tricked you into performing an action such as an account deletion on another site. Can be done using a form post and a simple button click. How do you avoid this? Synchronizer token: This is a hidden field on every user submittable form on a site that has a value that's private to the user's session. These tokens should be cryptographically strong random values so they can never be guessed or reverse engineered. These tokens should never be shared with anyone else. When the form is submitted, the token is validated against the user's session token, and if it matches, go ahead with the action, otherwise abort. Again, there are a number of frameworks and libraries out there that have anti-forgery built in. Check with your specific documentation. They go on to say that anything that is not a READ operation should have CSRF tokens. NEVER use GET requests for state changing operations! PagerDuty had a funny mention about an administrative site that included links to delete rows from the database using GET requests. However, as the browser pre-fetched the links, it deleted the database. OWASP dropped CSRF from the Top 10 in 2017 because the statistical data didn't rank it highly enough to make the list. Click-jacking Q: What is click-jacking? A: Click-jacking is when you are fooled into clicking on something you didn't intend to. For example, rendering a page over the top of an iframe, and anything that was clicked on that top page (that seemed innocent) would actually make the click happen on the iframe'd page, like clicking a Buy it Now button. Another example is moving a window as soon as you click causing you to click on something you didn't intend to click. The best way to prevent click-jacking is to lock down what an iframe can load using the HTTP header X-FRAME-OPTIONS, set to either SAMEORIGIN or DENY. (developer.mozilla.org) Account Enumeration Q: What is account enumeration? A: Account enumeration is when an attacker attempts to extract users or information from a website. Failed logins that take longer for one user than another may indicate that the one that took longer was a real user, maybe because it takes longer as it tries to hash the password. Similar type of thing could happen if c

We continue our discussion of PagerDuty's Security Training presentation while Michael buys a vowel, Joe has some buffer, and Allen hits everything he doesn't aim for. The full show notes for this episode are available at https://www.codingblocks.net/episode175. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Shortcut – Project management has never been easier. Check out how Shortcut is project management without all the management. Survey Says Do stick with your New Year's resolutions? Take the survey at: https://www.codingblocks.net/episode175. News Thanks for the reviews! iTunes: aodiogo Game Ja-Ja-Ja-Jamuary is coming up, sign up is open now! (itch.io) Encryption OWASP has the more generic "Cryptographic Failures" at #2, up from #3 in 2017. PagerDuty defines encryption as encoding information in such a way that only authorized readers can access it. Note that this is an informal definition that speaks to the most common use of the word. Encryption is really, really difficult to get right. There are people that spend their whole lives thinking about encryption, and breaking encryption. You may think you're a genius by coming up with a non-standard implementation, but unfortunately the attackers are really sophisticated and this strategy has shown to fail over and over. There are different types of encryption: Symmetric/Asymmetric – refers to whether the keys for reading and writing the encrypted data are the same. Block Cipher – Lets you encrypt and decrypt the data in whole chunks. You need to have an entire block to encrypt or decrypt the whole block at once. Public/Private Key – A kind of asymmetric encryption intended for situations where you want groups to be able to share one of the keys. For example, you can publish a public PGP key and then people can use that to send you a message. You keep the private key private, so you're the only entity that can read the message. Stream Cipher – Encode "on the fly", think about HTTPS, great for streaming. You can start reading before you have the entire message. Great for situations where performance is important, or you might miss data. Encryption in Transit Also known by other names such as data in motion. Designed to protect against entities that can snoop (or manipulate!) our communications. You can do this with HTTPS, TLS, IPsec. Perfect Forward Secrecy is the key to protecting past communications, by generating a new key for a single session so that compromised keys only affect the specific session they were used for. From Wikipedia "In cryptography, forward secrecy (FS), also known as perfect forward secrecy (PFS), is a feature of specific key agreement protocols that gives assurances that session keys will not be compromised even if long-term secrets used in the session key exchange are compromised." (Wikipedia) Encryption at Rest Simply means that data is encrypted where it's stored. An example of this is full disk encryption on laptops and desktops. The entire drive is encrypted so if someone were to steal the drive, it'd essentially be useless without the keys to decrypt the data on the drive. For PagerDuty, and many other companies, the most important information to protect is customer data, just as important as your own passwords. PagerDuty's data classifications: General data – This is anything available to the public. Business data – Includes operating data for the business, such as payroll, employee info, etc. This type of data is expected to be encrypted in transit and at rest. Customer data – This is data provided to the company by the customer and is expected to be encrypted in transit and at rest. Customer data includes controls such as authentication, access control, storage, auditing, encryption, and destruction. Business data has similar controls except without the auditing. PagerDuty called out when using cloud systems, make sure you're enabling the encryption on the various services, like S3, GCS, Blob storage, etc. They mentioned it's just a checkbox, but in reality you're probably using scripts, templates, etc. So make sure you know the configurations to include to enable encryption. Another interesting thing they do at PagerDuty: they get alerted when a resource is created without encryption enabled. What about third parties you use? Should they encrypt as well? YES!!! Perform vendor risk assessments prior to using the vendor. If they don't pass the security assessment, use a different vendor. Secret Management Q. What is it? A. Protecting and auditing access to secrets. Auditing so that you can see when someone is using your secrets that shouldn't, as well as keep track of systems that should and are using secrets. Hashicorp Vault has a great video to learn about the challenges of managing secrets. (YouTube) What are secrets? Secrets are sensit

We're taking our time as we discuss PagerDuty's Security Training presentations and what it means to "roll the pepper" while Michael is embarrassed in front of the whole Internet, Franklin Allen Underwood is on a full name basis, and don't talk to Joe about corn. The full show notes for this episode are available at https://www.codingblocks.net/episode174. Sponsors Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says How much personal time off do you take on average each year? Take the survey at: https://www.codingblocks.net/episode174. News Thanks for the reviews! iTunes: Goofiw, totalwhine, Kpbmx, Viv-or-vyv Game Ja-Ja-Ja-Jamuary is coming up, sign up is open now! (itch.io) Question about unit tests, is extra code that's only used by unit tests acceptable? Huge congrats to Jamie Taylor for making Microsoft MVP! Check out some of his podcasts: The .NET Core Podcast (DotNetCore.show) Tabs and Spaces Podcast (TabsAndSpaces.io) RJJ Software Ltd (RJJ-Software.co.uk) Why this topic? It's good to learn about the common security vulnerabilities when developing software! What they are, how they are exploited, and how they are prevented. WebGoat is a website you can run w/ known vulnerabilities. It is designed for you to poke at and find problems with to help you learn how attackers can take advantage of problems. (OWASP.org) "But the framework takes care of that for me" Don't be that person! Recent vulnerability with Grafana, CVE-2021-43798. (SOCPrime.com) The Log4j fiasco begins. (CNN) You can't always wait for a vulnerability patch to be released. You may need to patch one yourself. Basically, even if you're using a framework, it doesn't mean you can be naïve to everything about it. You shouldn't use the excuse "It's just for a hackathon" or "It's a proof of concept." This can include things like disabling firewalls, etc. Don't put things on a public repo, as you might accidentally share company secrets, intellectual property, etc. Open sourcing may be an option later, but it should be looked through first. NEVER use customer data when doing hackathons or proofs of concepts. Too many things can go wrong if it leaks out. Maybe a better rule of thumb would be to never use customer data for any type of development. Instead, always use fake data. The slides had an interesting story that was redacted: there was a software vulnerability that was discovered that existed due to a missing check-in of code, i.e. everything was functioning perfectly fine, and there was an effort already to plug a hole in the code, but it just never made it into the repo. Nearly impossible to detect by automated tools. Vulnerability #1 – SQL Injection OWASP has more a generic "Injection" as the #3 position, down from #1 in 2017. An example is manipulating a query at runtime with user provided input. This typically implies that strings are patched into a query directly, i.e. WHERE password = '$providedPassword'. Can be attacked by doing something like providedPassword = ' OR 1=1 --. Which effectively turns into WHERE password = '' OR 1=1 --. This is the basis for the tale of little Bobby Tables (xkcd). Users should NEVER be able to directly impact the runnable query. They can provide values, and those should be parameterized, or validated first. The real problem is that people with SQL knowledge can string multiple lines of SQL together to manipulate the original query in some scary ways. Blind Injection Boolean Boolean based attacks take time but the scripting throws errors if script results are true. Example they provided is "If the first database starts with an A, throw", "If the first database starts with a B, throw", etc. Time Based Uses the Boolean based attack, but puts them on a delay so they won't be as easily detected. So you can just regular expressions for keywords and escape quotes right?! Ummm … no! There's just too many combinations of things you'd need to know as well as weird characters and tricks you couldn't even be aware of, double or triple encoding, exceptions, etc. It's surprisingly tricky. For example, how would you allow single quotes? Replace them all with \'? Unless there's already a \ in front of it, but what if it's \? You can theoretically overcome all of these problems … but … why? Why not just do it the right way? The answer is to use prepared statements and/or parameterized queries. The difference between a prepared statement and what was mentioned above is the user's input doesn't directly modify a query, rather the input is substituted in the appropriate place. Side benefit is prepared statements often execute quicker than manually constructed SQL queries. Vulnerability #2 – Storing Passwords OWASP has the more generic "Cryptographic Failures" at #2, up from #3 in 2017. Never store passwords in plain text! I've heard hashing is good, right? Kind of, until you hear that there's this thing called rainbow tables. Rainbow tables are basicall

With Game Ja-Ja-Ja-Jamuary coming up, we discuss what makes a game engine, while Michael's impersonation is spot-on, Allen may really just be Michael, and Joe already has the title of his next podcast show at the ready. The full show notes for this episode are available at https://www.codingblocks.net/episode173. Sponsors Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says What's your container management of choice? Take the survey at: https://www.codingblocks.net/episode173. Game Jam '22 is coming up in Ja-Ja-Ja-Jamuary News Thanks for the reviews! Podchaser: Jamie Introcaso Game Ja-Ja-Ja-Jamuary is coming up, sign up is open now! (itch.io) What is a Game Engine? What's a… Library, Framework, Toolkit, … Engine? Want to see terrible explanations of a thing? Google "framework vs engine". Other types of engines: storage engine, rendering engine, for example. Q: Why do people use game engines? Well, they reduce costs, complexities, and time-to-market. Consistency! Q: Why do so many AAA games create their own custom engines? Common Features of Game Engines 2D/3D rendering engine Basic shapes (planes, spheres, lines), Particles, Shaders, Masking/Culling, Progressive enhancement (either by distance or by some other means) Physics engine Collision detection, Mass, Gravity, Torque, Force, Friction, Springiness, Fluid Dynamics, Wind Sound Multiple sounds at once, looping, spatial settings, etc. Scripting AI Networking Ever thought about how this works? Peer to peer, dedicated servers? Streaming Streaming assets, as in, the player hasn't installed your game. Scene Management Cinematics UI Often engines also include development tools to making working with these various systems easier … like an IDE. Some Really Cool Things About Unity Asset Store and Package Management, ProBuilder (Unity), Terrain, Animation Manager, Ad Systems and Analytics, Target multiple platforms: Xbox, Windows, Linux, Android, MacOS, iOS, PSX, Switch, etc. About the Industry How big is the industry? $150B in 2019, estimated $250B for 2025 (TechJury.net) How does it compare to other industries? Movies are $41B, Books are $25B, Netflix is $7B … that's about half of Nintendo, HBO is $2B How many companies and employees? 2,457 companies and 220k jobs … in 2015! (Quora) What's the breakdown on sales? Mobile $90B, PC $35B, console $49B (NewZoo.com) How many games released in a year? About 10k a year, on Steam (NewZoo.com) How long does it take? 1 – 10 years? The 10 Best Games Made By Just One Person (TheGamer.com) Commentary on Popular Game Engines Unity Publish for 20+ platforms 50% of games are made with Unity (GameDeveloper.com) List of Unity games (Wikpedia) Pricing range: Free to $2,400. You can use the free plan if revenue or funding is less than $100k! Program in C# Great learning resources (learn.unity.com) Unreal Many AAA games built with Unreal. Basically think of the top 10 biggest, most beautiful, AAA games; those are probably all Unreal or custom (RAGE, Frostbyte, Last of Us) Pricing: from free to "call for pricing", 5% royalty after $1mm List of Unreal Engine games (Wikipedia) Originally came out of the Unreal series of games, and a new one is coming out soon! (Epic Games) Program in C++ Godot Open Source Growing in popularity You can program in a variety of languages, officially C/C++ and GDScript but there are other bindings (Wikipedia) Custom Game Engines: GameMaker RPG Maker Specialized: Frostbyte, Cryo, etc. Korge libGDX Final Question Game Jam sign-up is live … what are you thinking for technology and mechanics? Allen: VR / Escape Room Michael: Something web based Joe: Going 3D, wanting to focus on level design and physics this time Resources We Like Splitgate, where Halo meets Portal (Splitgate.com) Mythic Quest (Apple) What's wrong with TikTok? (Washington Post) Blood, Sweat, and Pixels by Jason Schreier (Amazon) Tip of the Week ProBuilder is a free tool available in Unity that is great for making polygons and great for mocking out levels or building ramps. The coolest part is the way it works, giving you a bunch of tools that you do things like create vertices, edges, surfaces, extrude, intrude, mirror, etc. You have to add it via the package manager but it's worth it for simple games and prototypes. (Unity) Great blog on processing billions of events in real time at Twitter, thanks Mikerg! (blog.twitter.com) forEachIndexed is a nice Kotlin method for iterating through items in a collection, with an index for positional based computations (ozenero.com) How can you log out of Netflix on Samsung Smart TVs? Ever heard of the Konami code? Press Up Up Down Down Left Right Left Right Up Up Up Up (help.netflix.com)

We wrap up the discussion on partitioning from our collective favorite book, Designing Data-Intensive Applications, while Allen is properly substituted, Michael can't stop thinking about Kafka, and Joe doesn't live in the real sunshine state. The full show notes for this episode are available at https://www.codingblocks.net/episode172. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says How many different data storage technologies do you use for your day job? Take the survey at: https://www.codingblocks.net/episode172. News Game Ja Ja Ja Jam is coming up, sign up is open now! (itch.io) Joe finished the Create With Code Unity Course (learn.unity.com) New MacBook Pro Review, notch be darned! Last Episode … Best book evar! In our previous episode, we talked about data partitioning, which refers to how you can split up data sets, which is great when you have data that's too big to fit on a single machine, or you have special performance requirements. We talked about two different partitioning strategies: key ranges which works best with homogenous, well-balanced keys, and also hashing which provides a much more even distribution that helps avoid hot-spotting. This episode we're continuing the discussion, talking about secondary indexes, rebalancing, and routing. Partitioning, Part Deux Partitioning and Secondary Indexes Last episode we talked about key range partitioning and key hashing to deterministically figure out where data should land based on a key that we chose to represent our data. But what happens if you need to look up data by something other than the key? For example, imagine you are partitioning credit card transactions by a hash of the date. If I tell you I need the data for last week, then it's easy, we hash the date for each day in the week. But what happens if I ask you to count all the transactions for a particular credit card? You have to look at every single record. in every single partition! Secondary Indexes refer to metadata about our data that help keep track of where our data is. In our example about counting a user's transactions in a data set that is partitioned by date, we could keep a separate data structure that keeps track of which partitions each user has data in. We could even easily keep a count of those transactions so that you could return the count of a user's transaction solely from the information in the secondary index. Secondary indexes are complicated. HBase and Voldemort avoid them, while search engines like Elasticsearch specialize in them. There are two main strategies for secondary indexes: Document based partitioning, and Term based partitioning. Document Based Partitioning Remember our example dataset of transactions partitioned by date? Imagine now that each partition keeps a list of each user it holds, as well as the key for the transaction. When you query for users, you simply ask each partition for the keys for that user. Counting is easy and if you need the full record, then you know where the key is in the partition. Assuming you store the data in the partition ordered by key, it's a quick lookup. Remember Big O? Finding an item in an ordered list is O(log n). Which is much, much, much faster than looking at every row in every partition, which is O(n). We have to take a small performance hit when we insert (i.e. write) new items to the index, but if it's something you query often it's worth it. Note that each partition only cares about the data they store, they don't know anything about what the other partitions have. Because of that, we call it a local index. Another name for this type of approach is "scatter/gather": the data is scattered as you write it and gathered up again when you need it. This is especially nice when you have data retention rules. If you partition by date and only keep 90 days worth of data, you can simply drop old partitions and the secondary index data goes with them. Term Based Partitioning If we are willing to make our writes a little more complicated in exchange for more efficient reads, we can step up to term based partitioning. One problem with having each partition keeping track of their local data is you have to query all the partitions. What if the data's only on one partition? Our client still needs to wait to hear back from all partitions before returning the result. What if we pulled the index data away from the partitions to a separate system? Now we check this secondary index to figure out the keys, which we can then go look up on the appropriate indices. We can go one step further and partition this secondary index so it scales better. For example, userId 1-100 might be on one, 101-200 on another, etc. The benefit of term based partitioning is you get more efficient reads, the downside is that you are now writing to multiple

We crack open our favorite book again, Designing Data-Intensive Applications by Martin Kleppmann, while Joe sounds different, Michael comes to a sad realization, and Allen also engages "no take backs". The full show notes for this episode are available at https://www.codingblocks.net/episode171. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says Have you ever had to partition your data? Take the survey at: https://www.codingblocks.net/episode171. News Thank you for the review! iTunes: Wohim321 Best book evar! The Whys and Hows of Partitioning Data Partitioning is known by different names in different databases: Shard in MongoDB, ElasticSearch, SolrCloud, Region in HBase, Tablet in BigTable, vNode in Cassandra and Riak, vBucket in CouchBase. What are they? In contrast to the replication we discussed, partitioning is spreading the data out over multiple storage sections either because all the data won't fit on a single storage mechanism or because you need faster read capabilities. Typically data records are stored on exactly one partition (record, row, document). Each partition is a mini database of its own. Why partition? Scalability Different partitions can be put on completely separate nodes. This means that large data sets can be spread across many disks, and queries can be distributed across many processors. Each node executes queries for its own partition. For more processing power, spread the data across more nodes. Examples of these are NoSQL databases and Hadoop data warehouses. These can be set up for either analytic or transactional workloads. While partitioning means that records belong to a single partition, those partitions can still be replicated to other nodes for fault tolerance. A single node may store more than one partition. Nodes can also be a leader for some partitions and a follower for others. They noted that the partitioning scheme is mostly independent of the replication used. Figure 6-1 in the book shows this leader / follower scheme for partitioning among multiple nodes. The goal in partitioning is to try and spread the data around as evenly as possible. If data is unevenly spread, it is called skewed. Skewed partitioning is less effective as some nodes work harder while others are sitting more idle. Partitions with higher than normal loads are called hot spots. One way to avoid hot-spotting is putting data on random nodes. Problem with this is you won't know where the data lives when running queries, so you have to query every node, which is not good. Partitioning by Key Range Assign a continuous range of keys on a particular partition. Just like old encyclopedias or even the rows of shelves in a library. By doing this type of partitioning, your database can know which node to query for a specific key. Partition boundaries can be determined manually or they can be determined by the database system. Automatic partition is done by BigTable, HBase, RethinkDB, and MongoDB. The partitions can keep the keys sorted which allow for fast lookups. Think back to the SSTables and LSM Trees. They used the example of using timestamps as the key for sensor data – ie YY-MM-DD-HH-MM. The problem with this is this can lead to hot-spotting on writes. All other nodes are sitting around doing nothing while the node with today's partition is busy. One way they mentioned you could avoid this hot-spotting is maybe you prefix the timestamp with the name of the sensor, which could balance writing to different nodes. The downside to this is now if you wanted the data for all the sensors you'd have to issue separate range queries for each sensor to get that time range of data. Some databases attempt to mitigate the downsides of hot-spotting. For example, Elastic has the ability specify an index lifecycle that can move data around based on the key. Take the sensor example for instance, new data comes in but the data is rarely old. Depending on the query patterns it may make sense to move older data to slower machines to save money as time marches on. Elastic uses a temperature analogy allowing you to specify policies for data that is hot, warm, cold, or frozen. Partitioning by Hash of the Key To avoid the skew and hot-spot issues, many data stores use the key hashing for distributing the data. A good hashing function will take data and make it evenly distributed. Hashing algorithms for the sake of distribution do not need to be cryptographically strong. Mongo uses MD5. Cassandra uses Murmur3. Voldemort uses Fowler-Noll-Vo. Another interesting thing is not all programming languages have suitable hashing algorithms. Why? Because the hash will change for the same key. Java's object.hashCode() and Ruby's Object#hash were called out. Partition boundaries can be set evenly or done pseudo-randomly, aka consistent hashing

The Mathemachicken strikes again for this year's shopping spree, while Allen just realized he was under a rock, Joe engages "no take backs", and Michael ups his decor game. The full show notes for this episode are available at https://www.codingblocks.net/episode170. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says What's your favorite feature on the new MacBook Pro? Take the survey at: https://www.codingblocks.net/episode170. News Thank you to everyone that left a review! iTunes: BoldAsLove88 Audible: Tammy Joe's List Price Description "Fun" Answers $3,499.95 Jura x8 (Williams and Sonoma) $3,499.00 2021 Macbook Pro – 16″ Screen, M1 Max, 32GB RAM, 1TB drive (Apple) Robotics $359.99 Lego Mindstorms (Amazon) $149.99 Sphero BOLT (Amazon) Entertainment $499.99 Xbox Series X (Microsoft) $180 Game Pass (Microsoft) $179.00 Play Date (Website) Health $929.99 Trek Dual Sport 3 (Trek) $179.95 Fitbit Charge 5 (Amazon) $921.22 Dorito Dust Supplies (Recipe) Levelling Up $199 / year Educative.io (Website) $159 / year LeetCode Subscription (Website) $99 ACM Subscription (Sign Up) Allen's List Description Price Honorable mention: Steam Deck (Steam) $399.00 Honorable mention: Microsoft Surface Laptop Studio 14.4″ (Amazon) $2,700.00 LG 48″ C1 OLED TV (Amazon) $1,297.00 Honorable mention: Aorus 48″ OLED Gaming Monitor (Newegg) $1,500.00 HTC Vive Pro 2 (Amazon) $799.00 Valve Index Controllers (Steam/Valve) $279.00 Kinesis Advantage 2 (Amazon) $339.00 Corsair MP600 NVME PCIE x4 2TB (Amazon) $240.00 Arduino Ultimate Starter Kit (Amazon) $63.00 Michael's List Price Description My smart home can beat up your smart home $14.99 Kasa Smart Light Switch HS200 (Amazon) $16.99 Kasa Smart Dimmer Switch HS220 (Amazon) $26.99 Kasa Smart Plug Mini 15A 4-Pack EP10P4 (Amazon) $17.99 Kasa Outdoor Smart Plug with 2 Sockets EP40 (Amazon) For my health $529.00 Apple Watch Series 7 GPS + Cellular (Amazon) Need moar power! $34.00 Apple MagSafe Charger (Amazon) $12.99 elago W6 Apple Watch Stand (Amazon) $10.99 Honorable mention: elago W3 Apple Watch Stand (Amazon) $29.00 Honorable mention: Apple Watch Magnetic Charging Cable (0.3m) (Amazon) When I lose my stuff $98.99 Apple AirTag 4 Pack (Amazon) $10.99 Protective Case for Airtags (Amazon) $14.88 Honorable mention: Air Tags Airtag Holder for Dogs/Cat Pet Collar (Amazon) I need to get some work done $180.00 Code V3 104-Key Illuminated Mechanical Keyboard (Amazon) $169.00 Honorable mention: Das Keyboard 4 Professional Wired Mechanical Keyboard (Amazon) $280.00 Honorable mention: Drop SHIFT Mechanical Keyboard (Amazon) $240.00 Honorable mention: Drop CTRL Mechanical Keyboard (Amazon) If you insist on an ergo keyboard $199.00 Honorable mention: KINESIS GAMING Freestyle Edge RGB Split Mechanical Keyboard (Amazon) Turns out, keycaps matter $29.99 Honorable mention: Razer Doubleshot PBT Keycap Upgrade Set (Amazon) $24.99 Honorable mention: HyperX Pudding Keycaps (Amazon) Things I need to buy again $19.99 HyperX Wrist Rest (Amazon) $28.99 Honorable mention: Glorious Gaming Wrist Pad/Rest (Amazon) $34.99 Honorable mention: Razer Ergonomic Wrist Rest Pro (Amazon) When things go wrong $69.99 iFixit Pro Tech Toolkit (Amazon) $64.99 Honorable mention: iFixit Manta Driver Kit (Amazon) For all your calling needs $599.00 Rode RODECaster Pro Podcast Production Studio (Amazon) $549.99 Honorable mention: Zoom PodTrak P8 Podcast Recorder (Amazon) $12.95 On-Stage DS7100B Desktop Microphone Stand (Amazon) $199.99 Elgato Ring Light (Amazon) $159.99 Elgato HD60 S+ Capture Card (Amazon) Music to your ears $148.49 Kali Audio LP-6 Studio Monitor (Amazon) $189.00 Honorable mention: KRK RP5 Rokit G4 Studio Monitor (Amazon) $379.99 Honorable mention: Yamaha HS7I Studio Monitor (Amazon) $199.99 Honorable mention: ADAM Audio T5V Two-Way Active Nearfield Monitor (Amazon) $155.00 Honorable mention: JBL Professional Studio Monitor (305PMKII) (Amazon) $599.00 Kali Audio WS-12 12 inch Powered Subwoofer (Sweetwater) $65.00 Palmer Audio Interface (PMONICON) (Amazon) $169.99 Honorable mention: Focusrite Scarlett 2i2 (3rd Gen) USB Audio Interface (Amazon) For the decor $34.99 Dumb and Dumber Canvas (Amazon) $34.99 Honorable mention: The Big Lebowski Canvas (Amazon) $34.99 Honorable mention: Pulp Fiction Canvas (Amazon) $34.99 Honorable mention: Friday Canvas (Amazon) $34.99 Honorable mention: Jurassic Park (Amazon) $34.99 Honorable mention: Bridesmaids Canvas (Amazon) $34.99 Honorable mention: There's Something About Mary (Amazon) Resources We Like Security Now 834, Life: Hanging By A Pin (Twit.tv) Buyer Beware: Crucial Swaps P2 SSD's TLC NAND for Slower Chips (ExtremeTech.com) Samsung Is the Latest SSD Manufacturer Caught Cheating Its Customers (ExtremeTech.com) Tip of the Week VS Code … in the browser … just … there? Not all extensions work, but a lot do! (VSCode.dev) Skaffold is a tool you can use to build and maintain Kubernetes environments that we

We discuss the pros and cons of speaking at conferences and similar events, while Joe makes a verbal typo, Michael has turned over a new leaf, and Allen didn't actually click the link. The full show notes for this episode are available at https://www.codingblocks.net/episode169. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says How likely are you to give a presentation? Take the survey at: https://www.codingblocks.net/episode169. News The Kinesis Gaming Freestyle Edge RGB Split Mechanical Keyboard might be the current favorite. Thank you to everyone that left a review! iTunes: dahol1337, Pesri How long does it take to get the Moonlander? (ZSA.io) Is the Kinesis Gaming Freestyle the current favorite? (Amazon) Atlanta Code Camp was fantastic, see you again next year! (atlantacodecamp.com) What kind of speaking are we talking about? Conferences Meetups Does YouTube/Twitch count as tech presentations? There are some similarities! Streaming has the engagement, but generally isn't as rehearsed. Published videos are closer to the format but you have to make some assumptions about your audience and can get creative with the editing. Why do people speak? Can help you build an audience Establish credibility Check out Azure Steve! Promotional opportunities Networking Free travel/conferences Great way to learn something Become a better communicator Is it fun? Who speaks at conferences? People speak at conferences for different reasons Couple different archetypes of speakers: Sponsored: the speakers are on the job, promoting their company and products Practitioners: Talks from people in the trenches, usually more technical and focused on specific results or challenges Idea people: People who have a strong belief in something that is controversial, may have an axe to grind or an idea that's percolating into a product Professionals: Some companies encourage speakers to bolster the company reputation, promotions and job descriptions might require this How do you put together a talk? How do you pick a talk? Know who is selecting talks, go niche for larger conferences if you don't have large credentials/backing Sometimes conferences will encourage "tracks" certain themes for topics What are some talks you like? What do they do differently? Do you aim for something you know, or want to know? How do you write your talks? How do you practice for a talk? Differences between digital and physical presentations? How long does it take you? Where can you find places to speak? Is this the right question? What does this tell you about your motivation? Meet new people who share your interests through online and in-person events. (Meetup) Find your next tech conference (Confs.Tech) Google for events in your area! Final Questions Is it worth the time and anxiety? What do you want out of talks? What are some alternatives? Blogging Videos Open Source Participating in communities Resources Is Speaking At A Conference Really Worth Your Time? (Cleverism.com) We're 93% certain that Burke Holland gave a great talk about a dishwasher and Vue.js. (Twitter) Monitor you Netlify sites with Datadog (Datadog) Netlify (docs.datadoghq.com) Risk Astley – Never Gonna Give You Up (Official Music Video) (YouTube) Simple Minds – Don't You (Forget About Me) (YouTube) Foo Fighters With Rick Astley – Never Gonna Give You Up – London O2 Arena 19 September 2017 (YouTube) Tip of the Week Next Meeting is a free app for macOS that keeps a status message up in the top right of your toolbar so you know when your next meeting is. It does other stuff too, like making it easier to join meetings and see your day's events but … the status is enough to warrant the install. Thanks MadVikingGod! (Mac App Store) How do I disable "link preview" in iOS safari? (Stack Exchange) Here is your new favorite YouTube channel, Rick Beato is a music professional who makes great videos about the music you love, focusing on what makes the songs and artists special. (YouTube) Hot is a free app for macOS that shows you the temperate of your MacBook Pro … and the percentage of CPU you're limited to because of the heat! Laptop feels slow? Maybe it's too hot! (GitHub, XS-Labs) What is the meaning of $? in a shell script? (Stack Exchange) Did you know…You can install brew on Linux? That's right, the popular macOS packaging software is available on your favorite distro. (docs.brew.sh, brew.sh)

Joe goes full shock jock, but only for a moment. Allen loses the "Most Tips In A Single Episode: 2021" award, and Michael didn't get the the invite notification in this week's episode. The full show notes for this episode are available at https://www.codingblocks.net/episode168. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Shortcut - Project management has never been easier. Check out how Shortcut (formerly known as Clubhouse) is project management without all the management. Survey Says Well...no survey this week, but this is where it would be! News This book has a whole chapter on transactions in distributed systems Thank you to everyone that left a review! "Podchaser: alexi*********, Nicholas G Larsen, Kubernutties, iTunes: Kidboyadde, Metalgeeksteve, cametumbling, jstef16, Fr1ek Audible: Anonymous (we are like your mother - go clean your room and learn Docker) Atlanta Code Camp is right around the corner on October 9th. Stop by the CB booth and say hi! (AtlantaCodeCamp.com) Maintaining data consistency Each service should have its own data store What about transactions? microservices.io suggests the saga pattern (website) A sequence of local transactions must occur Order service saves to its data store, then sends a message that it is done Customer service attempts to save to its data store…if it succeeds, the transaction is done. If it fails, it sends a message stating so, and then the Order service would need to run another update to undo the previous action Sound complicated? It is…a bit, you can't rely on a standard 2 Phase Commit at the database level to ensure an atomic transaction Ways to achieve this - choreography or orchestration Choreography Saga - The Order Service receives the POST /orders request and creates an Order in a PENDING state - It then emits an Order Created event - The Customer Service's event handler attempts to reserve credit - It then emits an event indicating the outcome - The OrderService's event handler either approves or rejects the Order Each service's local transaction sends a domain event that triggers another service's local transaction To sum things up, each service knows where to listen for work it should do, and it knows where to publishes the results of it's work. It's up to the designers of the system to set things up such that the right things happened What's good about this approach? "The code I wrote sux. The code I'm writing is cool. The code I'm going to write rocks!" Thanks for the paraphrase Mike! Orchestration Saga - The Order Service receives the POST /orders request and creates the Create Order saga orchestrator - The saga orchestrator creates an Order in the PENDING state - It then sends a Reserve Credit command to the Customer Service - The Customer Service attempts to reserve credit - It then sends back a reply message indicating the outcome - The saga orchestrator either approves or rejects the Order There is an orchestrator object that tells each service what transaction to run The difference between Orchestration and Choreography is that the orchestration approach has a "brain" - an object that centralizes the logic and can make more advanced changes These patterns allow you to maintain data consistency across multiple services The programming is quite a bit more complicated - you have to write rollback / undo transactions - can't rely on ACID types of transactions we've come to rely on in databases Other issues to understand The service must update the local transaction AND publish the message / event The client that initiates the saga (asynchronously) needs to be able to determine the outcome The service sends back a response when the saga completes The service sends back a response when the order id is created and then polls for the status of the overall saga The service sends back a response when the order id is created and then submits an event via a webhook or similar when the saga completes When would you use Orchestration vs Choreography for transactions across Microservices? Friend of the show @swyx works for Temporal, a company that does microservice orchestration as a service, https://temporal.io/ Tips for writing Great Microservices Fantastic article on how to keep microservices loosely coupled https://www.capitalone.com/tech/software-engineering/how-to-avoid-loose-coupled-microservices/ Mentions using separate data storage / dbs per service Can't hide implementation from other services if they can see what's happening behind the scenes - leads to tight coupling Share as little code as possible Tempting to share things like customer objects, but doing so tightly couples the various microservices Better to nearly duplicate those objects in a NON-shared way - that way the services can change independently Avoid synchronous communication where possible This means relying on message brokers, polling, callbacks, etc Don't use shared test environments / appliance

Some things just require discussion, such as Docker's new licensing, while Joe is full of it, Allen *WILL* fault them, and Michael goes on the record. The full show notes for this episode are available at https://www.codingblocks.net/episode167. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Shortcut – Project management has never been easier. Check out how Shortcut (formerly known as Clubhouse) is project management without all the management. Survey Says How do you get prefer to get on the network? Take the survey at: https://www.codingblocks.net/episode167. News Thank you to everyone that left a review! iTunes: Badri Ravi Audible: Dysrhythmic, Brent Atlanta Code Camp is right around the corner on October 9th. Stop by the CB booth and say hi! (AtlantaCodeCamp.com) Docker Announcement Docker recently announced big changes to the licensing terms and pricing for their product subscriptions. These changes would mean some companies having to pay a lot more to continue using Docker like they do today. So…what will will happen? Will Docker start raking in the dough or will companies abandon Docker? Resources Docker is Updating and Extending Our Product Subscriptions (Docker) Minkube documentation (Thanks MadVikingGod! From the Tips n' Tools channel in Slack.) Open Container Initiative, an open governance structure for the purpose of creating open industry standards around container formats and runtimes. (opencontainers.org) Podman, a daemonless container engine for developing, managing, and running OCI containers. (podman.io) Getting Started with K9s (YouTube) How valuable is education? How do you decide when it's time to go back to school or get a certification? What are the determining factors for making those decisions? Full-Stack Road Map What's on your roadmap? We found a full-stack roadmap on dev.to and it's got some interesting differences from other roadmaps we've seen or the roadmaps we've made. What are those differences? Resources Full Stack Developer's Roadmap (dev.to) Bonus Tip: You can find the top dev.to articles for certain time periods like: https://dev.to/top/year. Works for week, month, and day, too. Where does your business logic go? Business logic should be in a service, not in a model … or should it? What's the right way to do this? Is there a right way? Resources How accurate is "Business logic should be in a service, not in a model"? (Stack Exchange) AnemicDomainModel (MartinFowler.com) Are the M1/M1X chips a good idea for devs? Last year's MacBook Pros introduced new M1 processors based on a RISC architecture. Now Apple is rolling out the rest of the line. What does this mean for devs? Is there a chance you will regret purchasing one of these laptops? Resources Apple Silicon M1: A Developer's Perspective (steipete.com) Tip of the Week Hit . (i.e. the period key) in GitHub to bring up an online VS Code editor while you are logged in. Thanks Morten Olsrud! (blog.yogeshchavan.dev) Shoutout to Coder, cloud-powered development environments that feel local. (coder.com) The podcast that puts together the the "perfect album" for the topic du jour: The Perfect Album Side Podcast (iTunes, Spotify, Google Podcasts) Bon Jovi – Livin' On A Prayer / Wanted Dead Or Alive (Los Angeles 1989) (YouTube) Docker's system prune command now includes a filter option to easily get rid of older docker resources. (docs.docker.com) Example: docker system prune --filter="until=72h" The GitHub CLI makes it easy to create PR by autofilling information, as well as pushing your branch to origin: Example: gh pr create --fill (cli.github.com) Apache jclouds is an open-source multi-cloud toolkit that abstracts the details of your cloud provider away so you can focus on your code and still support multiple providers. (jclouds.apache.org)

We step away from our microservices deployments to meet around the water cooler and discuss the things on our minds, while Joe is playing Frogger IRL, Allen "Eeyores" his way to victory, and Michael has some words about his keyvoard, er, kryboard, leybaord, ugh, k-e-y-b-o-a-r-d! The full show notes for this episode are available at https://www.codingblocks.net/episode166. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Clubhouse – Project management has never been easier. Check out how Clubhouse (soon to be Shortcut) is project management without all the management. Survey Says Do you find that you're more productive ... Take the survey at: https://www.codingblocks.net/episode166 News The threats worked and we got new reviews! Thank you to everyone that left a review: iTunes: ArcadeyGamey, Mc'Philly C. Steak, joby_h Audible: Jake Tucker Atlanta Code Camp is right around the corner on October 9th. Stop by the CB booth and say hi! (AtlantaCodeCamp.com) Water Cooler Gossip > Office Memos Are you interested in competitive programming? Michael gives a short term use review of his Moonlander. Spring makes Java better. Resources We Like CoRecursive episode 65: From Competitive Programming to APL With Conor Hoekstra (corecursive.com) Competitive Programming – A Complete Guide (GeeksForGeeks.org) Algorithms (GeeksForGeeks.org) Get started solving problems on Code Chef (CodeChef.com) Data Structures and Algorithms (CodeChef.com) Introduction to Dynamic Programming 1 (HackerEarth.com) Enhance your skills, expand your knowledge, and prepare for technical interviews with LeetCode. (LeetCode.com) Getting started with Competitive Programming – Build your algorithm skills (dev.to) ZSA Moonlander (zsa.io) Spring Framework Documentation (docs.spring.io) Spring Expression Language (SpEL) (docs.spring.io) RethinkDB, the open-source database for the realtime web. (RethinkDB.com) Tip of the Week Learn C the Hard Way: Practical Exercises on the Computational Subjects You Keep Avoiding (Like C) by Zed Shaw (Amazon) With Windows Terminal installed: In File Explorer, right click on or in a folder and select Open in Windows Terminal. Right click on the Windows Terminal icon to start a non-default shell. SonarLint is a free and open source IDE extension that identifies and helps you fix quality and security issues as you code. (SonarLint.org) Use docker buildx to create custom builders. Just be sure to call docker buildx stop when you're done with it. (Docker docs: docker buildx, docker buildx stop)

We decide to dig into the details of what makes a microservice and do we really understand them as Joe tells us why we really want microservices, Allen incorrectly answers the survey, and Michael breaks down in real time. The full show notes for this episode are available at https://www.codingblocks.net/episode165. Stop by, check it out, and join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says For your next laptop, are you leaning ... Take the survey at: https://www.codingblocks.net/episode165. News Want to know why we're so hot on Skaffold? Check out this video from Joe: Getting Started with Skaffold (YouTube) Atlanta Code Camp is coming up October 9th, come hang out at the CB booth! Want to know what's up with Skaffold? We Thought We Knew About Microservices What are Microservices? A collection of services that are… Highly maintainable and testable Loosely coupled (otherwise you just have a distributed monolith!) Independently deployable Organized around business capabilities (super important, Microservices are just as much about people organization as they are about code) Owned by a small team A couple from Jim Humelsine (Design Patterns Evangelist) Stateless Independently scalable (both in terms of tech, but also personnel) Note: we didn't say anything about size but Sam Newman's definition is: "Microservices are small, autonomous services that work together." Semantic Diffusion (vague term getting vaguer) Enables frequent and reliable delivery of complex applications Allows you to evolve your tech stack (reminiscent of the strangler pattern) They are NOT a silver bullet – Many downsides A Pattern Language A collection of patterns for apply microservice patterns Example Microservice Implementation: https://microservices.io/patterns/microservices.html 3 micro-services in the example: Inventory service Account service Shipping service Each services talks to a separate backend database – i.e., inventory service talks to inventory DB, etc. Fronting those micro-services are a couple of API's – a mobile gateway API and an API that serves a website When an order is placed, a request is made to the mobile API to place the order, the mobile API has to make individual calls to each one of the individual micro-services to get / update information regarding the order This setup is in contrast to a monolithic setup where you'd just have a single API that talks to all the backends and coordinates everything itself The macro problem with microservices (Stack Overflow) Pros of the Microservice Architecture Each service is small so it's easier to understand and change Easier / faster to test as they're smaller and less complex Better deployability – able to deploy each service independently of the others Easier to organize development effort around smaller, autonomous teams Because the code bases are smaller, the IDEs are actually better to work in Improved fault isolation – example they gave is a memory leak won't impact ALL parts of the system like in a monolithic design Applications start and run faster when they are smaller Allows you to be more flexible with tech stacks – you can change out small pieces rather than entire systems if necessary Cons of the Microservice Approach Additional complexity of a distributed system Distributed debugging is hard! Requires additional tooling Additional cost (overhead of services, network traffic) Multi-system transactions are really hard Implementing inter-service communication and handling of failures Implementing multi-service requests is more complex Not only more complex, but you may be interfacing with multiple developer teams as well Testing interactions between services is more complex IDEs don't really make distributed application development easier – more geared towards monolithic apps Deployments are more complex – managing multiple services, dependencies, etc. Increased infrastructure requirements – CPU, memory, etc. Distributed debugging is hard! Requires additional tooling How to Know When to Choose the Microservice Architecture This is actually a hard problem. Choosing this path can slow down development However, if you need to scale in the future, splitting apart / decomposing a monolith may be very difficult Decomposing an Application into Microservices Do so by business capability Example for e-commerce: Product catalog management, Inventory management, Order management, Delivery management How do you know the right way to break down the business capabilities? Organizational structure – customer service department, billing, shipping, etc Domain model – these usually map well from domain objects to business functions Which leads to decomposing by domain driven design Decompose by "verb" – ship order, place order, etc Decompose by "noun" – Account service, Order service, Billing service, etc Follow the Single Responsibility Principal – similar to software de

We dive into JetBrains' findings after they recently released their State of the Developer Ecosystem for 2021 while Michael has the open down pat, Joe wants the old open back, and Allen stopped using the command line. The full show notes for this episode are available at https://www.codingblocks.net/episode164. Stop by, check it out, and join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says What's your IDE of choice? Take the survey at: https://www.codingblocks.net/episode164. News We really appreciate the latest reviews, so thank you! iTunes: TenPal7 Allen has been making videos of some of our tips: HTTP Requests Made Easy in Visual Studio Code Easily See Text Diffs in Visual Studio Code Atlanta Code Camp is coming up October 9th, come hang out at the CB booth! Check out Allen's Quick Tips! Why JetBrains? JetBrains has given us free licenses to give out for years now. Sometimes people ask us what it is that we like about their products, especially when VS Code is such a great (and 100% free) experience…so we'll tell ya! JetBrains produces (among other things) a host of products that are all based on the same IDEA platform, but are custom tailored for certain kinds of development. CLion for C, Rider for C#, IntelliJ for JVM, WebStorm for front-end, etc. These IDEs support plugins but they come stocked with out-of-the-box functionality that you would have to add via plugins in a generalized Editor or IDE This also helps keep consistency amongst developers…everybody using the same tools for git, databases, formatting, etc Integrated experience vs General Purpose Tool w/ Plugins, Individual plugins allow for a lot of innovation and evolution, but they aren't designed to work together in the same way that you get from an integrated experience. JetBrains has assembled a great community Supporting user groups, podcasts, and conferences for years with things like personal licenses Great learning materials for multiple languages (see the JetBrains Academy) Community (free) versions of popular products (Android Studio, IntelliJ, WebStorm, PyCharm) Advanced features that have taken many years of investment and iteration (Resharper/Refactoring tools) TL;DR JetBrains has been making great products for 20 years, and they are still excelling because those products are really good! Survey Results Survey was comprised of 31,743 developers from 183 countries. JetBrains attempted to get a wide swath of diverse responses and they weighted the results in an attempt to get a realistic view of the world. Read more about the methodology What would you normally expect from JetBrain's audience? (Compare to surveys from StackOverflow or Github or State of JS) JetBrains are mainly known for non-cheap, heavy duty tools so you might expect to see more senior or full time employees than StackOverlow, but that's not the case…it skews younger Professional / Enterprise (63% full-time, 70.9% on latest Stack Overflow) JetBrains 3-5 vs StackOverflow 5-9 years of experience Education level is similar 71% of respondents develop for web backend! Key Takeaways JavaScript is the most popular language Python is more popular than Java overall, but Java is more popular as a main language Top 5 languages devs are planning to adopt: Go Kotlin TypeScript Python Rust Top 5 Languages devs learning in 2021: JS Python TS Java Go Languages that fell: Ruby Objective C Scala Top 5 Fastest Growing: Python TypeScript SQL Go Kotlin 71% of respondents develop for web backend Primary programming languages, so much JS! Developer OS: 61% Windows 47% linux 44% macOS Lifestyle and Fun What sources of information… Podcasts 31%! Glad to see this up there, of course 74% of the respondents use online ad-blocking tools Accounts: Github 84% Reddit…47%? Workplace and Events – pre covid comparisons Video Games are #1 hobby, last year was programming Databases Used in last 12 Months, Primary…so much MySQL Really cool to see relative popularity by programming language DevOps How familiar are you with Docker? DevOps engineers are 2x more likely to be architects, 30% more likely to be leads Kubernetes: went from 16% to 29% to 40% to…40%. Is Kubernetes growth stalling? 90% of devs who use k8s have SSD, have above average RAM 53% of hosting is in the cloud? Still moving up, but there's also a lot of growth with Hybrad AWS has a big lead in cloud services…GCP 2nd!? Let's speculate how that happened, that's not what we see in financial reports During development, where do you run and debug your code? (Come to Joe's skaffold talk!) Microservices 35% of respondents develop microservices!!!!! Can this be right? Mostly senior devs are doing microservices GraphQL at 14%, coming up a little bit from last year Miscellaneous How much RAM? (Want more RAM? Be DevOps, Architect, Data Analyst, leads) 79% of devs have SSD? Excellent! How old is your computer? Survey says….2 years? That's reall

It's time to take a break, stretch our legs, grab a drink, and maybe even join in some interesting conversations around the water cooler as Michael goes off script, Joe is very confused, and Allen insists that we stay on script. The full show notes for this episode are available at https://www.codingblocks.net/episode163. Stop by, check it out, and join the conversation. Sponsors Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. Survey Says Which desktop OS do you prefer? Take the survey at: https://www.codingblocks.net/episode163 News We really appreciate the latest reviews, so thank you! iTunes: EveryNickIsTaken2858, Memnoch97 Allen finished his latest ergonomic keyboard review: Moonlander Ergonomic Keyboard Long Term Review (YouTube) Sadly, the .http files tip from episode 161 for JetBrains IDEs is only application for JetBrains' Ultimate version. Test RESTful Web services (JetBrains) Meantime, at the watercooler…. GitHub Copilot (GitHub) In short, it's a VS Code Extension that leverages the OpenAI Codex, a product that translates natural language to code, in order to … wait for it … write code. It's currently in limited preview. What's the value? Is the code correct? Github says ~40-50% in some large scale test cases It works best with small, documented functions Does having the code written for you steer you towards solutions? Could this encourage similar bugs/security holes across multiple languages by people importing the same code? Is this any different from developers using the same common solutions from StackOverflow? Could it become a crutch for new developers? Better for certain kinds of code? (Boiler plate, common accessors, date math) Boiler Plate (like angular / controller vars) Common APIs (Twitter, Goodreads) Common Algorithms, Design Patterns Less Familiar Languages But is it useful? We'll see! Is this the future? We see more low, no, and now co-code solutions all the time, is this where things are going? This probably won't be "it", but maybe we will see things like this more commonly – in any case it's different, why not give it a shot? Is it Ethical? The "AI" or whatever has been trained on "billions of lines" of open-source code…but not strictly permissive licenses. This means a dev using this tool runs the risk of accidently including proprietary code Quake Engine Source Code Example (GPLv2) (Twitter) From an article in VentureBeat: 54 million public software repositories hosted on GitHub as of May 2020 (for Python) 179GB of unique Python files under 1MB in size. Some basic limitations on line and file length, sanitization: The final training dataset totaled 159GB. There is problem with bias, especially in more niche categories Is it ethical to use somebody else's data to train an AI without their permission? Can it get you sued? Would your thoughts change if the data is public? License restricted? Would your thoughts change if the product/model were open-sourced? Abstractions… how far is too far? Services should communicate with datastores and services via APIs that hide the details, these provide for a nice indirection that allows for easier maintenance in the future Do you abstract at the service level or the feature level? Are ORMs a foregone conclusion? What about services that have a unique communication pattern, or assist with cross cutting concerns for things like microservices (We are looking at you hear Kafka!) The 10 Best Practices for Remote Software Engineering From article: The 10 Best Practices for Remote Software Engineering (ACM) Work on Things You Care About Define Goals for Yourself Define Productivity for Yourself Establish Routine and Environment Take Responsibility for Your Work Take Responsibility for Human Connection Practice Empathetic Review Have Self-Compassion Learn to Say Yes, No, and Not Anymore Choose Correct Communication Channels Terminal Tricks (CodeMag.com) Some of Michael's (Linux/macOS) favorites from the article: Abbreviate your directories with tab completion when changing directories, such as cd /v/l/a, and assuming that that abbreviated path can uniquely identify something like, /var/logs/apache, tab completion will take care of the rest. Use nl to get a numbered list of some previous command's output, such as ls -l | nl. ERRATUM: During the episode, Michael mentioned that the output would first list the total lines, but that just happened to be due to output from ll and was unrelated to the output from nl. On macOS, you can use the powermetrics command to gain access to all sorts of metrics related to the internals of your computer, such as the temperature at various sensors. Use !! to repeat the last command. This can be especially helpful when you want to do something like prepend/append the previous command, such as sudo !!. ERRATUM: Wow, Michael really got this one wrong during the episo

We wrap up our replication discussion of Designing Data-Intensive Applications, this time discussing leaderless replication strategies and issues, while Allen missed his calling, Joe doesn't read the gray boxes, and Michael lives in a future where we use apps. If you're reading this via your podcast player, you can find this episode's full show notes at https://www.codingblocks.net/episode162. As Joe would say, check it out and join in on the conversation. Sponsors Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. Survey Says Do you have TikTok installed? Take the survey at: https://www.codingblocks.net/episode162. News Thank you for the latest review! iTunes: tuns3r Check out the book! Single Leader to Multi-Leader to Leaderless When you have leaders and followers, the leader is responsible for making sure the followers get operations in the correct order Dynamo brought the trend to the modern era (all are Dynamo inspired) but also… Riak Cassandra Voldemort We talked about NoSQL Databases before: Episode 123 Data Models: Relational vs Document What exactly is NewSQL? https://en.wikipedia.org/wiki/NewSQL What if we just let every replica take writes? Couple ways to do this… You can write to several replicas You can use a coordinator node to pass on the writes But how do you keep these operations in order? You don't! Thought exercise, how can you make sure operation order not matter? Couple ideas: No partial updates, increments, version numbers Multiple Writes, Multiple Reads What do you do if your client (or coordinator) try to write to multiple nodes…and some are down? Well, it's an implementation detail, you can choose to enforce a "quorom". Some number of nodes have to acknowledge the write. This ratio can be configurable, making it so some % is required for a write to be accepted What about nodes that are out of date? The trick to mitigating stale data…the replicas keep a version number, and you only use the latest data – potentially by querying multiple nodes at the same time for the requested data We've talked about logical clocks before, it's a way of tracking time via observed changes…like the total number of changes to a collection/table…no timezone or nanosecond differences How do you keep data in sync? About those unavailable nodes…2 ways to fix them up Read Repair: When the client realizes it got stale data from one of the replicas, it can send the updated data (with the version number) back to that replica. Pretty cool! – works well for data that is read frequently Anti-Entropy: The nodes can also do similar background tasks, querying other replicas to see which are out of data – ordering not guaranteed! Voldemort: ONLY uses read repair – this could lead to loss of data if multiple replicas went down and the "new" data was never read from after being written Quorums for reading and writing Quick Reminder: We are still talking about 100% of the data on each replica 3 major numbers at play: Number of nodes Number of confirmed writes Number of reads required If you want to be safe, the nodes you write to and the ones you write too should include some overlap A common way to ensure that, keep the number of writes + the number of reads should be greater than the number of nodes Example: You have 10 nodes – if you use 5 for writing and 5 for reading…you may not have an overlap resulting in potentially stale data! Common approach – taken number of nodes (odd number) + 1, then divide that number by 2 and that's the number of reader and writers you should have 9 Nodes – 5 writes and 5 reads – ensures non-stale data When using this approach, you can handle Nodes / 2 (rounded down) number of failed nodes How would you tweak the numbers for a write heavy workload? Typically, you write and read to ALL replicas, but you only need a successful response from these numbers What if you have a LOT of nodes?!? Note: there's still room for problems here – author explicitly lists 5 types of edge cases, and one category of miscellaneous timing edge cases. All variations of readers and writers getting out of sync or things happen at the same timing If you really want to be safe, you need consensus (r = w = n) or transactions (that's a whole other chapter) Note that if the number of required readers or writers doesn't return an OK, then an error is returned from the operation Also worth considering is you don't have to have overlap – having readers + writers Monitoring staleness Single/Multi Leader lag is generally easy to monitor – you just query the leader and the replicas to see which operation they are on Leaderless databases don't have guaranteed ordering so you can't do it this way If the system only uses read repair (where the data is fixed up by clients only as it is read) then you can have data that is ancient It's hard to give a good algorithm description h

We continue our discussion of Designing Data-Intensive Applications, this time focusing on multi-leader replication, while Joe is seriously tired, and Allen is on to Michael's shenanigans. For anyone reading this via their podcast player, this episode's show notes can be at https://www.codingblocks.net/episode161, where you can join the conversation. Sponsors Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. Survey Says How do you put on your shoes? Take the survey at: https://www.codingblocks.net/episode161. News Thank you very much for the new reviews: iTunes: GubleReid, tbednarick, JJHinAsia, katie_crossing Audible: Anonymous User, Anonymous User … hmm When One Leader Just Won't Do Talking about Multi-Leader Replication Replication Recap and Latency When you're talking about single or multi-leader replication, remember all writes go through leaders If your application is read heavy, then you can add followers to increase your scalability That doesn't work well with sync writes..the more followers, the higher the latency The more nodes the more likely there will be a problem with one or more The upside is that your data is consistent The problem is if you allow async writes, then your data can be stale. Potentially very stale (it does dial up the availability and perhaps performance) You have to design your app knowing that followers will eventually catch up – "eventual consistency" "Eventual" is purposely vague – could be a few seconds, could be an hour. There is no guarantee. Some common use cases make this particularly bad, like a user updating some information…they often expect to see that change afterwards There are a couple techniques that can help with this problem Techniques for mitigation replication lag Read You Writes Consistency refers to an attempt to read significant data from leader or in sync replicas by the user that submitted the data In general this ensures that the user who wrote the data will get the same data back – other users may get stale version of the data But how can you do that? Read important data from a leader if a change has been made OR if the data is known to only be changeable by that particular user (user profile) Read from a leader/In Sync Replica for some period of time after a change Client can keep a timestamp of it's most recent write, then only allow reads from a replica that has that timestamp (logical clocks keep problems with clock synchronization at bay here) But…what if the user is using multiple devices? Centralize MetaData (1 leader to read from for everything) You make sure to route all devices for a user the same way Monotonic Reads: a guarantee of sorts that ensures you won't see data moving backwards in time. One way to do this – keep a timestamp of the most recent read data, discard any reads older than that…you may get errors, but you won't see data older than you've already seen. Another possibility – ensure that the reads are always coming from the same replica Consistent Prefix Reads: Think about causal data…an order is placed, and then the order is shipped…but what if we had writes going to more than one spot and you query the order is shipped..but nothing was placed? (We didn't have this problem with a Single Replica) We'll talk more about this problem in a future episode, but the short answer is to make sure that causal data gets sent to the same "partition" Replication isn't as easy as it sounds, is it? Multi-Leader Rep…lication Single leader replication had some problems. There was a single point of failure for writes, and it could take time to figure out the new leader. Should the old leader come back then…we have a problem. Multi-Leader replication… Allows more than one node to receive writes Most things behave just like single-leader replication Each leader acts as followers to other leaders When to use Multi-Leader Replication Many database systems that support single-leader replication can be taken a step further to make them mulit-leader. Usually. you don't want to have multiple leaders within the same datacenter because the complexity outweighs the benefits. When you have multiple leaders you would typically have a leader in each datacenter An interesting approach is for each datacenter to have a leader and followers…similar to the single leader. However, each leader would be followers to the other datacenter leaders Sort of a chained single-leader replication setup Comparing Single-Leader vs Multi-Leader Replication Performance – because writes can occur in each datacenter without having to go through a single datacenter, latency can be greatly reduced in multi-leader The synchronization of that data across datacenters can happen asynchronously making the system feel faster overall Fault tolerance – in single-leader, everything is on pause while a new leader is elected In multi-leade

We dive back into Designing Data-Intensive Applications to learn more about replication while Michael thinks cluster is a three syllable word, Allen doesn't understand how we roll, and Joe isn't even paying attention. For those that like to read these show notes via their podcast player, we like to include a handy link to get to the full version of these notes so that you can participate in the conversation at https://www.codingblocks.net/episode160. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. Survey Says How important is it to learn advanced programming techniques? Take the survey at: https://www.codingblocks.net/episode160. News Thank you to everyone that left us a new review: Audible: Ashfisch, Anonymous User (aka András) The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair Douglas Adams Douglas Adams In this episode, we are discussing Data Replication, from chapter 5 of "Designing Data-Intensive Applications". Replication in Distributed Systems When we talk about replication, we are talking about keeping copies of the same data on multiple machines connected by a network For this episode, we're talking about data small enough that it can fit on a single machine Why would you want to replicate data? Keeping data close to where it's used Increase availability Increase throughput by allowing more access to the data Data that doesn't change is easy, you just copy it 3 popular algorithms Single Leader Multi-Leader Leaderless Well established (1970's!) algorithms for dealing with syncing data, but a lot data applications haven't needed replication so the practical applications are still evolving Cluster group of computers that make up our data system Node each computer in the cluster (whether it has data or not) Replica each node that has a copy of the database Every write to the database needs to be copied to every replica The most common approach is "leader based replication", two of the algorithms we mentioned apply One of the nodes is designated as the "leader", all writes must go to the leader The leader writes the data locally, then sends to data to it's followers via a "replication log" or "change stream" The followers tail this log and apply the changes in the same order as the leader Reads can be made from any of the replicas This is a common feature of many databases, Postgres, Mongo, it's common for queues and some file systems as well Synchronous vs Asynchronous Writes How does a distributed system determine that a write is complete? The system could hang on till all replicas are updated, favoring consistency…this is slow, potentially a big problem if one of the replicas is unavailable The system could confirm receipt to the writer immediately, trusting that replicas will eventually keep up… this favors availability, but your chances for incorrectness increase You could do a hybrid, wait for x replicas to confirm and call it a quorum All of this is related to the CAP theorem…you get at most two: Consistency, Availability and Partition Tolerance Site Note: Can you ever have Consistent/Available databases? https://codahale.com/you-cant-sacrifice-partition-tolerance/ The book mentions "chain replication" and other variants, but those are still rare Example: Chain replication in Mongo: https://docs.mongodb.com/manual/tutorial/manage-chained-replication/ Steps for Adding New Followers Take a consistent snapshot of the leader at some point in time (most db can do this without any sort of lock) Copy the snapshot to the new follower The follower connects to the leader and requests all changes since the back-up When the follower is fully caught up, the process is complete Handling Outages Nodes can go down at any given time What happens if a non-leader goes down? What does your db care about? (Available or Consistency) Often Configurable When the replica becomes available again, it can use the same "catch-up" mechanism we described before when we add a new follower What happens if you lose the leader? Failover: One of the replicas needs to be promoted, clients need to reconfigure for this new leader Failover can be manual or automatic Rough Steps for Failover Determining that the leader has failed (trickier than it sounds! how can a replica know if the leader is down, or if it's a network partition?) Choosing a new leader (election algorithms determine the best candidate, which is tricky with multiple nodes, separate systems like Apache Zookeeper) Recon

We couldn't decide if we wanted to gather around the water cooler or talk about some cool APIs, so we opted to do both, while Joe promises there's a W in his name, Allen doesn't want to say graph, and Michael isn't calling out applets. For all our listeners that read this via their podcast player, this episode's show notes can be found at https://www.codingblocks.net/episode159, where you can join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. ConfigCat – The feature flag and config management service that lets you turn features ON after deployment or target specific groups of users with different features. Survey Says How often do you leetcode? Take the survey at: https://www.codingblocks.net/episode159. News Thank you all for the latest reviews: iTunes: Lp1876 Audible: Jon, Lee Overheard around the Water Cooler Where do you draw the line before you use a hammer to solve every problem? When is it worth bringing in another technology? Can you have too many tools? APIs of Interest Joe's Picks Video game related APIs RAWG – The Biggest Video Game Database on RAWG – Video Game Discovery Service (rawg.io) RAWG Video Games Database API (v1.0) (api.rawg.io) PS: Your favorite video games might have an API: PokéAPI – The RESTful Pokémon API (pokeapi.co) Legends of Runeterra (developer.riotgames.com) The Bungie.Net API (GitHub) Blizzard Battle.net Developer Portal (develop.battle.net) Satellite imagery related APIs Planet – A leader in Earth observation (planet.com) bird.i – Satellite Imagery API (hibirdi.com) Get into the affiliate game Rainforest – The (missing) Amazon Product Data API (rainforestapi.com) Allen's Picks Amazon Market Web Service (docs.developer.amazonservices.com) We would *love* a Libsyn API! Should paid services provide an API? Michael's Picks Alpha Vantage – Free Stock APIs (alphavantage.co) Why so serious? icanhazdadjoke – The largest selection of dad jokes on the Internet (icanhazdadjoke.com) Channel your inner Stuart Smalley with affirmations. (affirmations.dev) HTTP Cats – The ultimate source for HTTP status code images. (http.cat) Relevant call backs from episode 127: Random User Generator – A free, open-source API for generating random user data. (randomuser.me) Remember the API – Programmer gifts and merchandise (remembertheapi.com) Resources We Like ReDoc – OpenAPI/Swagger-generated API Reference Documentation (GitHub) Google Earth – The world's most detailed globe. (google.com/earth) Google Sky – Traveling to the stars has never been easier. (google.com/sky) apitracker.io – Discover the best APIs and SaaS products to integrate with. (apitracker.io) ProgrammableWeb – The leading source of news and information about Internet-based APIs.(ProgrammableWeb.com) NASA APIs – NASA data, including imagery, accessible to developers. (api.nasa.gov) RapidAPI – The Next-Generation API Platform (rapidapi.com) Stuart Smalley (Wikipedia) Al Franken (Wikipedia) Muzzle – A simple Mac app to silence embarrassing notifications while screensharing. (MuzzleApp.com) Previously mentioned in episode 125. Tip of the Week Not sure what project to do? Google for an API or check out RapidAPI for a consistent way to farm ideas: RAWG Video Games Database API Documentation (rapidapi.com) Press F12 in Firefox, Chrome, or Edge, then go to the Elements tab (or Inspector in Firefox) to start hacking away at the DOM for immediate prototyping. All things K9s Getting Started with K9s – A Love Letter to K9s Use K9s to easily monitor your Kubernetes cluster Not only does K9s support skins and themes, but supports *cluster specific* skins (k9scli.io) If you like xkcd, Monkey User is for you! xkcd – A webcomic of romance, sarcasm, math, and language. (xkcd.com) Monkey User – Created out of a desire to bring joy to people working in IT. (MonkeyUser.com) Remap Windows Terminal to use CTRL+D, another keyboard customizations. (docs.microsoft.com) PostgreSQL and Foreign Data (postgresql.org) A listing of available foreign data wrappers for PostgreSQL on the wiki. (wiki.postgresql.org) Cheerio – Fast, flexible & lean implementation of core jQuery designed specifically for the server. (npmjs.com) JetBrains MPS (Meta Programming System) – Create your own domain-specific language (JetBrains) Case study – Domain-specific languages to implement Dutch tax legislation and process changes of that legislation. (JetBrains)

We talk about the various ways we can get paid with code while Michael failed the Costco test, Allen doesn't understand multiple choice questions, and Joe has a familiar pen name. This episode's show notes can be found at https://www.codingblocks.net/episode158, where you can join the conversation, for those reading this via their podcast player. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says Do you want to run your own business? Take the survey at: https://www.codingblocks.net/episode158. News Thank you all for the latest reviews: iTunes: PriestRabbitWalkIntoBloodBank, Sock-puppet Sophist sez, Rogspug, DhokeDev, Dan110024 Audible: Aiden Show Me the Money Active Income Active income is income earned by exchanging time for money. This typically includes salary and hourly employment, as well as contracting. Some types of active income blur the lines. Way to find active income can include job sites like Stack Overflow Jobs, Indeed, Upwork, etc. Government grants and jobs are out there as well. Active income is typically has some ceiling, such as your time. Passive Income Passive income is income earned on an investment, any kind of investment, such as stock markets, affiliate networks, content sales for things like books, music, courses, etc. The work you do for the passive income can blur lines, especially when that work is promotion. Passive income is generally not tied to your time. Passive Income Options Create a SaaS platform to keep people coming back. Don't let the term SaaS scare you off. This can be something smaller like a regex validator. Affiliate links are a great example of passive income because you need to invest the time once to create the link. Ads and sponsors: typically, the more targeted the audience is for the ad, the more the ad is worth. Donations via services like Ko-fi, Patreon, and PayPal. Apps, plugins, website templates/themes Create content, such as books, courses, videos, etc. Self-publishing can have a bigger reward and offer more freedom, but doesn't come with the built-in audience and marketing team that a publisher can offer. Arbitrage between markets. Grow an audience, be it on YouTube, Twitch, podcasting, blogging, etc. Things to Consider What's the up-front effort and/or investment? How much maintenance can you afford? How much will it cost you? Who gets hurt if you choose to quit? What can you realistically keep up with? What are the legal and tax liabilities? Resources We Like Apply for Grants To Fund Open Source Work (changeset.nyc) Government grants and loans (usa.gov, grants.gov) 35 Passive Income Ideas for Developers [All Types] (beginnerspassiveincome.com) 8 Side Income Ideas For Programmers (That Actually Work) (afternerd.com) Podcasts The Smart Passive Income Podcast with Pat Flynn (smartpassiveincome.com) Entrepreneurs On Fire (eofire.com) How I Built This with Guy Raz (npr.org) Who Moved My Cheese (Amazon) The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses (Amazon) These top Patreon creators earn more than $200,000 a year (blog.patreon.com) Ko-fi – Make an Income Doing What You Love (ko-fi.com) PayPal – Make your Donate button (paypal.com) How Long Does It Take To Create An Online Course? (onlinecoursehow.com) Udemy – Planning your online course (udemy.com) Am I Procrastinating? (amiprocrastinating.com) Google Ads Help: Use Keyword Planner (support.google.com) Tip of the Week Google developer documentation style guide: Word list (developers.google.com) In Windows Terminal, use CTRL+SHIFT+W to close a tab or the window. The GitHub CLI manual (cli.github.com) Use gh pr create --fill to create a pull request using your last commit message as the title and body of the PR. We've discussed the GitHub CLI in episode 142 and episode 155. How to get a dependency tree for an artifact? (Stack Overflow) xltrail – Version control for Excel workbooks (xltrail.com) Spring Initializr (start.spring.io) You can leverage the same thing in IntelliJ with Spring.

We discuss all things APIs: what makes them great, what makes them bad, and what we might like to see in them while Michael plays a lawyer on channel 46, Allen doesn't know his favorite part of the show, and Joe definitely pays attention to the tips of the week. For those reading this episode's show notes via their podcast player, you can find this episode's show notes at https://www.codingblocks.net/episode157 where you can be a part of the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says How do you prefer to be interviewed? Take the survey at: https://www.codingblocks.net/episode157. News Big thanks to everyone that left us a new review: iTunes: hhskakeidmd Audible: Colum Ferry All About APIs What are APIs? API stands for application programming interface and is a formal way for applications to speak to each other. An API defines requests that you can make, what you need to provide, and what you get back. If you do any googling, you'll see that articles are overwhelmingly focused on Web APIs, particularly REST, but that is far from the only type. Others include: All libraries, All frameworks, System Calls, i.e.: Windows API, Remote API (aka RPC – remote procedure call), Web related standards such as SOAP, REST, HATEOAS, or GraphQL, and Domain Specific Languages (SQL for example) The formal definition of APIs, who own them, and what can be done with them is complicated à la Google LLC v. Oracle America, Inc. Different types of API have their own set of common problems and best practices Common REST issues: Authentication, Rate limiting, Asynchronous operations, Filtering, Sorting, Pagination, Caching, and Error handling. Game libraries: Heavy emphasis on inheritance and "hidden" systems to cut down on complexity. Libraries for service providers Support multiple languages and paradigms (documentation, versioning, rolling out new features, supporting different languages and frameworks) OData provides a set of standards for building and consuming REST API's. General tips for writing great APIs Make them easy to work with. Make them difficult to misuse (good documentation goes a long way). Be consistent in the use of terms, input/output types, error messages, etc. Simplicity: there's one way to do things. Introduce abstractions for common actions. Service evolution, i.e. including the version number as part of your API call enforces good versioning habits. Documentation, documentation, documentation, with enough detail that's good to ramp up from getting started to in depth detail. Platform Independence: try to stay away from specifics of the platforms you expect to deal with. Why is REST taking over the term API? REST is crazy popular in web development and it's really tough to do anything without it. It's simple. Well, not really if you consider the 43 things you need to think about. Some things about REST are great by design, such as: By using it, you only have one protocol to support, It's verb oriented (commonly used verbs include GET, POST, PUT, PATCH, and DELETE), and It's based on open standards. Some things about REST are great by convention, such as: Noun orientation like resources and identifiers, Human readable I/O, Stateless requests, and HATEOAS provides a methodology to decouple the client and the server. Maybe we can steal some ideas from REST Organize the API around resources, like /orders + verbs instead of /create-order. Note that nouns can be complex, an order can be complex … products, addresses, history, etc. Collections are their own resources (i.e. /orders could return more than 1). Consistent naming conventions makes for easy discovery. Microsoft recommends plural nouns in most cases, but their skewing heavily towards REST, because REST already has a mechanism for behaviors with their verbs. For example /orders and /orders/123. You can drill in further and further when you orient towards nouns like /orders/123/status. The general guidance is to return resource identifiers rather than whole objects for complex nouns. In the order example, it's better to return a customer ID associated with the whole order. Avoid introducing dependencies between the API and the underlying data sources or storage, the interface is meant to abstract those details! Verb orientation is okay in some, very action based instances, such as a calculator API. Resources We Like API (Wikipedia) The Linux Kernel API (kernel.org) gRPC (Wikipedia) S3 Compatible API (Backblaze) Storing Data Shouldn't Cost More Than Generating It (Wasabi) Top 50 Most Popular APIs on RapidAPI (2021) (rapidapi.com) Free Public APIs for Developers APIs (rapidapi.com) API Reference (Datadog) OData – the best way to REST (odata.org) Understand OData in 6 steps (odata.org) Best practices for REST API design (Stack Overflow) Best Practices in API Design (swagger.io) Web API design (docs.microsoft.com) The Web API Checklist — 43 Things To Think

We discuss the parts of the scrum process that we're supposed to pay attention to while Allen pronounces the m, Michael doesn't, and Joe skips the word altogether. If you're reading this episode's show notes via your podcast player, just know that you can find this episode's show notes at https://www.codingblocks.net/episode156. Stop by, check it out, and join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says For your next car, you plan to buy: Take the survey at: https://www.codingblocks.net/episode156. News Hey, we finally updated the Resources page. It only took a couple years. Apparently we don't understand the purpose of the scrum during rugby. (Wikipedia) Standup Time User Stories A user story is a detailed, valuable chunk of work that can be completed in a sprint.\ Use the INVEST mnemonic created by Bill Wake: I = Independent – the story should be able to be completed without any dependencies on another story N = Negotiable – the story isn't set in stone – it should have the ability to be modified if needed V = Valuable – the story must deliver value to the stakeholder E = Estimable – you must be able to estimate the story S = Small – you should be able to estimate within a reasonable amount of accuracy and completed within a single sprint T = Testable – the story must have criteria that allow it to be testable Stories Should be Written in a Format Very Much Like… "As a _____, I want _____ so that _____.", like "As a user, I want MFA in the user profile so I can securely log into my account" for a functional story, or "As a developer, I want to update our version of Kubernetes so we have better system metrics" for a nonfunctional story. Stories Must have Acceptance Criteria Each story has it's own UNIQUE acceptance criteria. For the MFA story, the acceptance criteria might be:\ Token is captured and saved. Verification of code completed successfully. Login works with new MFA. The acceptance criteria defines what "done" actually means for the story. Set up Team Boundaries Define "done". Same requirement for ALL stories: Must be tested in QA environment, Must have test coverage for new methods. Backlog prioritization or "grooming". Must constantly be ordered by value, typically by the project owner. Define sprint cadence Usually 1-4 weeks in length, 2-3 is probably best. Two weeks seems to be what most choose simply because it sort of forces a bit of urgency on you. Estimates Actual estimation, "how many hours will a task take?" Relative estimation, "I think this task will take 2x as long as this other ticket." SCRUM uses both, user stories are compared to each other in relative fashion. By doing it this way, it lets external stakeholders know that it's an estimate and not a commitment. Story points are used to convey relative sizes. Estimation is supposed to be lightweight and fast. Roadmap and Release Plan The roadmap shows when themes will be worked on during the timeframe.\ You should be able to have a calendar and map your themes across that calendar and in an order that makes sense for getting a functional system. Just because you should have completed, functional components at the end of each sprint, based on the user stories, that doesn't mean you're releasing that feature to your customer. It may take several sprints before you've completed a releasable feature. It will take several sprints to find out what a team's stabilized velocity is, meaning that the team will be able to decently predict how many story points they can complete in a given sprint. Filling up the Sprint Decide how many points you'll have for a sprint. Determine how many sprints before you can release the MVP. Fill up the sprints as full as possible in priority order UNLESS the next priority story would overflow the sprint. Simple example, let's say your sprint will have 10 points and you have the following stories: Story A – 3 points Story B – 5 points Story C – 8 points Story D – 2 points Your sprints might look like: Sprint 1 – A (3) B(5), D(2) = 10 points Sprint 2 – C (8) Story C got bumped to Sprint 2 because the goal is to maximize the amount of work that can be completed in a given sprint in priority order, as much as possible. The roadmap is an estimate of when the team will complete the stories and should be updated at the end of each sprint. In other words, the roadmap is a living document. Sprint Planning This is done at the beginning of each sprint. Attendees – all developers, scrum master, project owner. Project owner should have already prioritized the stories in the backlog. The goal of the planning meeting is to ensure all involved understand the stories and acceptance criteria. Also make sure the overarching definition of "done" is posted as a reminder. Absolutely plan for a Q&A session. Crucial to make sure any misunderstandings of the stories are cleared here. Next the stories are broken down into speci

During today's standup, we focus on learning all about Scrum as Joe is back (!!!), Allen has to dial the operator and ask to be connected to the Internet, and Michael reminds us why Blockbuster failed. If you didn't know and you're reading these show notes via your podcast player, you can find this episode's show notes in their original digital glory at https://www.codingblocks.net/episode155 where you can also jump in the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. Survey Says While in Slack, do you reply ... Take the survey at: https://www.codingblocks.net/episode155. News Thank you all for the reviews: iTunes: DareOsewa, Miggybby, MHDev, davehadley, GrandMasterJR, Csmith49, ForTheHorde&Tacos, A-Fi Audible: Joshua, Alex Do You Even Scrum? Why Do We Call it Scrum Anyways? Comes from the game of Rugby. A scrummage is when a team huddles after a foul to figure out their next set of plays and readjust their strategy. Why is Scrum the Hot Thing? Remember waterfall? Plan and create documentation for a year up front, only to build a product with rigid requirements for the next year. By the time you deliver, it may not even be the right product any longer. Waterfall works for things that have very repeatable steps, such as things like planning the completion of a building. It doesn't work great for things that require more experimentation and discovery. Project managers saw the flaw in planning for the complete "game" rather than planning to achieve each milestone and tackle the hurdles as they show up along the way. Scrum breaks the deliverables and milestones into smaller pieces to deliver. The Core Tenants of Scrum Having business partners and stakeholders work with the development of the software throughout the project, Measure success using completed software throughout the project, and Allow teams to self-organize. Scrum Wants You to Fail Fast Failure is ok as long as you're learning from it. But those lessons learned need to happen quickly, with fast feedback cycles. Small scale focus and rapid learning cycles. In other words, fail fast really means "learn fast". It's super important to recognize that Scrum is *not* prescriptive. It's more like guardrails to a process. An Overview of the Scrum Framework The product owner has a prioritized backlog of work for the team. Every sprint, the team looks at the backlog and decides what they can accomplish during that sprint, which is generally 2-3 weeks. The team develops and tests their solutions until completed. This effort needs to happen within that sprint. The team then demonstrates their finished product to the product owner at the end of the sprint. The team has a retrospective to see how the sprint went, what worked, and what they can improve going forward. Focusing on creating a completed, demo-able piece of work in the sprint allows the team to succeed or fail/learn fast. Projects are typically comprised of three basic things: time, cost, and scope. Usually time and cost are fixed, so all you can work with is the scope. There are Two Key Roles Within Scrum Project owner – The business representative dedicated 100% to the team. Acts as a full time business representative. Reviews the team's work constantly to ensure the proper deliverable is being created. Interacts with the stakeholders. Is the keeper of the product vision. Responsible for making sure the work is continuously sorted per the ongoing business needs. The Scrum master – Responsible for helping resolve daily issues and balance ongoing changes in requirements and/or scope. This person has a mastery of Scrum. Also helps improve internal team processes. Responsible for protecting the team and their processes. Balances the demands of the product owner and the needs of the team. This means keeping the team working at a sustainable rate. Acts as the spokesperson for the entire team. Provides charts and other views into the progress for others for transparency. Responsible for removing any blockers. Project owner focuses on what needs to be done while the Scrum master focuses on how the team gets it done. Scrum doesn't value heroics by teams or team members. Scrum is all about Daily Collaboration Whatever you can do to make daily collaboration easier will yield great benefits. Collocate your team if possible. If you can't do that, use video conferencing, chat, and/or conference calls to keep communication flowing. The Team Makeup You must have a dedicated team. If members of your team are split amongst different projects, it will be difficult to accomplish your goals as you lose efficiency. The ideal team size is 5 to 9 members. You want a number of T-shaped developers. These are people can work on more than one type of deliverable. You also need some "consultants" you may be able to call on that have more specialized/focused skillsets that may not be core members of the team. Team Norms Teams will need to

We dig into recursion and learn that Michael is the weirdo, Joe gives a subtle jab, and Allen doesn't play well with others while we dig into recursion. This episode's show notes can be found at https://www.codingblocks.net/episode154, for those that might be reading this via their podcast player. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. News Thank you all for the reviews: iTunes: ripco55, Jla115 Audible: _onetea, Marnu, Ian Here I Go Again On My Own What is Recursion? Recursion is a method of solving a problem by breaking the problem down into smaller instances of the same problem. A simple "close enough" definition: Functions that call themselves Simple example: fib(n) { n Recursion pros: Elegant solutions that read well for certain types of problems, particularly with unbounded data. Work great with dynamic data structures, like trees, graphs, linked lists. Recursion cons: Tricky to write. Generally perform worse than iterative solutions. Runs the risk of stack overflow errors. Recursion is often used for sorting algorithms. How Functions Roughly Work in Programming Languages Programming languages generally have the notion of a "call stack". A stack is a data structure designed for LIFO. The call stack is a specialized stack that is common in most languages Any time you call a function, a "frame" is added to the stack. The frame is a bucket of memory with (roughly) space allocated for the input arguments, local variables, and a return address. Note: "value types" will have their values duplicated in the stack and reference types contain a pointer. When a method "returns", it's frame is popped off of the stack, deallocating the memory, and the instructions from the previous function resume where it left off. When the last frame is popped off of the call stack, the program is complete. The stack size is limited. In C#, the size is 1MB for 32-bit processes and 4MB for 64-bit processes. You can change these values but it's not recommended! When the stack tries to exceed it's size limitations, BOOM! … stack overflow exception! How big is a frame? Roughly, add up your arguments (values + references), your local variables, and add an address. Ignoring some implementation details and compiler optimizations, a function that adds two 32b numbers together is going to be roughly 96b on the stack: 32 * 2 + return address. You may be tempted to "optimize" your code by condensing arguments and inlining code rather than breaking out functions… don't do this! These are the very definition of micro optimizations. Your compiler/interpreter does a lot of the work already and this is probably not your bottleneck by a longshot. Use a profiler! Not all languages are stack based though: Stackless Python (kinda), Haskell (graph reduction), Assembly (jmp), Stackless C (essentially inlines your functions, has limitations) The Four Memory Segments source: Quora How Recursive Functions Work The stack doesn't care about what the return address is. When a function calls any other function, a frame is added to the stack. To keep things simple, suppose for a Fibonacci sequence function, the frame requires 64b, 32b for the argument and 32b for the return address. Every Fibonacci call, aside from 0 or 1, adds 2 frames to the stack. So for the 100th number we will allocate .6kb (1002 * 32). And remember, we only have 1mb for everything. You can actually solve Fibonacci iteratively, skipping the backtracking. Fibonacci is often given as an example of recursion for two reasons: It's relatively easy to explain the algorithm, and It shows the danger of this approach. What is Tail Recursion? The recursive Fibonacci algorithm discussed so far relies on backtracking, i.e. getting to the end of our data before starting to wind back. If we can re-write the program such that the last operation, the one in "tail position" is the ONLY recursive call, then we no longer need the frames, because they are essentially just pass a through. A smart compiler can see that there are no operations left to perform after the next frame returns and collapse it. The compiler will first remove the last frame before adding the new one. This means we no longer have to allocate 1002 extra frames on the stack and instead just 1 frame. A common approach to rewriting these types of problems involves adding an "accumulator" that essentially holds the state of the operation and then passing that on to the next function. The important thing here, is that the your ONE AND ONLY recursive call must be the LAST operation … all by itself. Joe's (Un)Official Recursion Tips Start with the end. Do it by hand. Practice, practice, practice. Joe Recursion Joe's Motivational Script 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 #!/usr/bin/env python3 import sys def getname(arg): print(f"{arg}") if arg == 'Joe': getname('Recursion') else: getname('Joe') return if __name__ == "__main__": sys.setrecur

It's been a minute since we last gathered around the water cooler, as Allen starts an impression contest, Joe wins said contest, and Michael earned a participation award. For those following along in their podcast app, this episode's show notes can be found at https://www.codingblocks.net/episode153. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. DataStax – Sign up today to get $300 in credit with promo code CODINGBLOCKS and make something amazing with Cassandra on any cloud. Survey Says When you start a new project, in regards to the storage technology, do you ... Take the survey at: https://www.codingblocks.net/episode153. News Thank you all for the latest reviews: iTunes: peter b :(, Jackifus, onetea_ Getting BSOD? Test your memory with MemTest86. Check out all of the test descriptions. Gather Around the Water Cooler Go deep on a single language? Or know enough about many of them? Who is hiring for remote work? Remote job resources: Datadog DataStax Microsoft Facebook Amazon Github GitLab Twitter Confluent Elastic MongoDB Netlify Heroku remotegamejobs.com Hand-picked remote jobs from Hacker News – Who is hiring (RemoteLeaf.com) remoteok.io Stack Overflow What companies are in your top 3? Know your storage technology. What it excels at and what it doesn't. Resources We Like DB-Engines Ranking (db-engines.com) Search Driven Apps (episode 83) Amazon CloudSearch FAQs (aws.amazon.com) Kubernetes Failure Stores (k8s.af) The Uber Engineering blow (eng.uber.com) Datadog and the Container Report, with Michael Gerstenhaber (Kubernetes Podcast from Google, episode 137) Tip of the Week Automated Google Cloud Platform Authentication with minikube. Be careful about how you use ARG in your Docker images. (docs.docker.com) Calvin and Hobbes the Search Engine (MichaelYingling.com) 11 Facts About Real-World Container Use (Datadog) Tips & Tricks for running Strimzi with kubectl (Strimzi)

We dig into all things Python, which Allen thinks is pretty good, and it's rise in popularity, while Michael and Joe go toe-to-toe over a gripe, ahem, feature. We realize that you _can_ use your podcast player to read these notes, but if you didn't know, this episode's show notes can be found at https://www.codingblocks.net/episode152. Check it out and join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. DataStax – Sign up today to get $300 in credit with promo code CODINGBLOCKS and make something amazing with Cassandra on any cloud. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says What's your favorite Python feature? Take the survey at: https://www.codingblocks.net/episode152. News The Coding Blocks Game Jam 2021 results are in! (itch.io) Our review page has been updated! (/review) Ergonomic keyboard reviews: Kinesis Advantage 2 Full Review after Heavy Usage (YouTube) Ergonomic Keyboard Zergotech Freedom Full Review (YouTube) Why Python? A Brief History of Python. Very Brief. Python is a general-purpose high-level programming language, which can be used to develop desktop GUI applications, websites, and apps that run on sophisticated algorithms. Python was created in 1991, before JavaScript or Java, but didn't make major leaps in popularity until 1998 – 2003, according to the Tiobe index. Coincidentally, this lines up with the early days of Google, where they had a motto of "Python where we can, C++ where we must". In 2009, MIT switched from Scheme to Python, and others in academia followed. Some Python Benefits, But Only Some It's an easy language for new developers as well as those who don't consider themselves developers, such as data scientists or hobbyists, but have a need write some code. Python has a great standard library when compared to languages like JavaScript that largely rely on third party libraries to provide depth in functionality. It's cross platform. As long as we're talking OS. Mobile? Not really, as that space is consumed with Swift, Java, and Objective-C. But with things like Pythonista, you can write and execute Python on mobile. Web? No, at least not on the client side. That space is dominated by JavaScript. But with frameworks like Django and Flask, you can use Python on the server side. In addition to the standard library, there are also many great/popular third party libraries, like NumPy, that are available on PyPi. The Downsides to Python Performance when compared to a natively compiled application because it's a dynamic, interpreted language. Which brings up the late binding type system. The lack of mobile and web presence as previously mentioned. Legacy issues when dealing with v2, which is still in use. Language features that haven't aged well, such as PEP-8. Quirks like self, or __init__, private functionality, and immutability. Resources We Like The Python Programming Language (Tiobe) Heavy usage of Python at Google (Stack Overflow) The Incredible Growth of Python (Stack Overflow) 2020 Developer Survey, Top Paying Technologies (Stack Overflow) What makes Python more popular than Ruby? (Reddit) 2020 Developer Survey, Most Loved, Dreaded, and Wanted (Stack Overflow) Top 10 Python Packages For Machine Learning (ActiveState.com) 56 Groundbreaking Python Open-source Projects – Get Started with Python (data-flair.training) Top 10 Python Packages Every Developer Should Learn (ActiveState.com) Choosing the right estimator aka the scikit-learn algorithm cheat-sheet (scikit-learn.org) Previously discussed as a Tip of the Week during episode 92, Azure Functions and CosmosDB from MS Ignite (/episode92) Introduction to Celery (docs.celeryproject.org) Is it possible to compile a program written in Python? (Stack Overflow) Pythonista 3 – A Full Python IDE for iOS (omz-software.com) Previously discussed as a Tip of the Week during episode 88, What is Algorithmic Complexity? (/episode88) Flask – Web development, one drop at a time (flask.palletsprojects.com) Django – A high-level Python Web framework that encourages rapid development and clean, pragmatic design. (DjangoProject.com) PyPi – The Python Package Index (PyPI) is a repository of software for the Python programming language. (pypi.org) Ten Reasons To Use Python (cuelogic.com) Top 10 Reasons Why Python is So Popular With Developers in 2021 (upgrad.com) Python – 12. Virtual Environments and Packages (docs.python.org) Python's virtual environments have been mentioned as a Tip of the Week twice: first during episode 102, Why Date-ing is Hard and again during episode 140, The DevOps Handbook – Enabling Safe Deployments. (/episode102, /episode140) PEP 8 — Style Guide for Python Code (python.org) The Gary Gnu Intro With Knock Knock – The Great Space Coaster (YouTube) Datadog has a blog article for everything: Tracing asynchronous Python code with Datadog APM (Datadog) Ho

We step back to reflect on what we learned from our first game jam, while Joe's bathroom is too close and Allen taught Michael something (again). Stop squinting to read this via your device's podcast player. This episode's show notes can be found at https://www.codingblocks.net/episode151, where you can join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard. DataStax – Sign up today to get $300 in credit with promo code CODINGBLOCKS and make something amazing with Cassandra on any cloud. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says What is your favorite lesson learned from Game Jam? Take the survey at: https://www.codingblocks.net/episode151. News We appreciate the new reviews, thank you! iTunes: ddausdd Audible: devops.rob Kinesis Advantage 2 Full Review after Heavy Usage (YouTube) Game Jam Tips Aim for the browser and to be embedded. Be careful sharing your custom URL when hosting somewhere other than the game jam as it splits your traffic and likely, your feedback. Time management is super important. Be realistic about how much time you have. You'll be tired by the end! Start with the Game Loop. Try to always be playable. Aim small and prioritize the "must haves". Know what you want to learn. Judge your game against what you can do. Beware of graphics and animations! Inspiration is fine, but it can become a sinkhole. Recall from the above tips about time management and focusing on a playable game. Play into the theme. Or don't. Use tools, asset stores, and libraries, such as Tiled, PyGame, Photoshop, and/or Butler, to simplify your effort and make maximum use of your time. Consider teamwork. Borrowing ideas is fine. Keep your "elevator pitch" in mind, and evolve it. Publish early and save energy for playing. Save time to write up your game's description, take video/screenshots, etc. for the submission. Keep your game testable by having a dev mode and/or the ability to initialize a certain game state. Think about the player over and over and over. How do you teach them the game's mechanics, physics, when the game is over, etc. And again, save time and energy for publishing your game, as well as, playing and rating other's games. Resources We Like The Coding Blocks Game Jam 2021 submissions (itch.io) Tip of the Week Sign up for a game jam! (itch.io) Use -A number, -B number, or -C number to include context with your next grep output. (gnu.org) -A number will print number lines after the match. -B number will print number lines before the match. -C number will print number lines before and after the match. Add your Git commit to your Docker images as a label like: docker build --tag 1.0.0.1 --label git-commit=$GIT_COMMIT . Where $GIT_COMMIT might be something like: GIT_COMMIT=$(git rev-parse HEAD) or GIT_COMMIT=$(git rev-parse --short HEAD) if you only want to use the abbreviated commit ID. In Jenkins, you can use ${env.GIT_COMMIT} to get the current commit ID that the current build is building,

We discuss all things open-source, leaving Michael and Joe to hold down the fort while Allen is away, while Joe's impersonations are spot on and Michael is on a first name basis, assuming he can pronounce it. This episode of the Coding Blocks podcast is about the people and organizations behind open-source software. We talk about the different incentives behind projects, and their governance to see if we can understand our ecosystem better. This episode's show notes can be found at https://www.codingblocks.net/episode150, if you're reading this via your podcast player. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard. Linode – Sign up for $100 in free credit and simplify your infrastructure with Linode's Linux virtual machines. Survey Says Which company has the best open source projects? Take the survey at: https://www.codingblocks.net/episode150. News We appreciate the new reviews, thank you! iTunes: @k_roll242, code turtle Upcoming Events: Joe is presenting at the San Diego Elastic Meetup Tuesday January 19th 2021 #CBJam is right around the corner, January 21st to 24th Subscribe to our Coding Blocks YouTube channel for all the upcoming keyboard reviews You Thought You Knew OSS Q: What do most developers think about when they think of "open-source" software? Q: Is the formal definition more important than the general perception? Formal Definitions of Open-Source opensource.org: Open source software is made by many people and distributed under an OSD-compliant license which grants all the rights to use, study, change, and share the software in modified and unmodified form. Software freedom is essential to enabling community development of open source software. opensource.com: Open source commonly refers to software that uses an open development process and is licensed to include the source code Pop Quiz, who created…? C: Dennis Ritchie Linux: Linus Torvolds Curl: Daniel Sternberg Python: Guido von Rossum JavaScript: Brandon Eich (Netscape) Node: Ryan Dahl Java: James Gosling (Sun) Git: Linus Torvalds C#: Microsoft (Anders Hejlsberg) Kubernetes: Google Postgres: Michael Stonebraker React: Facebook Rust: Mozilla Chromium: Google Flutter: Google TypeScript: Microsoft Vue: Evan You Q: It seems like most newer projects (with the exception of Vue) are associated with corporations or foundations. When and why did that change? GitHub Star Distribution Q: What are the most popular projects? Who were they made for, and why? https://github.com/EvanLi/Github-Ranking Who uses open-source software? There are a lot of stats and surveys…none great All surveys and stats agree that open-source is on the rise You kinda can't not use open-source software. Your OS, tools, networking hardware, etc all use copious amounts of open-souce software. Individuals Many (most) smaller libraries are written and maintained by individual authors, and have few or no contributors Some large / important libraries have thousands of contributors 10 most contributed GitHub projects in 2019 VS Code has almost 20k contributors Flutter has 13k contributors Kubernetes and Ansible have around 7k Q: Why do individuals create open source? What do they get out of it? Corporations A lot of corporate "open source" that are utilities or tools for working with those companies (ie: Azure SDK) Many open source projects are stewarded by a single company (Confluent, Elastic, MongoDB) Many open source projects listed below are now run by a foundation Let's look at some of the most prominent projects that were started by corporations. Note: many of these projects came in through acquisitions, and many have since been donated to foundations. Microsoft Maybe the biggest? Maybe? .NET, Helm, TypeScript, Postgres, VS Code, NPM, GitHub https://opensource.microsoft.com/ https://en.wikipedia.org/wiki/Microsoft_and_open_source Google Kubernetes, Angular, Chromium, Android, Go, Dart, Protobuff, TensorFlow, Flutter, Skaffold, Spinnaker, Polymer, Yeoman https://opensource.google/projects/explore/featured Facebook React, PyTorch, GraphQL, RocksDB, Presto, Jest, Flux https://opensource.facebook.com/projects Amazon Well, lots of toolkits and sdks for AWS… Oracle Java, MySQL https://developer.oracle.com/open-source/ Focused Corporations Sometimes a company will either outright own, or otherwise build a business centered around a technology. These companies will typically offer services and support around open-source projects. DataStax Elasticsearch Canonical MongoDB Q: Why do corporations publish open-source software? What do they get out of releasing projects? Foundations Foundations are organizations that own open-source projects Foundations have many different kinds of governance models, but generally they are responsible for things like… code stewardship (pull requests, versions, planning, contributors, lifecycle, support, certification*) financial support (domains, hosting, marketing, grants) legal issues (inc

We start off the year discussing our favorite developer tools of 2020, as Joe starts his traditions early, Allen is sly about his résumé updates, and Michael lives to stream. For those that read the show notes via their podcast player and find themselves wondering where they can find these show notes on their computer, the answer is simple: https://www.codingblocks.net/episode149. Check it out and join the discussion. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard. Survey Says Which hand do you type the 6 key with? Take the survey at: https://www.codingblocks.net/episode149. News Thank you for the new reviews we received this holiday! iTunes: larsankile, 0xACE, Tbetcha33, The_Shrike_, shineycloud Watch Joe speak at the virtual San Diego Elastic Meetup, Tuesday, January 19, 2021 at 5:00 PM PST, where he is talking about Easy Local Development with Elastic Cloud on Kubernetes using Skaffold. Signup at community.elastic.co. Join our game jam January 21 – 24, 2021 and let's make games! The Top Tools of 2020 K9s – A terminal UI to interact with your Kubernetes cluster. Popeye – A utility that scans a live Kubernetes cluster and reports potential issues with deployed resources and configurations. All things JetBrains, especially IntelliJ IDEA, PyCharm, and DataGrip. Amazing, cross-platform tools that developers from all tech stacks can use. Skaffold – Handles the workflow for building, pushing, and deploying your application within Kubernetes, allowing you to focus on writing your code. Kustomize – A template-free way to customize Kubernetes application configuration, built into kubectl. Oh My Zsh – An open source, community-driven framework for managing your zsh configuration. Netlify – Unites everything teams need to build and run dynamic web experiences, from preview to production, using an intuitive git-based workflow and a powerful serverless platform. Jira StopWatch – A desktop tool for recording time spent on different Jira issues. Lens – The Kubernetes IDE, Lens provides full situational awareness for everything that runs in Kubernetes, lowering the barrier of entry for those just getting started with Kubernetes, while radically improving productivity for the experienced users. Zoom – Keeping you connected wherever you are. Hands down, the best video calling software out there for screen sharing; you can actually read the other person's screen! Prometheus – Power your metrics and alerting with a leading open-source monitoring solution. Grafana – The world's most popular technology used to compose observability dashboards. Visual Studio Code – Code editing, redefined. Free and built on open source. Runs everywhere. Favorite VS Code extensions: Bracket Pair Colorizer – Matching brackets can be easily identified in color. GitLens – Supercharges the Git capabilities built into Visual Studio Code. Rainbow CSV – Highlight CSV files and run SQL-like queries. Cmder – A software package created out of pure frustration over the absence of nice console emulators on Windows. Unraid – Take control of your data, media, applications, and desktops, using just about any combination of hardware. Resources We Like Streaming: Debugging C# in Kubernetes and Skaffold vs Kustomize (YouTube) Tip of the Week Challenging projects every programmer should try (utk.edu) More challenging projects every programmer should try (utk.edu) Duck DNS – A free dynamic DNS hosted on AWS (duckdns.org) Honorable DNS mentions: 8.8.8.8/8.8.4.4 – Google Public DNS (Wikipedia) 9.9.9.9/9.9.9.10/9.9.9.11 – Quad9 (Wikipedia) 1.1.1.1/1.1.1.2/1.1.1.3 – Cloudflare's DNS (Wikipedia) MIT's Missing Semester of CS Education: The Missing Semester of Your CS Education (mit.edu) Missing Semester IAP 2020 (YouTube) 2020 Lectures (mit.edu)

It's the end of 2020. We're all tired. So we phone it in for the last episode of the year as we discuss the State of the Octoverse, while Michael prepared for the wrong show (again), Allen forgot to pay his ISP bill and Joe's game finished downloading. In case you're wondering where you can find these show notes in all there 1:1 pixel digital glory because you're reading them via your podcast app, you can find them at https://www.codingblocks.net/episode148, where you can also join the conversation. Sponsors Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. xMatters – Sign up today to learn how to take control of your incident management workflow and get a free xMatters t-shirt. Survey Says What's your favorite Christmas movie? Take the survey at: https://www.codingblocks.net/episode148. News Joe will be speaking at the virtual San Diego Elastic Meetup, Tuesday, January 19, 2021 at 5:00 PM PST, talking about Easy Local Development with Elastic Cloud on Kubernetes using Skaffold. Signup at community.elastic.co. We're hosting a game jam January 21 – 24, 2021 – let's make games! Follow all of our upcoming events! (/events) Watch and learn with Joe and Michael as they dive into Kubernetes: Local Kubernetes dev with Helm and Skaffold (YouTube) Streaming: Debugging C# in Kubernetes and Skaffold vs Kustomize (YouTube) Resources We Like The 2020 State of the Octo-verse (GitHub) Curl turns 20, HTTP/2, QUIC, The Changelog. Episode #299 (changelog.com) Parent Driven Development, a podcast about parenting in tech (parentdrivendevelopment.com) Tip of the Week K9s, a terminal UI to interact with your Kubernetes clusters (GitHub) Popeye, a utility that scans a live Kubernetes cluster and reports potential issues with deployed resources and configurations. (popeyecli.io) IPython, a powerful interactive shell and so much more. (ipython.org) The Google Authenticator app now makes it super easy to export your two factor settings to another device. (App Store, Google Play) Dungeon Map Doodler, a free drawing tool that allows you to easily create maps for all your gaming needs. (dungeonmapdoodler.com)

We discuss the things we're excited about for 2021 as Michael prepared for a different show, Joe can't stop looking at himself, and Allen gets paid by the tip of the week. For those that aren't in the know, these show notes can be found at https://www.codingblocks.net/episode147. Stop by and join the conversation. Sponsors Command Line Heroes – A podcast that tells the epic true tales of developers, programmers, hackers, geeks, and open source rebels who are revolutionizing the technology landscape. Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. xMatters – Sign up today to learn how to take control of your incident management workflow and get a free xMatters t-shirt. Survey Says What are the least amount of bits (or smallest data type) your annual salary, in whole dollars, could fit in? Take the survey at: https://www.codingblocks.net/episode147. News We really appreciate the new reviews. iTunes: 8BitToaster, NotResp, masterull We're hosting a game jam January 21 – 24, 2021 – let's make games! Follow all of our upcoming events! (/events) Who's Excited about What Joe Interactive online/streaming events DevOps and SRE technologies Python Game development Allen .NET 5 DevOps technologies Kubernetes Game Jams Big Data Video content creation Presentations IoT Machine Learning Michael Kubernetes all the things Kotlin Resources We Like 15 Kubernetes Tools For Deployment, Monitoring, Security, & More (phoenixNAP.com) Kubernetes Training and Certification (kubernetes.io) We're sad to see you go, Azure Notebooks (notebooks.azure.com) At least they list some alternatives. Announcing .NET 5.0 (devblogs.microsoft.com) Must Buys Price Description $17.99 Kasa Smart HS220 Dimmer Switch by TP-Link (Amazon, Best Buy) $17.46 Real-World Machine Learning (Amazon) Tip of the Week Manning Publications has a lot of their books available in audio form on Audible. Top 9 companies that are hiring software engineers to work remotely (HackReactor.com) iTerm2 – A terminal emulator for macOS that does amazing things. (item2.com) Use CMD+SHIFT+. to edit the command being pasted before running it. Some helpful tips for the holiday season: Capital One Shopping: Save in seconds (chrome web store) Automatically find and apply coupon codes with Honey. (chrome web store) Use PayPal Key as a virtual credit card to use your PayPal account anywhere credit cards are accepted. (PayPal) Use Privacy to create single or limited use credit cards (privacy.com) Port forward services from your Kubernetes cluster for external access to debug and test, like kubectl port-forward svc/svc-name 7000:8000. Use Port Forwarding to Access Applications in a Cluster (kubernetes.io)

We learn all the necessary details to get into the world of developer game jams, while Michael triggers all parents, Allen's moment of silence is oddly loud, and Joe hones his inner Steve Jobs. If you're reading these show notes via your podcast player and wondering where you can find them in your browser, well wonder no more. These show notes can be found at https://www.codingblocks.net/episode146 in all their 8-bit glory. Check it out and join the conversation. Sponsors Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. xMatters – Sign up today to learn how to take control of your incident management workflow and get a free xMatters t-shirt. Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard. Survey Says What kind of game do you want to make? Take the survey at: https://www.codingblocks.net/episode146. News Thank you to everyone that left us a new review! iTunes: AbhishekN12, shkpstrbtorurn, Herkamer's dad, Bamers22 Stitcher: Pour one out for Stitcher reviews, as sadly, they no longer have them. 8( Follow our upcoming events! (/events) Unlimited Google Photos' … for a limited time. (blog.google) Woo-hoo! We cracked the top 15 in the Apple Podcasts Technology category somehow! Is Game Dev your Jam? What are Game Jams? A timed challenged to create and publish games. Similar to a musical jam, people bring in their own perspectives and skills, and making new stuff for the world Did you know there is an International Game Jam conference? When's the next game jam? Popular Game Jams Ludum Dare (ldjam.com) An online event where games are made from scratch in a weekend. Check it out every April and October! Global Game Jam (https://globalgamejam.org) Empowers individuals worldwide to learn, experiment, and create together through the medium of games. January 2020, they had 934 locations in 118 countries create 9,601 games in one weekend! GGJ 2021 is scheduled for January 29-31, 2021. 7DRL (7drl.com) The 7DRL Challenges are NOT about being a fast coder, but rather proving you can release a finished, playable roguelike to the world. itch.io (itch.io) itch.io is a simple way to find and share indie games online for free. GMTK is 48hours, with 18k entrants. Anyone can instantly create and host a jam. 109,365 games have been created for jams hosted on itch.io. Why Should You do a Game Jam? Meet new people, see how they solved similar problems. Learn something new. Maybe you'll love it? Great for your GitHub, blog, Twitch, etc. content. Maybe make millions of $$$. Step 1. enter game jam, Step 2. … something about making a game …, and Step 3. profit! How can you Game Jam? Check Indie Game Jams for a jam that interests you. No need to take time off, just do what you can. You can even make your own jams. Popular Tools Unity Unreal Godot Game Maker RPG Maker App Game Kit Resources We Like Indie Game Jams (indiegamejams.com) International Conference on Game Jams (indiegamejams.com) Ludum Dare (ldjam.com) Ludum Dare 47 event stats (ldjam.com) Global Game Jam Online (globalgamejam.org) What is Global Game Jam? (YouTube) The 7DRL Challenge (7drl.com) 7DRL Challenge 2020 (7drl.com) itch.io Game Jams on itch.io GMTK Game Jam 2020 (itch.io) GMTK Game Jam 2020 submissions (itch.io) GMTK Game Jam 2020 Winners (gmtk.itch.io) Stay Safe! Jam submissions (itch.io) 10 Awesome Game Jam Success Stories (gamesparks.com) All Your Database Are Belong to Us (episode 13) Tip of the Week Some places to learn python (pyatl.dev) JSONPath StatusBar (marketplace.visualstudio.com) An awesome curated list of chaos engineering resources (GitHub) Ultra portable 13″ laptops that Michael has his eye one: Mentioned on-air Description Starting Price Dell XPS 13 Developer Edition (Dell) $1,049.00 Razer Blade Stealth 13 (Best Buy) online/clearance at $1,299.99 (seen for less in the store) Razer Book 13 (Razer) $1,199.99 Honorable (and/or forgotten) mentions Starting Price Description $999.00 Samsung Galaxy Book Flex 13.3″ (Amazon) $999.99 Apple MacBook Air 13.3″ with Apple M1 Chip (Amazon)

We wrap up our deep dive into The DevOps Handbook, while Allen ruined Halloween, Joe isn't listening, and Michael failed to… forget it, it doesn't even matter. If you're reading this via your podcast player, this episode's full show notes can be found at https://www.codingblocks.net/episode145 where you can join the conversation, as well find past episode's show notes. Sponsors Command Line Heroes – A podcast that tells the epic true tales of developers, programmers, hackers, geeks, and open source rebels who are revolutionizing the technology landscape. Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription. xMatters – Sign up today to learn how to take control of your incident management workflow and get a free xMatters t-shirt. Survey Says How often _should_ you update your resume? How often _do_ you update your resume? Take the surveys at: https://www.codingblocks.net/episode145. News Thank you to everyone that left us a new review! iTunes: AbhishekN12, Streichholzschächtelchen Mann Stitcher: emirdev Zoom introduces the Podtrak P8. (zoomcorp.com) Wrapping up The Third Way Use Chat Rooms and Bots to Automate and Capture Organizational Knowledge Chat rooms have been increasingly used for triggering actions. One of the first to do this was ChatOps at GitHub. By integrating automation tools within the chat, it was easy for people to see exactly how things were done. Everyone sees what's happening. Onboarding is nice because people can look through the history and see how things work. This helps enable fast organizational learning. Another benefit is that typically chat rooms are public. so it creates an environment of transparency. One of the more beneficial outcomes was that ops engineers were able to discover problems quickly and help each other out more easily. "Even when you're new to the team, you can look in the chat logs and see how everything is done. It's as if you were pair-programming with them all the time." Jesse Newland Automate Standardized Processes in Software for Re-Use Often times developers document things in wikis, SharePoint systems, word documents, excel documents, etc., but other developers aren't aware these documents exist so they do things a different way, and you end up with a bunch of disparate implementations. The solution is to put these processes and standards into executable code stored in a repository. Create a Single, Shared Source Code Repository for Your Entire Organization This single repository enables quick of sharing amongst an entire organization. In 2015, Google had a single repository with over 1 billion files over 2 billion lines of code. This single repository is used by every software engineer and every product. This repository doesn't just include code, but also: Configuration standards for libraries, infrastructure and environments like Chef, Ansible, etc., Deployment tools, Testing standards and tools as well as security, Deployment pipeline tools, Monitoring and analysis tools, and Tutorials and standards. Whenever a commit goes in, everything is built from code in the shared repo: no dynamic linking. This ensures everything works with the latest code in the repository. By building everything off a single source tree, Google eliminates the problems you encounter when you use external dependency management systems like Artifactory, Nuget, etc. Spread Knowledge by Using Automated Tests as Documentation and Communities of Practice Sharing libraries throughout an organization means you need a good way of sharing expertise and improvements. Automated tests are a great way to ensure things work with new commits and they are self-documenting. TDD turns tests into up-to-date specifications of a system. Want to know how to use the library? Take a look at the test suites. Ideally you want to have one group responsible for owning and supporting a library. Ideally you only ever have one version of that code out in production. It will contain the current best collaborative knowledge of the organization. The owner is also responsible for migrating each consumer of the library to the next version. This requires the consumers to have a good suite of automated testing as well. Another great use of chat rooms is to have one for each library. Design for Operations Through Codified Non-Functional Requirements When developers are responsible for incident response in their deployed applications, their applications become better designed for operations. As developers are involved in non-functional requirements, we design our systems for faster deployment, better reliability, the ability to detect problems, and allow for graceful degradation. Some of these non-functionals are: Production telemetry, Ability to track dependencies, Resilient and gracefully degrading services, Forward and backward compatibility between versions, Ability t

It's our favorite time of year where we discuss all of the new ways we can spend our money in time for the holidays, as Allen forgets a crucial part, Michael has "neons", and Joe has a pet bear. Reading this via your podcast player? If so, you can find this episode's full show notes at https://www.codingblocks.net/episode144, where you can join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard. Teamistry – A podcast that tells the stories of teams who work together in new and unexpected ways, to achieve remarkable things. Survey Says Which do you want the most? Take the survey at: https://www.codingblocks.net/episode144. News Thank you for the latest new review! Stitcher: wsha And I'm Spent Allen's List for the Content Creators Price Description $149.99 Elgato Stream Deck (Amazon) $69.99 TDisplay Capture Card (Amazon) $329.00 Oktava MK-012 Small Diaphragm Condenser mic (Amazon) $199.99 Zoom Podtrak P4 (Best Buy) $7.00 Oktava WS-012 foam Wind-Screen for MK-012 (Amazon) $16.99 LYRCRO Microphone Shock Mount Holder (Amazon) $274.99 product (TechSmith) $26.99 Kasa Smart Plugs 4 Pack (Amazon) $14.99 3 prong extension cables 1′ (Amazon) $29.99 Toazoe Articulating 11″ Arm (Amazon) $12.99 Mounting pole (Amazon) $14.99 Desk mounting bracket for pole (Amazon) $199.99 Elgato Key Light (Amazon) $90.00 Aputure Amaran MC RGBWW (Amazon) $186.20 Silverstone ATX CS380B (Amazon) $139.99 Ryzen 5 3400G (Amazon) $99.99 Silicon Power 32GB 3200Mhz (Amazon) $217.99 WD Easy Store 12TB (Amazon) $44.99 Thermaltake 500w 80+ (Amazon) $365.00 ZSA Moonlander (ZSA) Michael's List to Pimp Your Desk Description Price Apple iPad Pro (12.9-inch, WiFi, 256GB) (Amazon) $1,079.00 amFilm (2 Pack) Glass Screen Protector for iPad Pro 12.9 inch (2020 & 2018 Models) (Amazon) $12.99 ProCase iPad Pro 12.9 Case 4th Generation 2020 & 2018 (Amazon) $21.99 Apple AirPods with Charging Case (Wired) (Amazon) $129.00 Honorable mention: Apple AirPods Pro (Amazon) $219.00 Philips Hue 555334 BLE Lightstrip LED Light strip, 2m / 6ft Base Kit (Amazon) $79.99 Philips Hue Indoor Motion Sensor for Smart Lights (Amazon) $38.95 Elgato Stream Deck (15 Keys) (Amazon) $149.99 Knox Microphone Shock Mount for Audio-Technica ATR2100-USB and Samson Q2U (Amazon) $19.99 On-Stage Foam Ball-Type Microphone Windscreen (Amazon) $2.99 ZSA Moonlander (ZSA) $365.00 Glorious 3XL Extended Gaming Mouse Mat/Pad (Amazon) $49.99 Razer Gaming Mouse Bungee v2 (Amazon) $19.99 ALIENWARE AW3420DW Curved 34 Inch WQHD 3440 X 1440 120Hz Monitor (Amazon) $1,029.99 Honorable mention:Dell AW3418DW Alienware 34 Curved Gaming Monitor (Amazon) n/a Microsoft Xbox Series X (Amazon) $499.99 Castle Grayskull … Description Price Tilted Nation RGB Headset Stand and Gaming Headphone Display with Mouse Bungee Cord Holder with USB 3.0 HUB (Amazon) n/a Joe's List to Make Bank Well, first there's *the* chair … Price Description $3,299.00 Cluvens Scorpion Computer Cockpit (Cluvens) Flippa – Buy an online business, become an acquisition entrepreneur, and invest in digital real estate. Start Engine – Invest and buy shares in startups and small businesses. (DEFUNCT) Gratipay – An open source startup helping open source projects. Liberapay – A recurrent donations platform. Fund the creators and projects you appreciate. Wefunder – Back founders solving the problems you care about and help their startups grow. Crowdfunder – Where entrepreneurs and investors meet. Fundrise – The future of real estate investing. Localstake – Connecting businesses and local investors. SeedInvest – Own a piece of your favorite startups. Resources We Like Indoor Boom Microphones: Oktava MK-012, RODE NT5, Audio Technica AT4053b (YouTube) The Aputure MC Video LED Light is Amazing! (YouTube) Tip of the Week The Docker image for Google Cloud SDK is the easiest way to interact with the Google cloud. (hub.docker.com) Play Hades. Play it now. (Steam) Use git log --reverse to see the repo history from the beginning. (git-scm.com)

We dive into the benefits of enabling daily learning into our processes, while it's egregiously late for Joe, Michael's impersonation is awful, and Allen's speech is degrading. This episode's show notes can be found at https://www.codingblocks.net/episode143, for those reading this via their podcast player, where you can join the conversation. Sponsors Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard. Teamistry – A podcast that tells the stories of teams who work together in new and unexpected ways, to achieve remarkable things. Survey Says How often do you change jobs? Take the survey at: https://www.codingblocks.net/episode143. News Thank you to everyone that left us a new review! iTunes: John Roland, Shefodorf, DevCT, Flemon001, ryanjcaldwell, Aceium Stitcher: Helia Allen saves your butt with his latest chair review on YouTube. Enable and Inject Learning into Daily Work To work on complex systems effectively and safely we must get good at: Detecting problems, Solving problems, and Multiplying the effects by sharing the solutions within the organization. The key is treating failures as an opportunity to learn rather than an opportunity to punish. Establish a Just, Learning Culture By promoting a culture where errors are "just" it encourages learning ways to remove and prevent those errors. On the contrary, an "unjust" culture, promotes bureaucracy, evasion, and self-protection. This is how most companies and management work, i.e. put processes in place to prevent and eliminate the possibility of errors. Rather than blaming individuals, take moments when things go wrong as an opportunity to learn and improve the systems that will inevitably have problems. Not only does this improve the organization's systems, it also strengthens relationships between team members. When developers do cause an error and are encouraged to share the details of the errors and how to fix them, it ultimately benefits everyone as the fear of consequences are lowered and solutions on ensuring that particular problem isn't encountered again increase. Blameless Post Mortem Create timelines and collect details from many perspectives. Empower engineers to provide details of how they may have contributed to the failures. Encourage those who did make the mistakes to share those with the organization and how to avoid those mistakes in the future. Don't dwell on hindsight, i.e. the coulda, woulda, and shoulda comments. Propose countermeasures to ensure similar failures don't occur in the future and schedule a date to complete those countermeasures. Stakeholders that should be present at these meetings People who were a part of making the decisions that caused the problem. People who found the problem. People who responded to the problem. People who diagnosed the problem. People who were affected by the problem. Anyone who might want to attend the meeting. The meeting Must be rigorous about recording the details during the process of finding, diagnosing, and fixing, etc. Disallow phrases like "could have" or "should have" because they are counterproductive. Reserve enough time to brainstorm countermeasures to implement. These must be prioritized and given a timeline for implementation. Publish the learnings and timelines, etc. from the meeting so the entire organization can gain from them. Finding more Failures as Time Moves on As you get better at resolving egregious errors, the errors become more subtle and you need to modify your tolerances to find weaker signals indicating errors. Treat applications as experiments where everything is analyzed, rather than stringent compliance and standardization. Redefine Failure and Encourage Calculated Risk Taking Create a culture where people are comfortable with surfacing and learning from failures. It seems counter-intuitive, but by allowing more failures this also means that you're moving the ball forward. Inject Production Failures The purpose is to make sure failures can happen in controlled ways. We should think about making our systems crash in a way that keeps the key components protected as much as possible i.e. graceful degradation. Use Game Days to Rehearse Failures "A service is not really tested until we break it in production." Jesse Robbins Introduce large-scale fault injection across your critical systems. These gamedays are scheduled with a goal, like maybe losing connectivity to a data center. This gives everyone time to prepare for what would need to be done to make sure the system still functions, failovers, monitoring, etc. Take notes of anything that goes wrong, find, fix, and retest. On gameday, force an outage. This exposes things you may have missed, not anticipated, etc. Obviously the goal is to create more resilient systems. Resources We Like The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (Amazon) The Phoenix Project: A Novel about IT, DevOps, and Helping