Oxide and Friends

Jordan Hendricks joined Bryan and Adam to talk about her work virtualizing time--particularly challenging when migrating virtual machines from one physical machine to another!We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from June 12th, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague Jordan Hendricks.The (lightly edited) live chat from the show:DanCrossNYC: The TSC ticks at a fixed rate now days, regardless of voltage scaling on the CPU.jbk: just x86 doesn't provide a consistent want to determine what the rate isjbk: (I guess some chips will tell you via CPUID, but I've yet to actually encounter such chips)jbk: some hypervisors will tell you via an MSRzorg24: Looks the Linux kernel docs have some documentation on the x86 TSC and PIT https://www.kernel.org/doc/html/next/virt/kvm/x86/timekeeping.htmlDanCrossNYC: CPUID or an MSR, but yeah, most systems sample over a fixed interval (determined by another time source) to figure it out.jbk: no, versus some other present component that allows you to measure the frequencyDanCrossNYC: No, the PIT or HPET or something.jbk: https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/tscc_pit.c?r=236cb9a8jbk: is how it uses the PITjbk: (the HPET code needs to improve it's accuracy, so it's only used when the PIT isn't there at the moment)jbk: some Intel NUCs have no PITjbk: so HPET is the only optionbcantrill: https://github.com/illumos/illumos-gate/commit/717646f7112314de3f464bc0b75f034f009c861eDanCrossNYC: Two big ones: system maintenance without disturbing guest workloads, and also load balancing across a rack."Sevan: ah, thanks.https://github.com/illumos/illumos-gate/blob/717646f7112314de3f464bc0b75f034f009c861e/usr/src/test/bhyve-tests/tests/common/common.c#L166"bcantrill: https://github.com/oxidecomputer/tsc-simulator/tree/masterDanCrossNYC: The guest may well be running NTP itself.iangrunert: I assume you could also check that NTP is alive / has synced recently before doing a migration right?aka_pugs: Do people use IEEE 1588/PTP in datacenters? Maybe finance wackos?zorg24: also it might be tricky to check if NTP synced recently if it is happening in usermodeiangrunert: Might've missed this - is it just the hypervisor that has to run NTP recently or the VM as well?saone: I believe it was just the hypervisorDanCrossNYC: The host.DanCrossNYC: A guest may or may not; that's up to the guest.jbk: but IIUC, if the guest IS running NTP, then the host definitely needs it to avoid any time warpsDanCrossNYC: Yup.DanCrossNYC: Fortunately, there's a bit of an out for the blackout window during migration: SMM mode can effectively pause a machine for an indefinite period of time.DanCrossNYC: We don't USE SMM anywhere, but robust systems software kinda needs to handle the case where the machine goes out to lunch for a minute.zorg24: 🙌 hooray for hardware with no SMM useDanCrossNYC: We have done everything we can to turn it off.ahl: https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.mdahl: https://github.com/oxidecomputer/propolisearltea: it worked so well I almost thought the VM didn't migrate 😅saone: It's easy to forget that there's a world outside the cloud, but edge deployments that have physical peripherals hooked up need to maintain those connections to peripherals; migrating those peripherals to cloud environments and managing that integration has been a big challenge for my group.iangrunert: https://signalsandthreads.com/clock-synchronization/ Good listen about clock synchronization and PTP in the ""finance weirdos"" world. MiFID 2 time sync requirements require timestamping key trading event records to within 100 microseconds of UTC.jhendricks: a bit belated, but the propolis side of these changes: https://github.com/oxidecomputer/propolis/commit/7ed480843d3b5cfd9fd07dce41772f8eac4e9171saethlin: The calvalry??saethlin: Are we just going to let that slidesaethlin: Is this a pronunciation situation againzorg24: not the first time I've heard it pronounced that way 🤷saethlin: Well maybe it's me learning this timeDanCrossNYC: CalvaryDanCrossNYC: That's the religious thing.ahl: https://github.com/illumos/illumos-gate/blob/0c5967db436935325af441af2b27d337f4e64af5/usr/src/uts/common/os/cyclic.c#L44zooooooooo: thought this was rust typescript at first 😳DanCrossNYC: Dunno... I missed it. 🙂ahl: * Starting in about 1994, chip architectures began specifying high resolution * timestamp registers. As of this writing (1999), all major chip families * (UltraSPARC, PentiumPro, MIPS, PowerPC, Alpha) have high resolution * timestamp registers, and two (UltraSPARC and MIPS) have added the capacity * to interrupt based on timestamp values. These timestamp-compare registers * present a time-based interrupt source which can be reprogrammed arbitrarily * often without introducing error. Given the low cost of implementing such a * timest

Bryan and Adam are joined by Ashley Williams to talk about open source governance... and the recently, and various stumblings of the Rust project leadership.

Bryan and Adam are joined by Jonathan and Jignesh from Samtec to discuss working together to build the Oxide Rack. We've all seen bad vendors--what does it mean to be a great partner? Also: silicon photonics are (still!) just 18 months away!

Bryan and Adam are joined by Oxide colleagues Arjen, Matt, John, and Nathaneal to talk about the management network--the brainstem of the Oxide Rack. Just as it ties together so many components, this episode ties together many many (many!) topics we've discussed in other episodes.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from May 8th 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Arjen Roodselaar, Matt Keeter, John Gallagher, and Nathanael Huffman.This built on work described in many previous episodes:Cabling the Backplane Prior to going all-in on a cabled backplane with blind-mated server sleds (i.e. no plugging, unplugging, mis-plugging network cables). We (Bryan) espoused an "NC-SI or bust" mantra... at least in part to avoid doubling the cable count. With the cabled backplane, the reasons for NC-SI disappeared (which let the many reasons against truly shine).The Pragmatism of Hubris in which we talk about our embedded operating system, Hubris (and it's companion debugger, Humility). Hubris runs on the service processors that are the main endpoints on the management network. Matt's work controlling the management network switch (the VSC7448) is in the context of Hubris, as is John's work communicating with the sleds over the management network.The Power of Proto Boards showed and told about the many small boards we've used in development. Several of those were purpose built for controlling and simulating parts of the management network.The Oxide Supply Chain Kate Hicks joined us to talk about the challenges of navigating the supply chain. Mentioned here in the context of "supply-chain-driven design": we designed around the parts we could procure! Tip: stay away from "automotive-quality" parts when the auto industry is soaking them all up.Holistic Boot in which we talked about how (uniquely!) Oxide boots from nothing to its operating system and services. Over the management network, we can drive server recovery by piping in a RAMdisk over the network and then (slowly) through the UART to the CPU.Get You a State Machine for Great Good Andrew joined us to talk about his work on a state-machine driven text-UI and its companion replay debugger. We mentioned this in the context of John replaying the long upload process in seconds rather than hours to fix a UI bug.Major components of the management networkMatt's VSC7448 dev kitMatt's remote tuning setup via webcamManagement network debuggingManagement network debugging

Erin Kissane joins Bryan and Adam to talk the new social network "Bluesky" through the lens of her blog post "Blue Skies Over Mastodon". Long-time friends of Oxide and social-media aficionados Time Bray and Steve Klabnik also helped shed light on technical and social aspects of the net network.Blue Skies Over Mastodon (with Erin Kissane and Tim Bray)We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from May 1st, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Erin Kissane and long-time acquaintances of the show Tim Bray and Steve Klabnik.Some of the topics we hit on, in the order that we hit them:Erin's blog post Blue Skies Over MastodonMastodon blog (5/1) A new onboarding experience on Mastodon]Tim's blog post from November Bye Twitter"Buy the rumor, sell the news"Hellthread"Skeet" is to "Tweet" is to "Toot" (aka "Publish")skyline.gayBluesky blog Composable ModerationLobstersPhanpySo you've been publically shamed by Jon RonsonIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

The Rust Foundation caused a fracas with their proposed new trademark rules. Bryan and Adam were lucky enough to be joined by Ashley Williams, Adam Jacob, and Steve Klabnik for an insightful discussion of open source governance and communities--in particular as applied to Rust.Rust Trademark: Argle-bargle or Foofaraw?We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from April 17th, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Ashley Williams, Adam Jacob, and Steve Klabnik.Some of the topics we hit on, in the order that we hit them:SuccessionThe Simpsons (explaining the title of this episode)The WireThe Wire at 20 PodcastThe Register: Rust Foundation Apologizes for Trademark PolicyJomboy (our aspiration)Ice WeaselPamela ChestekBryan's talk from Node Summit 2017: Platform as a Reflection of ValuesLinux Foundation form 990Rust Foundation BoardRust Foundation participation rulesIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam are joined by Doug Wibben and Robert Keith to talk about the mechanical design of the cabled backplane of the Oxide rack that allows for "blind-mated" server sleds--no network and power cables to plug, unplug, and mis-plug! Watch the chapter art for relevant pictures.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from April 3rd, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague, Robert Keith, and special guest, Doug Wibben.00:00 03:02 09:52 11:09 12:16 12:58 ...

Andrew Stone of Oxide Engineering joined Bryan, Adam, and the Oxide Friends to talk about his purpose-built, replay debugger for the Oxide setup textual UI. Andrew borrowed a technique from his extensive work with distributed systems to built a UI that was well-structured... and highly amenable to debuggability. He built a custom debugger "in a weekend"!Some of the topics we hit on, in the order that we hit them:tui-rsCrosstermThe reedline crateEpisode about the "Sidecar" switchElm time-travel debuggingReplay.ioDevtools.fm episode on Replay.ioAADEBUG conferenceCalifornia horse meat lawThe (lightly) edited live chat from the show:MattCampbell: I'm gathering that this is more like the fancy pseudo-GUI style of TUI, which is possibly bad for accessibilityahl: we are also building with accessibility in mind, stripping away some of the non-textual elements optionallyMattCampbell: oh, coolahl: Episode about the "Sidecar" switch: https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.mdMattCampbell: ooh! That kind of recording is definitely better for accessibility than a video.uwaces: Were you inspired by Elm? (The programming language for web browsers?)bcantrill: Here's Andrew's PR for this, FWIW: oxidecomputer/omicron#2682uwaces: Elm has a very similar model. They have even had a debugger that let you run events in reverse: https://elm-lang.org/news/time-travel-made-easybch: I’m joining late - 1) does this state-machine replay model have a name 2) expand on (describe ) the I/o logic separation distinction?ahl: http://dtrace.org/blogs/ahl/2015/06/22/first-rust-program-pain/zk: RE: logic separation in consensus protocols: the benefit of seperating out the state machine into a side-effect free function allows you to write a formally verified implementation in a pure FP lang or theorem prover, and then extract a reference program from the proof.we're going to the zoo: lol i’m a web dev && we do UI tests via StorybookJS + snapshots of each story + snapshots of the end state of an interactionig: At that point you could turn the recording into an “expect test”. https://blog.janestreet.com/the-joy-of-expect-tests/we're going to the zoo: TOFU but for tests 🥰uwaces: Are you at all worried that you are replicating the horror that is the IBM 3270 terminal? — I have personal history programming on z/OS where the only interface is a graphical EBCDIC 3027 interface — the horror is that people write programs to interact with graphical window (assuming a certain size).ahl: https://docs.rs/serde/latest/serde/#data-formatsahl: SHOW NOTES Bryan as "semi-elderly" engineerMattCampbell: didn't Bryan write a blog post on this?MattCampbell: http://dtrace.org/blogs/bmc/2008/11/16/on-modalities-and-misadventures/uwaces: https://www.replay.ioahl: https://devtools.fm/episode/9ahl: e.g. https://altsysrq.github.io/proptest-book/intro.htmlwe're going to the zoo: https://github.com/AFLplusplus/LibAFLig: Are you using proptest, quickcheck, or something else?nickik: This really started with Haskell https://hackage.haskell.org/package/QuickCheck Its also cool that it does 'narrowing' meaning it will try to find an error, and then try to generate a simpler error case.endigma: how different is something like this from what go calls "fuzzing"Riking: Fuzzing does also have a minimization stepwe're going to the zoo: https://github.com/dubzzz/fast-checkRiking: Property-based testing tends to be structured differently in philosophy, while fuzzers are more aligned to "give you a bag of bytes"nickik: http://www.quviq.com/products/erlang-quickcheck/endigma: yeah I can tell its a different structure, but the overall goal seems similarwe're going to the zoo: they are nonexclusive approaches to testingpapertigers: I think Kelly was doing a bunch of tests at Joyent based on quick check and prop test. First time I encountered itwe're going to the zoo: libafl provides a #[derive(Arbitrary)] macro that will provide the correct values for a structuwaces: Lots of stuff in Rust existed first in Haskell (build.rs, quote!, Derive macros, Traits, ect….)…nixinator: https://tenor.com/view/%C3%B3culos-escuro-exterminador-terminator-arnold-schwarzenegger-gif-14440790we're going to the zoo: “what do these means” depends on who you ask lolwe're going to the zoo: fast-check is 🔥 for TypeScriptendigma: if the tested function is deterministic and the test is testing arbitrary input and testing against the result to be derivative in some way of the input function by some f(x), don't you end up re-implementing the tested function to provide the expected result? how does the author choose what properties of a system to test without falling into a "testing the test" pit?we're going to the zoo: Rust: “Here comes the Haskell plane!”nixinator: Isn’t rust == oxidationendigma: yesendigma: in a scientific sensenixinator: Iron oxide 🙂 lolnixinator: Very good!GeneralShaw: Is prop test a way of formal verification? Is it same/different?ahl: https://dl.acm.org/c

Bryan and Adam and the Oxide Friends take on GPT and its implications for software engineering. Many aspiring programmers are concerned that the future of the profession is in jeopardy. Spoiler: the Oxide Friends see a bright future for human/GPT collaboration in software engineering.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from March 20th, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers on MM DD included Josh Clulow, Keith Adams, Ashley Williams, and others. (Did we miss your name and/or get it wrong? Drop a PR!)Live chat from the show (lightly edited):ahl: John Carmack's tweetahl: ...and the discussionWizord: https://twitter.com/balajis/status/1636797265317867520 (the $1M bet on BTC, I take)dataphract: "prompt engineering" as in "social engineering" rather than "civil engineering"Grevian: I was surprised at how challenging getting good prompts could be, even if I wouldn't quite label it engineeringTronDD: https://www.aiweirdness.com/search-or-fabrication/MattCampbell: I tested ChatGPT in an area where I have domain expertise, and it got it very wrong.TronDD: Also interesting https://www.youtube.com/watch?v=jPhJbKBuNnAWizord: the question is, when will it be in competition with people?Wizord: copilot also can review code and find bugs if you ask it in a right wayag_dubs: i suspect that a new job will be building tools that help make training sets better and i strongly suspect that will be a programming job. ai will need tools and data and content and there's just a whole bunch of jobs to build tools for AI instead of peopleWizord: re "reading manual and writing DTrace scripts" I think it's possible, if done with a large enough token window.Wizord: (there are already examples of GPT debugging code, although trivial ones)flaviusb: The chat here is really interesting to me, as it seems to miss the point of the thing. ChatGPT does not and can not ever 'actually work' - and whether it works is kind of irrelevant. Like, the Jaquard Looms and Numerical Control for machining did not 'work', but that didn't stop the roll out.Columbus: Maybe it has read the dtrace manual 😉JustinAzoff: I work with a "long tail" language, and chatgpt sure is good at generating code that LOOKS like it might work, but is usually completely wrongclairegiordano: Some definite fans of DTrace on this showag_dubs: a thing i want to chat about is how GPT can affect the "pace" of software developmentsudomateo: I also think it's a lot less than 100% of engineers that engage in code review.Wizord: yes, I've had some good experience with using copilot for code reviewag_dubs: chatgpt is good at things that are already established... its not good at new things, or things that were just publishedWizord: very few people I know use it for the purpose of comments/docs. just pure codegen/boilerplayeschadbrewbaker: "How would you write a process tree with dtrace?" (ChatGPT4)#!/usr/sbin/dtrace -s BEGIN { printf(""%5s %5s %5s %s\n"", ""PID"", ""PPID"", ""UID"", ""COMMAND""); } proc:::exec-success { printf(""%5d %5d %5d %s\n"", pid, ppid, uid, execname); }TronDD: That's interesting as expensive, specialized code analysis tools have been varying level of terrible for a long timeJustinAzoff: I did an experiment before where I asked it to write me some php to insert a record into a database. so of course it generated code with sql injectionchiefnoah: It's ability seems to scale with how many times someone has done the exact thing you're trying to do beforeJustinAzoff: but then I asked if sql injection was bad, which it explained that it was. then I asked if the code it wrote me was vulnerable to sql injection. it then explained it wasColumbus: It misses empirical verification; forming a hypothesis, testing it, and learning from the result. There have been some attempts to implement this by feeding back e.g. command output into the promptJustinAzoff: so then the crazy part, I asked if sql injection was bad, why did it give me code that was vulnerable to sql injection. It the went on to say that the first thing it gave me was just for example purposesJustinAzoff: so no wonder people get into "prompt engineering" since it's clear that you need to do things like ask for code that does something, and is secure, high performance, does not have buffer overflows or sql injection vulnsMattCampbell: In my test case ("Write a Win32 UI Automation provider in C++"), all it did was produce plausible-sounding crapag_dubs: pattern matching over very very large data setsclairegiordano: Bryan just said this and I wanted to write it down, re GPT-3: "the degree that it changes its answers when you tell GPT-3 to think like someone else"JustinAzoff: or even just, "do that, but better"ag_dubs: i think a lot of the awe of gpt is recognizing how simple our own strategies are instead of how complex the AI's strategy ischadbrewbaker: "How would Bryan Cantrill re-write this script?" (ChatGPT4)

Eric Vishria of Benchmark and Oxide CEO, Steve Tuck, join Bryan and Adam to talk about Silicon Valley Bank, its role in the startup ecosystem, and the short- and long-term effects of its collapse.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from March 17th, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined special guests Eric Vishria and Steve Tuck.(Did we miss your name and/or get it wrong? Drop a PR!)Curated chat log from the show:davidf: Sharing this here because I loved every bit of it: My Startup Banking Story by Mitchell Hashimotoewen: 'The teller looks at the paper, then looks at me, then looks back at the paper, then asks ""Are you the HashiCorp guy?"" ' 😮 (Definitely agree that post looks relevant, and is worth reading; thanks for sharing. There's quite the impedance mismatch between ""traditional banking"" and ""startup"" approaches to things. Which I suspect in part explains how SVB was so widely used by startups.)"antranigv: Question: Are there any reasons why the US is behind in these banking things? all countries in the EU and developing countries have solved these problems decade(s) ago.statuscalamitous: my personal, barely informed take: we built this infra earlier, so we have more legacya172: It sounds like what SVB was providing that was so rare was a kind of business as a service.statuscalamitous: my favorite "scare a developer" story: the way ACH payments work. that's right, SFTP!antranigv: I think you mean FTPS? did they move to SFTP? 😄drkamoz: I think the opposite is also true, without the infra, Africa’s been very early to adopt mobile banking https://www.bbc.com/worklife/article/20131217-east-africa-a-mobile-banking-hubdrkamoz: Can you explain sweep funds?Eric Likness - carpetbomberz.com: 6 months of runway some place else. Not what Peter Thiel was telling people.antranigv: What was his response?arjenroodselaar: Eject! Eject!ahl: this was a fun summary: https://svbhallofshame.wordpress.com/ahl: https://www.fdic.gov/news/press-releases/2023/pr23016.htmlantranigv: This Venture Debt is intriguing, specially for startups who have a good background but are having a hard time... kinda? I guess?ahl: Acquired: Benchmark Part Iahl: Acquired: Benchmark Part II: The DinnerIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam are joined by a number of members of the Oxide networking team to talk about the networking software that drives the Oxide rack. It turns out that rack-scale networking is hard... and has enormous benefits!We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from February 27th, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers included Ryan Goodfellow, Levon Tarver, Ben Naecker, and Arjen Roodselaar.LinksIntel Tofino SeriesP4 (programming language) - Wikipediap4lang/p4c: P4_16 reference compileroxidecomputer/p4: A P4 compilerThe quote crate: Rust quasi-quotingRIFT WG - Routing In Fat Trees | IETF Community WikiHere's (much of) the live chat from the show:ahl https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.mdahl That's the Sidecar switch episodebcantrill https://p4.org/admchl What does "at line rate" mean?Riking Line rate = As fast as the packets could possibly come. 1Gbit, 10Gbit, 100Gbit, etcadmchl Do you need ASICs to hit that speed? I assume x86_64 is not going to be fast enough for these specialised operations?levon Yes, the Tofino 2 is the ASICbcantrill You need ASICsbnaecker Yes, you really can't do these kinds of operations on a general purpose CPU.rng_drizzt Yeah, you need specialized silicon here.JustinAzoff Right, also often across all ports at the same time in both direction. a 48 port 10gbps switch will have a line rate of 960gbps (10 ** 48 ** 2)duckman So the advantage is being able to offload compute to the switch?bnaecker Yes, and specifically that you can separate the data plane (operations on the packets) from the control plane (decisions about what operations to allow or make).tahnok What's TCAM?levon Ternary Content Addressable Memorybnaecker https://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMsryaeng Sure beats logging into a number of Cisco switches and making changes at the console.admchl This is my favourite episode in a long time, this is all really fascinating.rng_drizzt the first Sidecar episode was nearly 1.5 years ago 🤯 , right after we cut the first revlevon That episode blew my mindduckman This sounds like a big deal on the scale of ebpfduckman Or biggerbnaecker It is extremely useful for understanding the processing pipelines. As long as you only run single-packet integration tests 🙂od0 just want to go out and find things to write P4 code forJustinAzoff <@354365572554948608> yeah one way to think about that sort of thing is that xdp can be used to run little programs on a nic, where p4 is kind of like that, but running on effectively a nic with 48+ portsbcantrill https://github.com/oxidecomputer/p4SyntheticGate sidecar is the "codename" of our switch boxSyntheticGate "gimlet" is our server sledbcantrill https://github.com/oxidecomputer/propoliswmf So you have P4 and OPTE in the hypervisor at the same time?bnaecker OPTE is in the host kernel.arjenroodselaar The P4 runtime Ry described only exists in the test bed, where it high level simulates the switches. OPTE is part of the production environment.arjenroodselaar The rough difference between P4 and OPTE is that P4 works on individual packets without much concept of a session (so it can't reason about TCP streams, packet order etc, so no firewall like functionality), while OPTE aims to operate on streams of packets.JustinAzoff So you can run 100 VMs on a test system and wire them up to your virtual switch compiled by x4c?arjenroodselaar Correct.bcantrill OPTE == Oxide Packet Transformation Engineadmchl Gimlet?rng_drizzt Compute serverrng_drizzt The Sidecar switch is actually just a PCIe peripheral to a Gimlet.bnaecker The Gimlet managing the Sidecar is often called a "Scrimlet" for "Sidecar attached Gimlet"Riking and "how do i reconfigure this giant network without hosing my ability to reconfigure this giant network"ShaunO can identify with that - we seriously struggle to keep our own products inter-operating, let alone anyone else'slevon It can feel like a Sisyphean task.a172 Setup a much smaller/simpler network in parallel that is accessible from "not your network" that gets you to the management interface.levon It's a whole new world when you can look at the actual table definitions in P4rng_drizzt Owning all the layers here is immensely beneficiallevon Those DTrace probes have been very helpfulbnaecker Those probes turned out to be everywhere. They are are in: SQL queries, HTTP queries, log messages, Propolis hypervisor state, virtual storage system, networking protocol messages, the P4 emulator, and probably more that I'm forgetting about.levon For those unfamiliar with the DTrace tool, or the rationale behind leveraging DTrace over other tracing / debugging tools: https://www.cs.princeton.edu/courses/archive/fall05/cos518/papers/dtrace.pdfbcantrill https://github.com/oxidecomputer/progenitorahl some notes on rust codegen: https://github.com/ahl/codegen-templatearjenroodselaa

Yael Grauer joined Bryan, Adam, Steve Klabnik, and the Oxide Friends to talk about her recent Consumer Reports article on memory safety and memory safe languages. How do we inform the general public? How do we persuade practitioners and companies? Thanks for joining us, Yael!In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Yael Grauer, and Steve Klabnik.Some of the topics we hit on, in the order that we hit them (experiment in turning the show live-chat into notes):Nahum: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ if anyone wants to read up on the 3-2-1 Backup strategy. 👅Cyborus: can we get a link to the talk?Nahum: https://www.youtube.com/watch?v=Q9s2NxILBK8Nahum: https://digital-lab-wp.consumerreports.org/wp-content/uploads/2023/01/Memory-Safety-Convening-Report-.pdf via https://digital-lab-wp.consumerreports.org/2023/01/23/new-report-future-of-memory-safety/Nahum: https://en.wikipedia.org/wiki/Pegasus_(spyware)Cyborus: "can we talk" => "hey. you. have a panic attack. anyways i got a cool sandwich"AaronW: "of course we should have seatbelts" 😄MattCampbell: but then you've got the C die-hards who say that Rust itself is too complexAaronW: https://twitter.com/markrussinovich/status/1571995117233504257?s=46DanCrossNYC: People used to say the same thing about PL/I and recently the COBOL people have been saying the same thing. Nothing new under the sun.statuscalamitous: https://blog.yossarian.net/2023/02/11/The-unsafe-language-doom-principleDanCrossNYC: People who still want to treat C as a high-level assembler are saying the same stuff the PL/I people were saying when I was young.Eric Likness - carpetbomberz.com: In support of Yael, Ralph Nader wasn't/isn't an automotive engineer and he could still argue for lowering safety risks to car buyers. It's advocacy.cdaringe: As an ocaml user, i was hoping revery would take off https://github.com/revery-ui/reverystatuscalamitous: https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metricsSaethlin: Wake up babe, new 0xide reading assignment droppedAaronW: Labelled like a can of pringles -- "20% more malloc() free()!"Nahum: Relevant to rules based accounting: https://www.schneier.com/blog/archives/2023/02/hacking-the-tax-code.htmldrew: Rigorous definitions of “unsafe code” just wont cut it igig: 40% less direct pointer arithmetic than the leading brand of operating systemsa172: How does principle based accounting even work? Like, how do you define if something violates the principle or not, without just turning it back into rules based?Eden: Checkboxes are meaningful for operational checklists. Aviation and medicine use them pretty heavily. Not so meaningful for systemic work like developing a new aircraft or a new surgery.Eden: So I guess a rules-based approach works for lines of code, but breaks down for project-level decisions such as which language to use.Saethlin: The S in IoT is for securitybenstoltz: ifixit repairability score for HW should have an analog for SW/FW.DanCrossNYC: That's precisely what the pl/i folks acted like 25 years ago.sam801: c++ will live on thru carbon, cppfront, and val.DanCrossNYC: Prediction: carbon is doa.Saethlin: I'll believe it once anyone uses thoseig: I think the other part is there's some really important pieces of software that everyone uses daily which use memory unsafe languages. Our web browsers, and our operating systems.AaronW: I live in a condo and I still unplug expensive electronics during a thunderstorm. Maybe it's because I had many electronics fried when I was young, and my first language was C++.Eric Likness - carpetbomberz.com: Same with answering a landline during a thunderstorm.DanCrossNYC: Had to stop training during thunderstorms in the Marines.Eden: My day job is security. 😉 I rail against compliance checklists on a regular basis because a lot of auditors insist on the checkbox rather than proper security consideration. For example, PCI-DSS requires password rotation, which everyone has known for decades leads to users picking worse passwords.alilleybrinker: https://www.usenix.org/system/files/sec22summer_alexopoulos.pdfstatuscalamitous: https://security.googleblog.com/2022/12/memory-safe-languages-in-android-13.htmla172: Google and Mozilla are making pretty good strides in migrating their browser to Rust. Still a ton of work to go, but entire systems have been moved to Rust.JamesBrock: "Lindy" https://en.wikipedia.org/wiki/Lindy_effectstatuscalamitous: https://security.googleblog.com/2021/04/rust-in-android-platform.htmlDanCrossNYC: Another issue with C/C++ in particular is that UB causes latent bugs to surface years later.alilleybrinker: In the paper linked above, the average lifetime looks to have been about 3.5 years.Saethlin: I learned Rust faster than C++alilleybrinker: Related, you might be interested in EPSS: https://www.first.org/epss/DanCrossNYC: Rust requires a bit of humility. For veteran C programmers, that can be a gut punch

Members of the Oxide team join Bryan and Adam to talk about our journey through compliance (spoiler: we passed!). Oxide and Friends: February 6th, 2023 We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from February 6th, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers on February 6th included Arjen Roodselaar, Nathanael Huffman, Robert Keith, Eric Aasen, and Josh Clulow,Some of the topics we hit on, in the order that we hit them:If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends: January 23rd, 2023Revisiting UnikernelsWe've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from January 23rd, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers on January 23rd included Steve Klabnik, Dan Cross, and others.Some of the topics we hit on, in the order that we hit them:Bryan's 2016 blog post Unikernels are unfit for productionIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!Give feedback

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from January 16th, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Nathanael Huffman, Cliff Biffle, Rick Altherr, Matt Keeter, Eric Aasen, and Dan Cross.Check out the show notes on github to browse the images.(00:00) - Intro (11:42) - Gemini (18:33) - Root of Trust (RoT) carrier (20:53) - Power (23:41) - Trimmed Power (28:11) - SPI MUX (29:38) - SPI MUX rework (33:14) - Gimletlet (41:10) - Gimletlet NIC (46:28) - DIMMlet (56:39) - Gimletlet mk2 (58:27) - Adapters (59:54) - Adapters zoom (01:01:47) - Ignition (FPGA) (01:04:40) - Gimletlet peripherals (01:06:12) - Gimletlet with management switch (1/2) (01:07:22) - Gimletlet with management switch (2/2) (01:09:21) - Kludge.2 (K.2) (01:16:23) - Donglet (01:25:49) - RoT carrier carrier (01:26:17) - Tranceiver load tester (01:29:06) - Load slammer for Tofino 2 (01:31:30) - Power (improved) (01:32:08) - Part Toaster (01:33:28) - K.2r2 Images of each proto board:@11:42 Gemini @18:33 Root of Trust (RoT) carrier @20:53 Power @23:41 Trimmed Power @28:11 SPI MUX

See github for the list of predictions (and add your own!)

Break it down with Ian BrownWe've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from December 26th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Ian Brown.

A Debugging OdysseyWe've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from December 19th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Dave Pacheco.

Oxide and Friends: December 12th, 2022Podcasts for Podcast-LoversWe've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from December 12th, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on MM DD included XXX, and YY. (Did we miss your name and/or get it wrong? Drop a PR!)Podcasts mentioned on the showZiade&Ford AdvisorsBrady HeywoodEconTalkLamorna Ash on Dark, Salt, ClearThe Amp Hour Electronics PodcastTools & CraftResilient web designSoftware Defined TalkKubernetes: The Documentary part 1 and part 2devtools FMScience and Futurism with Isaac ArthurAs the Ice Cream ChurnsLost TerminalLet's Make A Sci-FiInvest Like the Best: Shane BattierNew York Times: The No-Stats All-StarMark and CarrieNerdOut@Spotify: Open Source Work Is WorkThe Flop HouseThe MothIt Could Happen HerePlaydate PodcastThe Chernobyl PodcastBehind the BastardsBehind the PoliceTerrorism BadGuys We F****dHow Did This Get Made: Holy MatrimonyStartup: How Not to Pitch a BillionaireBoom / Bust HQ TriviaAcquired: TSMCThe Pitch ShowBad BetsToolsEmbedded.fmHuff DufferOther links from the audienceBSD NowHardcore HistoryEMCrit Podcasthttps://podcasts.google.com/feed/aHR0cHM6Ly9lbWNyaXQub3JnL2ZlZWQvcG9kY2FzdC8?sa=X&ved=0CAMQ4aUDahcKEwiIuNvvuPX7AhUAAAAAHQAAAAAQAQ&hl=enhttps://podcasts.apple.com/us/podcast/if-books-could-kill/id1651876897The Weirdest Thing I Learned This Weekhttps://99percentinvisible.org/about/the-show/https://intel.com/aipodcasthttps://nextcloud.com/podcast/https://gzmshows.com/shows/listing/the-big-fib/http://wandb.com/podcasthttps://feeds.captivate.fm/gradient-dissent/https://blart.libsyn.com/https://darknetdiaries.com/https://theretrohour.com/https://www.youtube.com/channel/UCPxHg4192hLDpTI2w7F9rPghttps://reasonablysound.comhttps://www.youtube.com/@NoBoilerplatehttps://www.youtube.com/@Namtaohttps://signalsandthreads.com/https://www.bbc.co.uk/programmes/w13xtvg9/episodes/downloadshttps://darknetdiaries.com/https://blog.mainframe.dev/https://podcasts.apple.com/us/podcast/revolutions/id703889772https://gimletmedia.com/shows/mystery-showhttps://mast.hpc.social/@freemin7https://feeds.megaphone.fm/uncivilhttps://open.spotify.com/episode/4cJ7NqRen0OSJ2a4Wg4uaO?si=XYNwryI0Sc6MeYRboeUcgAhttps://mango.pdf.zone/finding-former-australian-prime-minister-tony-abbotts-passport-number-on-instagramhttps://open.spotify.com/episode/5EUBBoQVZtQMkhTjfSIvzu?si=j9wcHSKGSfCiP2khHsWBughttp://www.autonocast.comhttps://risky.bizhttps://share.transistor.fm/s/7809611e

Oxide and Friends: November 28th, 2022Leaving Twitter with Tim BrayWe've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from November 28th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Tim Bray. Other speakers on November 28th included Adam Jacob, Toasterson, and raggi. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:Bye, Twitter by Tim Brayjwz: PSA: Do Not Use Services That Hate The Internetjwz: Mastodon stampede "Federation" now apparently means "DDoS yourself."Tim Bray On AlgorithmsOn terrible Twitter ads: @intelnews: "Moore’s Law only stops when innovation stops.”PRs needed!If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 14th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Kris Nova. Other speakers on November 14th included XXX, and YY. (Did we miss your name and/or get it wrong? Drop a PR!)

Oxide and Friends Twitter Space: November 7th, 2022Tech LayoffsWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 7th, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on November 7th included XXX, and YY. (Did we miss your name and/or get it wrong? Drop a PR!)

Oxide and Friends Twitter Space: October 31st, 2022Open Source FirmwareWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 31st, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guests were Christian Walker and Philipp Deppenwiese.

Oxide and Friends Twitter Space: October 27th, 2022Let That Sink In! (Whither Twitter?)We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 27th, 2022.In this special, breaking news edition of Oxide and Friends, Bryan Cantrill was joined by Joshua M. Clulow, Charity Majors, mick, Rishi Desai, linear cannon, ignaloidas, Craig Traynor, Cargo Occultist, and Aaron David Goldman.

Oxide and Friends Twitter Space: October 24th, 2022Open Source Inside BaseballWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 24th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was dear friend-of-Oxide, Stephen O'Grady. Some of the topics we hit on, in the order that we hit them:For non-American and/or non-baseball fans "inside baseball" is an idiom meaning "an expert's take or opinion"Also, Stephen, Bryan, and Adam love actual baseball so there was quite a bit of that as well...For the baseball fans, the Bryce Harper at bat we were so excited aboutThe main event: Stephen's The Dead End

Oxide and Friends Twitter Space: October 10th, 2022Holistic BootWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 10th, 2022.

Inspired by the incentives at Google that apparently promote launching--but not sustaining--new products, Bryan, Adam and the Oxide Friends discuss the efficacy of various incentives... and the incentives that can lead to unintended and negative outcomes.

Bryan and Adam interview Sean Silcoff, co-author of "Losing the Signal: The Untold Story Behind the Extraordinary Rise and Spectacular Fall of BlackBerry"... Soon to be a major motion picture! Losing the Signal with Sean Silcoff (The Rise and Fall of BlackBerry)We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 26th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our esteemed guest was one of the authors of Losing the Signal, Sean Silcoff.Not many links, mostly anecdotes from Losing the Signal: The Untold Story Behind the Extraordinary Rise and Spectacular Fall of BlackBerry, the book Sean co-wrote with Jacquie McNishSome of the topics we hit on, in the order that we hit them:@05:01 The aha moment for BBM@09:42 Ruffled feathers from partner interactionsBell South thought they had an exlusive deal and they didn'tYears later, people didn't seem to hold it against them@12:40 Brian's anecdote about a meeting with Balsillie and Lazaridis@15:30 Lost opportunities to course-correctThe touch screen button@20:00 The Blackberry StormPotentially rushed to market when it was not up to standardsRIM's own testing lab found serious problems but shipped it anywayRIM was a great innovator and a terrible follower, some have said that of Apple, though@25:40 Lazaridis as the product guyThis I get (keypad), this I don't get (touchscreen)Story on the way up is as important as on the way down@30:20 Parallels with DECAmazing riseFailure to pass controlco-CEO model at RIM - worked really well until it didn't@34:19 NTP Patent caseCase briefJury selection was weighed incredibly far towards lay folks with very little technical understandingTechnical demonstration goes sideways@45:10 Trusting the other co-CEO and the option backdating scandalLazaridis didn't really understand the options stuff and felt Balsillie had put the company at riskKept their fights private, but people could tell "mom and dad weren't getting along"Is it right or wrong if everyone was doing it?They left an extensive digital paper trail making it easy to make a caseThanks to Tom for the clarification - options backdating was okay, failing to report it was not@52:50 Larry ConleeFire and brimstone, had a pocket veto, spoke with the voice of the CEOsCo COOs!Carriers were afraid of becoming dumb pipes, so were anti-app storeBlackberry didn't care about doing an app store, then AT&T bent to Apple and allowed them to have an app storeRIM did not believe that Silicon Valley would be let in the front door at the carriersRIM would talk about Apple as "the toy company" while being actively devouredIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

A problem has been eating at Adam: we use async/await in many languages and yet we're not so good at explaining the moving parts. Bryan and the Oxide Friends therapeutically explore the space.

Kate Hicks from Oxide operations joins to talk about the supply chain meltdown, war stories from the past, and the innovative ways she and her team have charted a steady course through these turbulent waters

The Oxide electrical engineers share their experience bringing up a 100Gb link--it's got everything from a purpose-built probing station to a 100Ω resistor that proved to be the difference between life and death (of the company)

Bryan, Adam, and Steve consider nuggets of conventional wisdom that turn out to be turds.

The Oxide Friends pour one out for Optane, Intel's great hope that never managed to find traction.

Seth Winterroth and Ian Rountree join Bryan, Adam, and the Oxide Friends to talk about investing in deep tech / hard tech.

Oxide and Friends Twitter Space: July 18th, 2022Across the Chasm with RustWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for July 18th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guests were Steve Klabnik and Luqman Aden. Other speakers included Dan Cross, Tim McNamara, and others. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:@0:27 let_chains are stable in Rust 1.64Adam's tweetThe stabilization PR, with the full saga leading up to stabilizationAs Steve mentions, the feature dates all the way back to 2017 and extends the Swift-inspired if let expressions Rust has had for a whileSome Rust features, like async functions in traits, are huge rabbit holesDiscussion about Rust's commitment to stability and how it's enforced with things like craterAs an example of the process leading to burnout in programming language communities: Guido stepping down as BDFL after PEP 572 (Assignment Expressions, "the walrus operator")Discussion about Ruby also taking stability seriously: flip-flops weren't removed in Ruby 2.0 in part because of this pretty incredible snippet from Yusuke EndohQuines and variations, Yusuke Endoh's Qlobe (reproduced here), their infamous quine-relay, and their other projectsThe G-Portugol programming languageThe unstable features mechanism in Rust ("first class support for experimental features") and how this allows for user experimentationExclusive range patterns in Rust and some of their perils, specifically in tockContrasting the Rust unstable feature mechanism with Haskell language pragmas: the former requires a nightly compiler to use, the latter does not@18:20 Discussion about the Rust process; going from RFC to stable RustThe Rust inline assembly feature (tracking issue)The Rust RFC repoThe Generic Associated Types (GATs) Rust RFChubris is on nightly Rust but with an allow list of featuresNaked functions in Rust (tracking issue), Destructuring assignments, #[cmse_nonsecure_entry]Talking about LWN-style reports and curation as a way to lessen the pain of using Zulip style chat platforms for discussionLWN is hiring, looking for someone to keep up with Rust development, among other things[[partial notes]]

Oxide and Friends Twitter Space: July 11th, 2022Integrating Hardware and Software TeamsWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for July 11th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our esteemed guest was Jon Masters. Other speakers included Nathaneal Huffman, Tom Lyon, Dan Cross, Rick Altherr, Matt Keeter, Peter Corless, Timon, Siddharth Joshi, Bob Mader, Aaron David Goldman, Simeon Miteff, Remy Goldschmidt, and MattSci. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:@4:32 Fostering of mutual hatred between hardware and software peopleHuge difference in cost of errors in both time and money@9:38 Dealing with perishable pre-preg material Tachyon 100GTachyon 100G@15:06 The black magic that is DDRDIMM training demo@21:58 Open source tooling for EEsOpen FPGA toolingOpen RISCRISC VZero to ASIC courseLinux from scratchBen Eater's 8bit computerPhil's lab, KiCad 6 PCB design walkthoughPhil's lab, Altium Designer PCB design walkthough@33:18 Matt Keeter's take on ECAD toolsEagle CADSmaller breakout boards made with KiCad for unit testing@36:55 Timon's take on EE curriculumMath-heavy electrical engineering curriculumArts of ElectronicsKnowing at least basics of adjacent disciplines goes a long way@49:03 Software shouldn't pierce abstractions in order to work reliably, but people should to deepen their knowledge@1:04:54 Making microchips at homeSam Zeloof, maskless-photolithographyJeri Elseworth, making microchips at home@1:06:05 Oxide gets a Pick'n'Place machine?Open Hardware Pick'n'Place machine@1:09:40 Bob's take on silosSMM, System Management Mode@1:22:15 Vintage gaming as an intro into embedded softwareWiFi Game Boy Cartridge@1:26:14 Fabs at UNI@1:29:40 Intel Tofino (TM) Series Programmable Ethernet Switch ASICIntel Tofino@1:31:13 Google's open source high level synth. (HLS) tool XLSXLSBluespecChiselIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for June 27th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Scott Johnson. Other speakers included XX and YY. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:@2:00 Beautiful C++@13:45 DEC is Dead, Long Live DEC@15:00 Almost Perfect@17:20 The Friendly Orange Glow@18:10 Steve Jobs & the NeXT Big Thing@18:50 The Soul of a New Machine@36:08 Reinventing the Wheel@38:13 Creative Capital

pending

Oxide and Friends Twitter Space: June 13th, 2022The Rise and Fall of DECWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for June 13th, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on June 13th included Tom Lyon, Dan Cross, Tim Bray, Ian Grunert, and XXX. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:Pronunciation and mispronunciationBryan's DEC reading list:The Ultimate Entrepreneur by Glenn Rifkin, George HarrarLearn, Earn & Return - My Life as a Computer Pioneer by Harlan AndersonHigh-tech Ventures: The Guide For Entrepreneurial Success by C. Gordon Bell, John McNamaraComputer engineering: A DEC view of hardware systems design by C. Gordon Bell, J. Craig Mudge, John McNamaraCreative Capital: Georges Doriot and the Birth of Venture Capital by Spencer E. AnteDEC Is Dead, Long Live DEC: The Lasting Legacy of Digital Equipment Corporation by Edgar H. Schein, Paul J. Kampas, Michael M. Sonduck, Peter S. Delisi@1:29:05 Ian mentions Computer History Museum's oral history program prompting strong recommendations:Ian: Bernie LacrouteAdam: Pierre LamondBryan: Dave CutlerIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: May 23rd, 2022Surviving the Dot-Com BustWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for May 23rd, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on May 23rd included Dan McDonald, Dan Cross, Joshua Clulow, Steve Tuck, Matt Campbell and Theo Schlossnagle. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:@1:37 Pick and shovels story circulating at SunOakes Ames "King of Spades", pbs article, wikiboo.com wikipets.com wiki@11:00 IPOs and public exposuretheGlobe.com wiki@18:20 "The Correction"Feasting like 19th century robber baronsNov 2000, free fallTrilogy, Inc wiki@28:49 Students looking for placementClarity of the bust@36:35 Billboards on the 101garden.com, cnn blurb@39:13 Theo's story, roulette at TrilogyExpansion and contraction of CS student enrollment@46:20 Matt's memoriesAllAdvantage wikiExcite@Home wikiJohn Talbott "The Fall of Silicon Graphics" articleFucked Company wikiCamaraderie over watching your companies imploding@53:39 Looking towards the looming housing bubbleDuring Oxide raise, race against time for VC fundingPandemicHot venture environment, over-valued companiesStimulus, spending on non-essentials, exacerbating income inequality@58:56 Differences from the dot-com era, more defined revenue models?Food delivery services, harbingers of bust?Steve anecdote: Dellionaires, margin call day, layoffs@1:08:12 Dan's second startup experienceWindows on the World wiki@1:10:15 Matt's question: can Oxide weather a tech bust?Adam: downturn can motivate seeking value, looking away from (stable, pricey) incumbents to (riskier, cheaper) new offeringsBryan: dot com bust pushed us toward open-source, for economic reasons.@1:15:51 Are we headed for a bust? How deep?How does a company survive the lean times?Negative human consequencesmammon (money, material wealth) wikiAdvice for practitioners?Dan McDonald: controlling inflation, starting companiesTheo: downturn will hit industries differently, concerns over global supply chainAdam: don't forget about helping others, looking out for other people, for the future of our world.Dan Cross: if it looks to good to be true, it probably is.If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: May 16th, 2022Debugging MethodologiesWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for May 16th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guests on May 16th were Jordan Hendricks and Luqman Aden. Other speakers included jasonbking, Rick Altherr and Ben Kimock. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:Green Room wikiNVMe wiki (Non-Volatile Memory. PCI Express)@3:38 Jordan's storyJordan's thorough bug write-up, (reported by Josh Clulow as "nvme_quiesce() can hang preventing reboot")Non-maskable interrupt wiki@8:04 Adam interrupts a box with a kitchen knifekmdb man page and page in the mdb book@14:11 Josh recites a poem about timeoutsAvoiding getting stuck, experimenting@20:10 A previous encounter with NVMe/PCIe issues (see also: Jordan's NVMe Hotplug discussion video ~26mins)mdb format character "j" (for Jordan!) (and jazzed-up) feature@26:50 Normal and abrupt shutdown notification, breakthrough, writing up a narrative@32:27 Luqman's storyThe blog post "Achievement Unlocked: rustc segfault"dtrace usdtcscope, rust analyzer@43:50 Inspecting LLVM IR, RustC MIRasync blocks, inline assemblyboiling down reproducible casesmaking quality write-ups, telling a story, teaching debuggingpopular on Hacker Newsdead reproducible?@1:03:02 Bugs: psychotic, non reproducibleDebugging mindsetDifferent tools and methodologies for different problemsanonymous tracing book page, speculative tracing page@1:10:03 Jason: number literal formats with underscores, now in mdb@1:12:35 Ben prompts a debugging story, checking conditions in debug, program abort on errorud2 instructionRick describes the Oxide boot loaderXMODEM wikiTriple fault wikiRust "heapless" crateIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: April 25th, 2022Fail WhalingWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for April 25th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Jason Hoffman. Other speakers included Joshua Clulow, Matt Campbell and Ian. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:Debugging RailsJason walks chain of events leading to "twttr"@10:46 The first mock up, SMS@13:42 Twitter goes live, early daysTcl, Mongrel, LiteSpeed@19:30 First problemsBryan's debugging story, exceptions and backtraces, index out of boundsDiscovery of the problem was not met with gratitude@29:53 Jason tells another problem story, production directories full of junk test files@38:30 Story of the first Hadoop cluster@42:22 Matt's comment on directory limits@46:35 Companies growing up, on-prem and cloud infrastructure@49:26 The Fail WhaleRuby runtime, Ryan DahlMoved to Java, Scala eventuallyDTrace and dynamic languagesRaku, Parrot VM, MoarVM@59:53 Changing language and hardware landscapes, video presentation sharing, short social media handles, ahl, getting into hockey@1:12:30 Billionaire's playground?Quick diversion, history trivia bet@1:18:43 ModerationMicrosoft Tay bot (shutdown 16 hours after launch)Can anything kill Twitter?@1;29:26 Matt: what replaces Spaces?How could an alternative be built? What would it look like?Bryan predicts: change of headquarters, "burning the flag"Adam predicts: resale or IPO within 3 yearsSee also: Jason Hoffman and Bryan Cantrill CTO vs VP Engineering video ~45mins (audio is rough, content is good)If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: April 18th, 2022More Tales from the Bringup LabWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for April 18th, 2022.In addition to Bryan Cantrill and Adam Leventhal, we were joined by members of the Oxide team: Arjen Roodselaar, Nathaneal Huffman, Robert Mustacchi, Aaron Hartwig, Steve Tuck, Matt Keeter, Eric Aasen, Rick Altherr, and RFK.Some of the topics we hit on, in the order that we hit them:@2:25 Overview of upcoming themes related to the bringup lab@4:28 Defining the different terms and code-names of the hardware in development at oxide@4:40 Gimlet, the compute node@5:10 Sidecar, a board based on a switching ASIC from Intel@7:24 Arjen's twitter thread with details related to the bringup and Eric's description of the challenges in designing the PDN (Power Delivery Network) ATT@15:34 The load-slammer, an electronic load to simulate the power draw of an ASIC / BGA-part LS@19:06 Bouncing supply cables on load steps@22:27 FPGA that controls everything on the Sidecar board@24:05 TOML's unstable table order made the team pop a couple ICs off the board searching for bugs@31:41 Brown-out in the hotel during first bringup session from a blown bus duct@33:45 Debugging ground bounce issues while testing the PDN with the load-slammer (phantom over/undershoot)@40:15 Hardware team pranks the management during a meeting with a potential investor@43:20 Chonky heat sink that weighs 8 pounds / moment arm crisis@48:19 First time powering up, checking temperature with thermal camera, learning about "puppy dog warm"@52:12 Matt talks about the second, "lesser" network switch on the Sidecar board@57:28 Secret 8051 cores, slew-rate woes: impedance missmatch on SPI traces that manifested in unreliable communication in full bandwidth mode of the SPI/GPIO driver@1:03:19 PLL config issues and Matt's verbose config tool to fix them@1:04:26 Load-bearing dongles@1:06:37 Debugging PCIe link, Arjen's Frankenstein PCIe analyzer/exerciser@1:22:36 Gimlet, stumbling blocks found in January@1:30:08 Arjen's big breakthrough on the Sidecar, shouting at the T6@1:32:08 Cursed pull-downs, Rick's remote hardware debugging support by incrementally breaking his T6 boards to find issue with the DUT@1:36:24 T6 finally comes out of reset, "we're gonna live! we're gonna live!"@1:41:06 Rick reworks gnarly footprint error, on multiple ICs, to verify design for Rev. B - dead bug style.@1:53:12 Sidecar progress continuation, cable oupsi, off-by-one error@1:59:42 Dedicated support by IC vendor with very understanding wives@2:01:20 Summary and parting thoughtsIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: April 4th, 2022Another LPC55 ROM VulnerabilityWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for April 4th, 2022.In addition to Bryan Cantrill and Adam Leventhal, our special guest was Laura Abbott.Other speakers on April 4th included Ian, jasonbking, Todd Gamblin?, Ben ?, MattSci, jasonbking and Evan?. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:Jonathan Goldstein's Heavyweight podcastOxide and Friends podcasttransistor.fm launch point, has links to Spotify, Google, Amazon etc playersLaura did talk about the first LPC55 vulnerability in the May 3, 2021 space, but the recording for that day missed it.Laura Abbott (30 April, 2021) Exploiting Undocumented Hardware Blocks in the LPC55S69 write-upAnd DEF CON talk with Rick Altherr@4:01 Today's topic: Laura Abbott (23 March 2022) Another vulnerability in the LPC55S69 ROM write upHow do you brick a chip?@7:20 The spreadsheet, ROM patch after bootCompany dismisses or downplays vulnerabilitiesSees CVEs as optional??@15:19 CVEs as more software focused. What does a CVE for hardware even mean?NXP doesn't want to open their software"Even though we are not believers in security by obscurity, the product specific ROM code is not open to external parties except for approved test labs for vulnerability reviews"@19:43 The story of the current vulnerabilityGhidra@27:26 Picking apart the codeBounds checks, writing outside the bounds of the bufferDICE by Trusted Computing GroupRequest for DiscussionEvaluating potential chips when building a product@41:09 Secure hardware, work around potential pitfallsOpen source would help@45:37 Disclosed to NXP, more receptive this timeDiscussion on HN@54:21 Security review industry@57:11 Ian: building up your own (open) documentation on LPC55?@1:01:31 Jason: questionable definitions of "open" sourceAccess to source as building confidence in the product@1:05:20 Todd: securing supply chain for code in large scale projects with lots of contributorsVulnerabilities can occur so easily@1:08:54 Ben: custom setups abound. Hard to trust a whole stack of assembled pieces@1:12:16 Matt: what is the ROM doing? Assembly or C? Could the provider's hands be tied as far as releasing proprietary code?@1:17:19 Jason: X.509 parsing as a good place to look for vulnerabilities?@1:18:25 Evan: encouragement around fuzzing X.509Next time: more tales from the bringup lab!If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: March 28th, 2022Time, Timezones, Metric Time, Losing and SavingWe’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 28th, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on March 28th included Tom Lyon, jasonbking, Matt Campbell, Akshay Kumar, Aaron Goldman and Simeon Miteff. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:[@8:07](https://youtu.be/BHtfqleSHAs?t=487) Y2K, leap years The Staff of Ra“at” command[@15:28](https://youtu.be/BHtfqleSHAs?t=928) Matt’s stories elm email[@23:29](https://youtu.be/BHtfqleSHAs?t=1409) Jason: daylight saving time in Indiana “Time in Indiana” wiki[@26:31](https://youtu.be/BHtfqleSHAs?t=1591) Time zone database John Bemelmans Marciano (2014) Whatever Happened to the Metric System? How America Kept Its Feet bookGeopolitical aspects of timeEastman plan calendar[@32:23](https://youtu.be/BHtfqleSHAs?t=1943) Aaron’s stories, setting clocks back, Leap Day[@35:54](https://youtu.be/BHtfqleSHAs?t=2154) Akshay: Ken Thompson’s six day work week?Leap seconds Time of day hardware bug[@48:54](https://youtu.be/BHtfqleSHAs?t=2934) 2038 - the end of time Y2K problemsGPS week number rollover wiki[@57:58](https://youtu.be/BHtfqleSHAs?t=3478) Matt: Cory Doctorow’s “Epoch” short story podcast commissioned by Mark Shuttleworth[@1:00:28](https://youtu.be/BHtfqleSHAs?t=3628) Ultimate, penultimate, antepenultimateOxide and Friends podcast!! transistor.fm launch point, has links to Spotify, Google, Amazon etc playersLaura Abbott (23 March 2022) Another vulnerability in the LPC55S69 ROM write upIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: March 21st, 2022Trolltron, Assemble!We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 21st, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on March 21st included Antranig Vartanian, Dan Cross, Ian, jasonbking, Jason Ozolins, Ken and Drew Vogel. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:[@9:23](https://youtu.be/WrEef_bsWas?t=563) I was learning from people who were further down the track than I was Startups can have problems when founders fail to learn from the experiences of others[@12:43](https://youtu.be/WrEef_bsWas?t=763) Dan: hubris of youth is an age old problem, see middle ages nobilityFor some “child wonders”, their childhood is effectively sacrificed because their adulthood arrives too early[@16:22](https://youtu.be/WrEef_bsWas?t=982) When I went to school, there was a math prodigy.. Challenging operating system course[@25:44](https://youtu.be/WrEef_bsWas?t=1544) Ian: for early accelerated learners, the work is easy until it isn’t. They didn’t need to spend long hours studying, so they didn’t practice it. > You have to take that youthful ego, and gently massacre it. Then build them upThe Dropout series, premiered March 2022[@31:26](https://youtu.be/WrEef_bsWas?t=1886) Jason O: praising ability vs effort, negative effects[@34:55](https://youtu.be/WrEef_bsWas?t=2095) 30 under 30, and such thingsEmpathy, learning to compromise, learning from being a parent[@41:04](https://youtu.be/WrEef_bsWas?t=2464) How venture views human capital Student loans, (some predatory lenders)Does making a young person comfy lead to their best work?Taking a share of future earnings, kinda demotivating. Misaligned incentivesLambda school, coding bootcamps. Fixed costs and incoming sharing[@50:17](https://youtu.be/WrEef_bsWas?t=3017) Sourcing these kids? “You’re a baseball card for someone” story, why are these kids at this party??[@57:40](https://youtu.be/WrEef_bsWas?t=3460) Sometimes kids who are extremely comfortable aren’t terribly motivated to put in the hours[@59:25](https://youtu.be/WrEef_bsWas?t=3565) Drew: Income sharing and other schemes to pay for education Ken: doesn’t feel like kids would be set up for success[@1:04:07](https://youtu.be/WrEef_bsWas?t=3847) Background beef leading to this hairy scheme Some entrepreneurs have trouble seeing the role of luck in their successThiel Fellowship wikiIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: March 14th, 2022UkraineWe’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 14th, 2022.In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Andrey Akselrod.Some of the topics we hit on, in the order that we hit them:If you’re interested in donating to support Ukrainians, Andrey recommends Nova Ukraine[@1:52](https://youtu.be/EdJU8mSWzQk?t=112) Andrey introduces himself, background in computing[@11:20](https://youtu.be/EdJU8mSWzQk?t=68) Andrey talks about where he lived in Ukraine, DniproConfluence of culturesMoves to New York[@22:53](https://youtu.be/EdJU8mSWzQk?t=1373) Events of 2014, family and coworkers in Ukraine Crimea[@29:12](https://youtu.be/EdJU8mSWzQk?t=1752) Earlier disputed regions (Crimea, Donbas) and relations to current events Ukrainian national identity[@38:21](https://youtu.be/EdJU8mSWzQk?t=2301) Armed forces, self governance Business as usual, life goes on[@44:45](https://youtu.be/EdJU8mSWzQk?t=2685) Characterizing Ukraine as European democracy, and economic functions/trade Nuclear reactors[@49:12](https://youtu.be/EdJU8mSWzQk?t=2952) Invasion Leadership disconnect with reality[@1:02:28](https://youtu.be/EdJU8mSWzQk?t=3748) Family still in Dnipro Electronic communicationsKids understanding of what’s happening[@1:07:59](https://youtu.be/EdJU8mSWzQk?t=4079) How to help?[@1:16:50](https://youtu.be/EdJU8mSWzQk?t=4610) Andrey’s coworkers and team members remaining in Ukraine > Yes it’s war, but, the economy needs to continue to be healthy.[@1:21:24](https://youtu.be/EdJU8mSWzQk?t=4884) Where is this going?If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: March 7th, 2022The Future Of WorkWe’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 7th, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on March 7th included Lucas Ives, Dan McDonald, Steve Tuck, Ian, Matt Campbell, MattSci, Jim Rybarski, Austin, Aaron Goldman, Jake Demarest-Mays, Jason Ozolins, Tom Lyon, Timon, Matthew Amdur, jasonbking, and Horace. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:[@8:15](https://youtu.be/GTluipbKeII?t=495) Lucas’ storyRemote before pandemic, comparisons[@16:29](https://youtu.be/GTluipbKeII?t=989) Sidebar chat, backchannel[@22:49](https://youtu.be/GTluipbKeII?t=1369) Pre-recorded talks, speaker commenting in chat engaging with questions Multitasking during meetings, different from in-person single-threaded meetingsRecording meetings for later reviewHolding onto a thought may detract from fully listening to another’s point[@34:40](https://youtu.be/GTluipbKeII?t=2080) Oxide’s full team meetup, what did they focus on?[@38:01](https://youtu.be/GTluipbKeII?t=2281) Austin’s remote experience[@44:30](https://youtu.be/GTluipbKeII?t=2670) Dan’s question: remote employees “pilgrimage” back to home often, how often?[@50:23](https://youtu.be/GTluipbKeII?t=3023) Disadvantages to full remote?[@56:15](https://youtu.be/GTluipbKeII?t=3375) Jake’s experience, asynchronous work style Meetings as unprepared group think sessions, not valuable as decision makingRequests for discussion, as decision making tools[@1:02:29](https://youtu.be/GTluipbKeII?t=3749) Jason: service delivery vs product delivery Class devision between “the desked” and “the un-desked”[@1:07:17](https://youtu.be/GTluipbKeII?t=4037) Is “back to office” about command and control? Other factors: big tech companies receive substantial local subsidies[@1:14:00](https://youtu.be/GTluipbKeII?t=4440) Timon on working in different timezones Recorded meetings/discussions as valuable contentPandemic boosted remote work tool quality[@1:23:32](https://youtu.be/GTluipbKeII?t=5012) Difficulties with remote? Building rapport, judging emotions and nuanced communicationOrganic, unplanned communications with in-person office spaces (watercooler)[@1:33:24](https://youtu.be/GTluipbKeII?t=5604) Matt: remote work as cost savings?Value of “down time” communication, unstructured[@1:43:50](https://youtu.be/GTluipbKeII?t=6230) Starting career, making connections, in all-remote world[@1:47:58](https://youtu.be/GTluipbKeII?t=6478) Future of remote work since pandemic[@1:51:30](https://youtu.be/GTluipbKeII?t=6690) Horace’s experience with remote workIf we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Oxide and Friends Twitter Space: February 21st, 2022Engineering CultureWe’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for February 21st, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on February 21st included Tom Lyon, Tom Killalea, Ian, Antranig Vartanian, Matt Campbell, Simeon Miteff, Matt Ranney and Aaron Hartwig. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them:Alex Heath’s tweet on FB meeting about updated values: “meta, metamates, me”[@4:44](https://youtu.be/w9MQJbC26h4?t=284) Can an established company “change its values” in any sense?[@8:43](https://youtu.be/w9MQJbC26h4?t=523) Draw the owl > Twilio CEO: Yes, it was a meme, but it’s a great representation of our job. > There is no instruction book and no one is going to tell us how to do our work. > It’s now woven into our culture and used as a cheeky, but encouraging reply to > those who email colleagues at Twilio asking how to do something.[@12:42](https://youtu.be/w9MQJbC26h4?t=762) How do you establish engineering culture? Copy-paste values?[@20:44](https://youtu.be/w9MQJbC26h4?t=1244) When are values set down in a company’s history? Amazon’s brand image, expanding beyond booksAssessing values when hiring[@27:51](https://youtu.be/w9MQJbC26h4?t=1671) Principles vs values Principles are absolutes, cannot be taken too farValues are about relative importance, in balance with other valuesACM Code of EthicsRelative importance of values. Can some values be learned, while others cannot?[@45:11](https://youtu.be/w9MQJbC26h4?t=2711) “Turn-around CEOs”, trying to change an established company culture[@47:39](https://youtu.be/w9MQJbC26h4?t=2859) Sun culture, early days[@54:32](https://youtu.be/w9MQJbC26h4?t=3272) Connection between values and business model Urgency in context, requires nuance[@1:03:37](https://youtu.be/w9MQJbC26h4?t=3817) Values on the wall. When are values simply ignored? Jack Handey wiki, Deep Thoughts recurring SNL short sketches, eg Thanksgiving ~30secs“Sharpen fast”[@1:13:49](https://youtu.be/w9MQJbC26h4?t=4429) What are the important things to get set early? Bryan and Adam on Joyent and Delphix[@1:22:05](https://youtu.be/w9MQJbC26h4?t=4925) Matt Ranney on his time at Uber Trying to shape an established cultureLeadership’s values vs engineersBusiness ethics[@1:35:47](https://youtu.be/w9MQJbC26h4?t=5747) GEThomas Gryta and Ted Mann (2020) Lights Out: Pride, Delusion, and the Fall of General Electric book[@1:37:03](https://youtu.be/w9MQJbC26h4?t=5823) Conclusions Adam: Get it right first, but it’s not a lost cause if you don’t.Bryan: Look for value alignment in organizations you might want to join, it’s tough to change course after the fact.Matt: generous compensation has an effect on how closely one cares to scrutinize their organization’s values ¯_(ツ)_/¯If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!