The Lindahl Letter

Perhaps you were looking for a bit more of a deep dive about deep generative models than will be contained in this relatively short missive. You could go check out Stanford University’s CS236 from Fall 2021 course [1]. That would help you begin to figure out just how unsupervised learning could be used to figure out the data distribution well enough to generate predicted other data elements. The content is broken up into 5 sections and you could contribute to the class GitHub if you wanted to provide feedback or improvement suggestions. Alternatively, you could learn more about this one from Prakash Pandey over at Towards Data Science from back in 2018 [2]. It's a faster read and pretty easy to digest compared to taking on a college level course. You could go the academic paper route for an introduction and dig into a work from Ruthotto & Haber from 2021:Ruthotto, L., & Haber, E. (2021). An introduction to deep generative modeling. GAMM‐Mitteilungen, 44(2), e202100008. https://arxiv.org/pdf/2103.05180.pdf My initial run at digging into deep generative models opened the door to a bunch of different topics within the space. Right at the start I ran into semi-supervised learning, graphs, urban mobility, and molecular science. You can imagine the urban mobility one made me a little curious what people were doing to model that with deep generation. Apparently, Google Scholar had enough data to offer up three paths including migration, mobility, and morphology. To get started I dug in with a quick search on "Deep generative models" with urban mobility [3]. None of the articles within this search space have a lot of references. It might be a relatively small area of academic inquiry at the moment. You could read most of the relevant academic content related to deep generative models in an afternoon. Trying to dig into using them for some type of use case will take a bit more effort in terms of setup, technology, and selection of that use case. Here are 3 papers that showed up with the urban mobility search:Eigenschink, P., Vamosi, S., Vamosi, R., Sun, C., Reutterer, T., & Kalcher, K. (2021). Deep Generative Models for Synthetic Data. ACM Computing Surveys. https://epub.wu.ac.at/8394/1/Deep_Generative_Models_for_Sequential_Data__WU_ePub_.pdfAnda, C., & Ordonez Medina, S. A. (2019). Privacy-by-design generative models of urban mobility. Arbeitsberichte Verkehrs-und Raumplanung, 1454. https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/357034/3/ab1454.pdfJohnsen, M., Brandt, O., Garrido, S., & Pereira, F. (2022). Population synthesis for urban resident modeling using deep generative models. Neural Computing and Applications, 34(6), 4677-4692. https://arxiv.org/ftp/arxiv/papers/2011/2011.06851.pdf 3 decently cited academic papers:Salakhutdinov, R. (2015). Learning deep generative models. Annual Review of Statistics and Its Application, 2, 361-385. https://www.utstat.toronto.edu/~rsalakhu/papers/Russ_thesis.pdfKingma, D. P., Mohamed, S., Jimenez Rezende, D., & Welling, M. (2014). Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27. https://proceedings.neurips.cc/paper/2014/file/d523773c6b194f37b938d340d5d02232-Paper.pdfMaaløe, L., Sønderby, C. K., Sønderby, S. K., & Winther, O. (2016, June). Auxiliary deep generative models. In International conference on machine learning (pp. 1445-1453). PMLR. http://proceedings.mlr.press/v48/maaloe16.pdf Links and thoughts:“Teaching MLOps at scale with GitHub - Universe 2022”“#83 Dr. ANDREW LAMPINEN (Deepmind) - Natural Language, Symbols and Grounding [NEURIPS2022 UNPLUGGED]”Top 5 Tweets of the week:Footnotes:[1] https://deepgenerativemodels.github.io/ [2] https://towardsdatascience.com/deep-generative-models-25ab2821afd3 [3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22Deep+generative+models%22+urban+mobility&btnG= What’s next for The Lindahl Letter?* Week 100: Overcrowding and ML* Week 101: Back to the ROI for ML * Week 102: ML pracademics* Week 103: Rethinking the future of ML* Week 104: That 2nd year of posting recapI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are going to break out of the planned programming and go where everybody else involved in the machine learning space is going to go this week. Something new arrived this week and has captured the attention of the public mind. Seriously, I thought stable diffusion would be the big thing for 2022, but something else arrived in the intellectual space that might be more influential in the long run. The contents of this post were written right after the release, and I have been tinkering with that content throughout the week. Staying current and staying well-grounded in the field of machine learning has been increasingly difficult. Those two things are ultimately very hard to achieve at the same time. This is an area with a great deal of breath and depth. I say that after writing 98 consecutively published weekly installments of a Substack machine learning newsletter. The main example this week would be of the new interactive session based chat framework (bot) that OpenAI released this week. Like most of the people actively curious about how bot’s have improved these days, I went out to https://chat.openai.com/chat which the team over at OpenAI blogged about here https://openai.com/blog/chatgpt/. All you need to do is create an account and you can sign in and chat with the bot. Each new session is tabula rasa as a reset to the model without the additional layer of your previous interactions. This is the interesting part of the equation as building a model and then having the model keep context within a conversation or a series of conversations. That ability to keep conversational context across multiple conversations is not a part of the current deployment. The research preview they are sharing right now does not have access to the internet. It was also trained on data from about a year ago. Given the size of the language model they are invoking I would think it has a very large knowledge graph included , but that does not really appear to be the case. I gave it the following series of prompts to see what would happen in terms of how it generates content. The answers are in screenshot to help identify that the content was not created by me as a part of my normal writing output.My prompt: “write 10 words about machine learning”My prompt: “write 25 words about machine learning”My prompt: “write 50 words about machine learning”My prompt: “write 100 words about machine learning”The last prompt I used was to ask it to “write about paper about machine learning with citations” which caused it to spit out 5 paragraphs that were pretty good.This topic is way too early for academic articles [1]. A lot of news articles have been written about OpenAI’s chatbot they recently shared called ChatGPT. Here are 4 of them that came out this week:“OpenAI’s new chatbot can explain code and write sitcom scripts but is still easily tricked”https://www.theverge.com/23488017/openai-chatbot-chatgpt-ai-examples-web-demo“OpenAI’s new ChatGPT is scary-good, crazy-fun, and—so far—not particularly evil.” https://slate.com/technology/2022/12/chatgpt-openai-artificial-intelligence-chatbot-whoa.html“OpenAI invites everyone to test new AI-powered chatbot—with amusing results”https://arstechnica.com/information-technology/2022/12/openai-invites-everyone-to-test-new-ai-powered-chatbot-with-amusing-results/“OpenAI’s ChatGPT shows why implementation is key with generative AI”https://techcrunch.com/2022/12/02/openais-chatgpt-shows-why-implementation-is-key-with-generative-ai/Links and thoughts:“E106: SBF's media strategy, FTX culpability, ChatGPT, SaaS slowdown & more”“Why Do I Keep Getting Called Out - WAN Show December 2, 2022”Top 6 “ChatGPT” Tweets of the week:Footnotes:[1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=chatgpt&btnG= What’s next for The Lindahl Letter?* Week 99: Deep generative models* Week 100: Overcrowding and ML* Week 101: Back to the ROI for ML* Week 102: ML pracademics* Week 103: Rethinking the future of ML* Week 104: That 2nd year of posting recapI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Recently, I started spending a bit more time writing about quantum machine learning and quantum computing in general. One of the things I became curious about was related to a thread of thoughts about how they code something for a quantum computer. One of the first things that I came across while trying to learn more about how people were coding with Twist was an article in IEEE Spectrum called, “Meet Twist: MIT’s Quantum Programming Language: Keeping tabs on data entanglement keeps reins on buggy quantum code” [1]. This article referenced out to CSAIL or the Computer Science and Artificial Intelligence Laboratory which happens to be located at the Massachusetts Institute for Technology. You can find their delightful website right here: https://www.csail.mit.edu/ and it does pretty easily direct you toward Twist with a few searches [2].Within that website you will find that the researchers Charles Yuan, Christopher McNally, and Michael Carbin shared a paper at POPL 2022 [3].Proceedings of the ACM on Programming LanguagesVolume 6Issue POPL January 2022 Article No.: 30pp 1–32 https://doi.org/10.1145/3498691Or you could go out to the paper on arXiv and download the PDF.Yuan, C., McNally, C., & Carbin, M. (2022). Twist: sound reasoning for purity and entanglement in Quantum programs. Proceedings of the ACM on Programming Languages, 6(POPL), 1-32. https://arxiv.org/abs/2205.02287You might be curious how many papers cite that paper and the answer at this very moment from Google Scholar happens to be 6 [4]. None of this content seems to be highly citated at this point in terms of a network of academic coverage.However, if you were wondering about an MIT course you could take related to this, then you are in luck. For only $2,249.00 you could take the 4-week course that starts on January 23, 2023: https://learn-xpro.mit.edu/quantum-computing.At this point in the process, I started looking around for coding examples or notebooks with something to try to absorb. You are going to end up with the paper reading pages 46-53 of that paper linked above from arXiv.I took a look at this artifact up on GitHub here: https://github.com/psg-mit/twist-popl22Over the next year I’ll be looking for more practical coding examples or maybe a tutorial that really explains how to use the Twist quantum programing language. I could not find an emulator or a code dojo to test out things either. That is problematic given that I’m probably not going to pay for time on a quantum computer to learn how to write Twist code or more to the point you are paying to engage in the activity of coding.Links and thoughts:“A Hard Fork in the Road: FTX’s Unraveling and Elon’s Loyalty Oath”“#81 JULIAN TOGELIUS, Prof. KEN STANLEY - AGI, Games, Diversity & Creativity [UNPLUGGED]”“Galactica: A Large Language Model for Science (Drama & Paper Review)”“A Sports Card Documentary IN THEATERS?! 👀 (Behind The Card)”Top 5 Tweets of the week:Footnotes:[1] https://spectrum.ieee.org/quantum-programming-language-twist[2] https://www.csail.mit.edu/news/language-quantum-computing[3] https://popl22.sigplan.org/details/POPL-2022-popl-research-papers/30/Twist-Sound-Reasoning-for-Purity-and-Entanglement-in-Quantum-Programs[4] https://scholar.google.com/scholar?cites=6310617750987639534&as_sdt=4005&sciodt=0,6&hl=enWhat’s next for The Lindahl Letter?* Week 98: Deep generative models* Week 99: Overcrowding and ML* Week 100: Back to the ROI for ML* Week 101: Revisiting my MLOps paper* Week 102: ML pracademics* Week 103: Rethinking the future of ML* Week 104: That 2nd year of posting recapI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Within the broader generative AI space, the part I tend to focus on is related to the written word. Right now, all of the visual generation parts of generative AI in terms of images and videos are wholesale living in the public mind [1]. Creative people are generating thumbnails and playing with all sorts of plausible image generation technology. A few teams are rapidly working on how to make video from that same type of generative AI and that is going to be interesting. We are probably going to see generative AI shows that people create very soon. I have previously written (Week 78) that I think all public trust in imagines is going to erode and that we are going to hit a zero-trust wall when it comes to being able to believe what we see [2]. This missive will be about the future of where large language models are going and a bit of a reflection on what has happened in the last couple of years.Back on October 26, 2021, the folks over on Hugging Face shared out a post called, “Large Language Models: A New Moore's Law? [3]. This post starts out with a very familiar graphic of the models in terms of billions of parameters over time. This is a relatively recent phenomenon with a start during 2018 and massive acceleration after 2020. You may well have heard about the Megatron-Turing natural language generation model (MT-NLG) [4][5][6]. You can imagine that people were thinking they should make larger models. After all what is better than a billion-parameter model? It obviously has to be a trillion-parameter model. I would argue that the reality of having unique parameters within that large of a search space is probably something that is being disregarded at this point, but that has not stopped the march for more and more parameters. You might be thinking that nobody has really done that type of effort in practice.The M6 model happens to be 10 trillion parameters [7]. Yeah, a 10 trillion parameter model exists. That is mind bogglingly large.One of the longest papers with the most authors was published by a lot of people from Stanford was “On the opportunities and risks of foundation models” [8]. You probably could have guessed that I was going to throw the title of that paper into Google Scholar to see what other people were adding with that citation to the scholarly world aka the academy [9]. That query offered up about 565 results to consider.Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. https://arxiv.org/pdf/2108.07258.pdfThis link will take you right to the 412 papers that Google Scholar believes have a direct citation to that massive 212 page paper [10].One of the real concerns when that paper got published was that it covered so much ground and had so many coauthors that it would very quickly become an anchor citation that was heavily cited. Some people were worried it would just become a stock or default citation for people in literature reviews. I’m pretty sure that the number of scholars that came together on that work will pretty much guarantee that it gets cited a ton going forward. The other element that will help with that is that the paper is highly readable for people wanting to learn and understand large language models. Together those two elements of it being useful to read and known by a large number of scholars from the start pretty much guarantee that people will hear about it for years to come.Links and thoughts:“[ML News] Multiplayer Stable Diffusion | OpenAI needs more funding | Text-to-Video models incoming”“We've Made Some Big Mistakes - WAN Show November 18, 2022”Top 5 Tweets of the week:Footnotes:[1] https://venturebeat.com/ai/how-2022-became-the-year-of-generative-ai/[2] [3] https://huggingface.co/blog/large-language-models[4] https://developer.nvidia.com/megatron-turing-natural-language-generation[5] https://arxiv.org/abs/1909.08053[6] https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/[7] https://towardsdatascience.com/meet-m6-10-trillion-parameters-at-1-gpt-3s-energy-cost-997092cbe5e8[8] https://arxiv.org/pdf/2108.07258.pdf[9] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22On+the+Opportunities+and+Risks+of+Foundation+Models%22&btnG=[10] https://scholar.google.com/scholar?cites=9595110325981705564&as_sdt=4005&sciodt=0,6&hl=en What’s next for The Lindahl Letter?* Week 97: MIT’s Twist Quantum programming language* Week 98: Deep generative models* Week 99: Overcrowding and ML* Week 100: Back to the ROI for ML* Week 101: Revisiting my MLOps paper* Week 102: ML pracademics* Week 103: Rethinking the future of ML* Week 104: That 2nd year of posting recapI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it wit

We are living on the edge of meeting the weekly Friday publishing deadline at this point. As we quickly approach the 104th post and the major two-year milestone I’m still working on the same Saturday and Sunday schedule of early morning writing. I’m just having to be extra mindful of making sure I don’t get sidetracked into working on other things. This week’s topic could be an entire book full of insight. People are certainly going to fill the shelves with quantum machine learning books in the coming years. It is certainly starting to turn into whole conferences and other gatherings of people interested in telling people all about and sharing stories of quantum machine learning.Nobody is really bringing a quantum computer to any of these events. These are not something that is going to fit in your car and be ready to head out to a conference event. Seriously, to the best of my knowledge no laptop or portable quantum computer exists at this time and if it did nobody is using it for machine learning. They would probably be taking it to conferences or other gathering to show people how delightfully wonderful it is to be able to carry around such a power computing device. I imagine that it will be like Steve Wozniak showing up and assembling an early homebrew computer club kit. Those were moments of endless possibility, delight, and wonder. Bringing back the chance at some type of epic moment like that is certainly something that could very well happen within this space. My money is on the team over at IBM making that happen at some point.Let’s rewind the coverage here for a moment and reflect on two of my previous Substack posts:Week 42: Time crystals and machine learning (this one was epic)Week 77: Is quantum machine learning gaining momentum?You may have forgotten from the first paragraph that this is week 95 of The Lindahl Letter and quantum computing has received 3 different weeks of coverage. You can tell from that degree of focus that I believe it is a topic that will eventually change the nature of compute. All right let’s jump right into the best academic articles about quantum machine learning.Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., & Lloyd, S. (2017). Quantum machine learning. Nature, 549(7671), 195-202. https://arxiv.org/pdf/1611.09347.pdfSchuld, M., Sinayskiy, I., & Petruccione, F. (2015). An introduction to quantum machine learning. Contemporary Physics, 56(2), 172-185. https://arxiv.org/pdf/1409.3097.pdfI did enjoy this 35-minute YouTube video about quantum computing from TechTechPotato which is hosted by Dr. Ian Cutress, “Quantum Computing: Now Widely Available!”Being able to see some of these videos with the crew from IBM make me feel better about this technology actually existing. This would be much easier to accept as science fiction. Within that video the part of the coverage that caught my attention the most was that really it was academics and some startups that were coding for practical use cases to do something with quantum computing. For the most part, the type of machine learning efforts that would be easily transition over into this type of compute are not called out super clearly.One of the things I’m considering for next year is maybe coding something simple up and running it on one of these IBM systems. To really get a handle on what is happening within this quantum computing system it feels like only some hands work will close the gap for me here.Links and thoughts:“The Uses of IBM's Next Generation 433 Qubit Chip”“#036 - Max Welling: Quantum, Manifolds & Symmetries in ML”This is a podcast from the folks who are in the room with Twitter… “E104: FTX collapse with Coinbase CEO Brian Armstrong + election results, macro update & more”Top 5 Tweets of the week:Footnotes:None.What’s next for The Lindahl Letter?* Week 96: Where are large language models going?* Week 97: MIT’s Twist Quantum programming language* Week 98: Deep generative models* Week 99: Overcrowding and ML* Week 100: Back to the ROI for MLI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

I must have missed it when Samuel K. Moore covered this topic back on February 24, 2022 in an article titled, “RISC-V AI Chips Will Be Everywhere Esperanto Technology’s chip heralds new era in open-source architecture; Intel set to cash in,” [1]. I’m going to admit that I did fix the spelling on the word technology when I cut and pasted that title over to this Substack post. Fast forward just a couple of months and in April an article in Forbes talked about people actually sampling the 1,000 core hardware [2]. Within the broader context of things 1,000 cores is a lot of computing cores for a CPU chip. This computer for example is running an Intel i7 series chip and has 6 cores. You can see the difference in the number of cores is very large.You can pretty easily get to the website for Esperanto Technologies https://www.esperanto.ai/ and it has some information on it. They clearly believe that RISC-V is the future of computing innovation. For those of you who might be chuckling that “RISC architecture is going to change everything” yes that line from the 1995 feature film Hackers certainly continues to resonate with people. The reduced instruction set computer architecture certainly has had a strong run. I’m pretty sure that the RISC-V design is about 12 years old (2010) and it really is an open standard instruction set architecture.You can read about at the Berkeley Architecture Research site here: https://bar.eecs.berkeley.edu/projects/riscv.html Some of you may have already guessed that I was going to search Google Scholar for anything on this topic that might be interesting [3]. You can pretty quickly dig into the 18 results that came back for that search. Here are a sampling of those results:Imran, H. A., Mujahid, U., Wazir, S., Latif, U., & Mehmood, K. (2020). Embedded development boards for edge-AI: A comprehensive report. arXiv preprint arXiv:2009.00803. https://arxiv.org/ftp/arxiv/papers/2009/2009.00803.pdfReuther, A., Michaleas, P., Jones, M., Gadepally, V., Samsi, S., & Kepner, J. (2022). AI and ML Accelerator Survey and Trends. arXiv preprint arXiv:2210.04055. https://arxiv.org/pdf/2210.04055.pdfDokic, K., Mikolcevic, H., & Radisic, B. (2021). Inference speed comparison using convolutions in neural networks on various SoC hardware platforms using MicroPython. In RTA-CSIT (pp. 67-73). http://ceur-ws.org/Vol-2872/paper10.pdfOverall, I read this article in Interesting Engineering that talked about RISC-V having shipped more than 10 billion cores already [4]. The only reference to RISC-V AI in that article does reference Esperanto and its in the last paragraph. I was really hoping to find more content about these AI hardware chips. My guess here is that some other phrase is being used to describe the technology. I had spent some time looking around at searches related to, “Esperanto Technologies competitors.” That did not really yield anything major to share here. This is one I will need to circle back to at some point to see how AI specific hardware is changing. My efforts going down that road ended up looking at the IBM, “AI Hardware Center” [5]. I feel like a lot more companies are working in this space and more coverage could help highlight that at some later point.Links and thoughts:“Why Signal won’t compromise on encryption, with president Meredith Whittaker”“Stanford Seminar - ML Explainability Part 1 I Overview and Motivation for Explainability”Top 6 Tweets of the week:Footnotes:[1] https://spectrum.ieee.org/risc-v-ai[2] https://www.forbes.com/sites/karlfreund/2022/04/20/risc-v-startup-esperanto-technologies-samples-first-ai-silicon/?sh=37506f9c773d[3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22risc-v+ai%22&btnG=[4] https://interestingengineering.com/innovation/rise-of-risc-v-the-computer-chip[5] https://research.ibm.com/collaborate/ai-hardware-centerWhat’s next for The Lindahl Letter?* Week 95: Quantum machine learning* Week 96: Generative AI: Where are large language models going?* Week 97: MIT’s Twist Quantum programming language* Week 98: Deep generative models* Week 99: Overcrowding and MLI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are going to get to the 104th Substack post before you know it here for The Lindahl Letter publication. Things are moving along, and we are on the very tail end of that journey. Don’t panic about this post not being full of links. (Spoiler alert) You will have plenty of perspectives to read this week that are linked to for your reading pleasure.This is one of the topics that deserves a lot of attention. I circle back to asking people to always consider ethics within the context of both ML and AI. One of the considerations within that argument would be to really try to understand the outcomes of what the technology is being used to achieve or the negative externalities that would be possible from it’s use. You have heard me say it before and you will certainly hear it again, “Just because you can do a thing, does not mean you should.” It’s a real consideration within the AI/ML space. A lot of the things that can be done with both ML and AI are unconscionable and should be avoided. That is why ethics should be a core part of the AI/ML journey without question. Full stop.Beyond that consideration I tried to gather up a bunch of papers critical of ML in general. It was actually much harder in practice to find written criticism of ML than I expected in published article forms. You are certainly welcome to out and subscribe to the Substack of Gary Marcus who publishes, “The Road to AI We Can Trust,” [1]. A good scroll across those posts will give you a real sense of criticism and questions about the ML space. It has some really solid engagement as well from people who care enough to deeply question things. I wholesale consider that to be a healthy part of the process.Marcus, G. (2018). Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631. https://arxiv.org/ftp/arxiv/papers/1801/1801.00631.pdfNakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12), 124003. https://arxiv.org/pdf/1912.02292.pdfLake, B., & Baroni, M. (2018). Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. https://openreview.net/pdf?id=H18WqugAbMitchell, M. (2021). Why AI is harder than we think. arXiv preprint arXiv:2104.12871. https://arxiv.org/pdf/2104.12871.pdfBiderman, S., & Scheirer, W. J. (2020). Pitfalls in machine learning research: Reexamining the development cycle. http://proceedings.mlr.press/v137/biderman20a/biderman20a.pdfHenderson, P., & Brunskill, E. (2018). Distilling information from a flood: A possibility for the use of meta-analysis and systematic review in machine learning research. arXiv preprint arXiv:1812.01074. https://arxiv.org/pdf/1812.01074.pdfVinny, P. W., Garg, R., Padma Srivastava, M. V., Lal, V., & Vishnu, V. Y. (2021). Critical Appraisal of a Machine Learning Paper: A Guide for the Neurologist. Annals of Indian Academy of Neurology, 24(4), 481–489. https://doi.org/10.4103/aian.AIAN_1120_20 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8513942/You can get to a bit more content outside of scholarly articles when it comes to finding critics of machine learning. I’m going to share a handful of links to different things that I found interesting along the way to researching this Substack post.“5 myths about learning and innateness” https://open.substack.com/pub/garymarcus/p/5-myths-about-learning-and-innateness?r=8oh0m&utm_campaign=post&utm_medium=web“The Limitations of Machine Learning”https://towardsdatascience.com/the-limitations-of-machine-learning-a00e0c3040c6“When Machine Learning Goes Off the Rails”https://hbr.org/2021/01/when-machine-learning-goes-off-the-rails“The way we train AI is fundamentally flawed”https://www.technologyreview.com/2020/11/18/1012234/training-machine-learning-broken-real-world-heath-nlp-computer-vision/“Why deep-learning AIs are so easy to fool”https://www.nature.com/articles/d41586-019-03013-5“How a Pioneer of Machine Learning Became One of Its Sharpest Critics”https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/“AI researchers allege that machine learning is alchemy”https://www.science.org/content/article/ai-researchers-allege-machine-learning-alchemyLinks and thoughts:“Generative AI is Here. Who Should Control It?”“Twitter is now an Elon Musk company”“Apple's new App Store tax, Microsoft Surface reviews, and Meta's earnings”“Emergency Pod: Elon Musk Owns Twitter”Top 5 Tweets of the week:Footnotes:[1] Gary Marcus’s Substack “The Road to AI We Can Trust”What’s next for The Lindahl Letter?* Week 94: AI hardware (RISC-V AI Chips)* Week 95: Quantum machine learning* Week 96: Generative AI: Where are large language models going?* Week 97: MIT’s Twist Quantum programming language* Week 98: Deep generative modelsI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or re

It seems like having a national artificial intelligence initiative is popular these days. Back on February 18, 2022, I shared my week 56 Substack post, “Comparative analysis of national AI strategies.” That missive continues to get a good bit of traffic so I thought now would be a good time to go ahead and revisit national AI strategies, advisory committees, institutes, legislation, and the myriad of research institutes or think tanks that are jumping into this area of consideration. This is an area that I think could be a good place for some solid academic contributions. Instead of digging into all of those areas my attention really got focused on one advisory committee. That will become clear here in the next couple of sections of content.This week I have considered shifting The Lindahl Letter over to being an AI strategy advisory committee after spending a bunch of time reading about them this week. I’m not going to do that as it would limit my creative output to just one area and that sounds intellectually exhausting. One of them you can read about would be the National AI Advisory Committee (NAIAC) [1]. The next committee meeting was about to happen before writing this post. I had plans to listen live and I was totally signed up for everything [2]. Go forward I’m fully registered and signed up for alerts from the NAIAC. I would be happy to provide them guidance on effective national AI strategies from a comparative perspective, but that has not happened so far. This topic is an interesting space to consider at length. We are seeing a huge amount of academic work and companies like Hugging Face democratizing AI through community. Consider for a moment just how fast stable diffusion showed up and then was actively built into things and deployed. We are seeing massive changes within the ML/AI space and the deployment cycle is super-fast based on how interconnected the community happens to be worldwide. That has huge ramifications for any advisory committee considering the national level of AI strategy. Adapting to the rate of change and decentralized nature of things requires a different type of national AI strategy. I’ll be listening to the NAIAC in October to see how things are going. You can find the sessions on YouTube by searching for “NAIAC” pretty easily.“National Artificial Intelligence Advisory Committee (NAIAC) Meeting”“National Artificial Intelligence Advisory Committee (NAIAC) Field Hearing”You could read the meeting minutes from May 4, 2022.https://www.ai.gov/wp-content/uploads/2022/07/NAIAC-Minutes-05042022.pdfI went out to Google Scholar and took a look to see if anybody had published or shared anything with this advisory committee referenced [3]. Nothing really came up except the above-mentioned meeting minutes from May 4, 2022. Nothing really showed up during a search of arXiv either [4]. It’s possible in about 6 months more content will show up reacting to the hours of meetings that are linked above. Right now, we appear to be a little bit ahead of things in terms of reactions to the work being done by this advisory committee. I’m going to keep an eye out for more content related to NAIAC. It’s possible sometime next year it will be the right time to dig back into this one.Links and thoughts:“5 Practical Machine Learning Lessons You’re NOT Taught in School”“This Has Never Happened Before - WAN Show October 14, 2022”“Stanford CS330: Deep Multi-Task & Meta Learning I 2021 I Lecture 4”“Confusing new Apple products, Netflix password sharing, and NFT cults”Top 5 Tweets of the week:Footnotes:[1] https://www.ai.gov/naiac/[2] https://events.nist.gov/profile/form/index.cfm?PKformID=0x17861abcd[3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=National+Artificial+Intelligence+Advisory+Committee&btnG=[4] https://search.arxiv.org/?in=&query=%22National%20Artificial%20Intelligence%20Advisory%20Committee%22 What’s next for The Lindahl Letter?* Week 93: Papers critical of ML* Week 94: AI hardware (RISC-V AI Chips)* Week 95: Quantum machine learning* Week 96: Where are large language models going?* Week 97: MIT’s Twist Quantum programming languageI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

For those of you that keep track of these types of things we are now in real time based on my publishing schedule. Over the course of the next few weeks no backlog exists as we make the run to 104 consecutive Substack posts spanning 2 years of content creation on this platform. It’s week 91 right now in the publishing schedule which means that only 13 blocks of super exciting writing about machine learning stand between you and the completion of that penultimate tasking. To that end I’ll be working without a sizeable backlog that would prevent procrastination or a loss of focus from breaking the streak. At this very moment, I’m probably writing about that consideration to help refocus my efforts on completing this last stretch. You may recall that after the 2-year mark I’m going to mix things up a bit and switch focus from machine learning to artificial intelligence in general. The format might change a bit as well, but you will have to stay tuned to see what ends up showing up every Friday.My physical notebook where I write things down by hand with a Parker Sonnet fountain pen includes a few different sketches related to building a universal request handler. A lot of the voice assistants receive queries that need to be given to some type of ML model to solve. Making a decision about which model to apply and then how to manage and sort those results back into the general knowledge graph is an interesting problem to solve. Generally, one solution is to just fire off the request to a series of API channels and the one that reports back a probable answer in the shortest time is what gets served up by the voice assistant. All of that happens so very quickly that you don’t really notice a long delay. We as people handle super complex reasoning tasks and work on things with an extreme depth without even questioning the solution selection process.Perhaps as a subset of that general selection of what model to use when questioning something else props up from researchers and practitioners. One of the things people who are involved with work in the ML models space ask from time to time is about why you cannot just combine all the ML models together and make a super model. One of the ways people are working to bring models together involves the ensemble method for machine learning models. This methodology involves making a few models and then combining them to improve results. This is not a method to just stack random models and try to make it work. The ensemble method is a technique that is based on working with the same dataset and maybe combining for example a bunch of favorable random forests or some other set of similar models to form an ensemble. From what I have been able to tell from reading articles in this space its not a super solution to just bring all machine learning models together in one unified model theory.Dietterich, T. G. (2000, June). Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Springer, Berlin, Heidelberg. https://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdfDietterich, T. G. (2002). Ensemble learning. The handbook of brain theory and neural networks, 2(1), 110-125. https://courses.cs.washington.edu/courses/cse446/12wi/tgd-ensembles.pdfThis one is out and in use in the wild. You can actually utilize ensemble ML models from some of the systems like scikit-learn [1]. You can also pretty quickly implement ensemble models with the “The Functional API” as a part of TensorFlow core [2]. You can pretty quickly get up to speed and use this one in notebooks or other places.Links and thoughts:Lex Fridman Podcast “#324 – Daniel Negreanu: Poker”Lex Fridman Podcast “#315 – Magnus Carlsen: Greatest Chess Player of All Time”“Microsoft's Surface event, Pixel 7 and Pixel Watch reviews, and Meta Connect 2022”“Mark Zuckerberg on the Quest Pro, future of the metaverse, and more”Top 5 Tweets of the week:Footnotes:[1] https://scikit-learn.org/stable/modules/ensemble.html[2] https://www.tensorflow.org/guide/keras/functionalWhat’s next for The Lindahl Letter?● Week 92: National AI strategies revisited● Week 93: Papers critical of ML● Week 94: AI hardware (RISC-V AI Chips)● Week 95: Quantum machine learning● Week 96: Where are large language models going?I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

The post this week is going to be on the shorter side of things. I think that is in part due to the very straightforward nature of the topic under consideration. It really could have just been a link to a single book on the subject with a polite note that reading it would help you understand pretty much everything you need to know. To that end, it looks like the book on probabilistic machine learning from Kevin Patrick Murphy has been downloaded 168 thousand times [1]. That is pretty darn good for something in the machine learning space where the inflection point is generally over or under around 10,000 points of interest on a topic. It appears that Kevin really surpassed that ceiling by a ton of downloads. The book is very easy to get to and the search engines really seem to algorithmically love it as well. Given that this topic has a lot of refences to Bayesian decision theory you probably could predict that it would get my full attention. The topics that generally grounds all of my efforts in the machine learning space is that background in statistics and my enjoyment of working with Bayesian pooling. Let’s begin to breakdown the idea of probabilistic machine leaning involves understanding two general steps. First, you must accept that you want to explain observed data with your machine learning models. Second, those explanations are going to need to come from inferring plausible models to aid you in that explanation. Together those two steps help you begin to evaluate data in a probabilistic way which means that you are aided by the power of statistical probability grounding you to a rational approach. To me this sort of spells out an approach that is not based on randomness or anything particularly chaotic.Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press. https://research.google/pubs/pub38136.pdfProbabilistic machine learning papersGhahramani, Z. (2015). Probabilistic machine learning and artificial intelligence. Nature, 521(7553), 452-459. https://www.repository.cam.ac.uk/bitstream/handle/1810/248538/Ghahramani%25202015%2520Nature.pdf?sequence=1Rain, C. (2013). Sentiment analysis in amazon reviews using probabilistic machine learning. Swarthmore College. https://www.sccs.swarthmore.edu/users/15/crain1/files/NLP_Final_Project.pdfProbabilistic deep learning papersNie, S., Zheng, M., & Ji, Q. (2018). The deep regression bayesian network and its applications: Probabilistic deep learning for computer vision. IEEE Signal Processing Magazine, 35(1), 101-111. https://sites.ecse.rpi.edu/~cvrl/Publication/pdf/Nie2018.pdfPeharz, R., Vergari, A., Stelzner, K., Molina, A., Shao, X., Trapp, M., ... & Ghahramani, Z. (2020, August). Random sum-product networks: A simple and effective approach to probabilistic deep learning. In Uncertainty in Artificial Intelligence (pp. 334-344). PMLR. http://proceedings.mlr.press/v115/peharz20a/peharz20a.pdfAndersson, T. R., Hosking, J. S., Pérez-Ortiz, M., Paige, B., Elliott, A., Russell, C., ... & Shuckburgh, E. (2021). Seasonal Arctic sea ice forecasting with probabilistic deep learning. Nature communications, 12(1), 1-12. https://www.nature.com/articles/s41467-021-25257-4?tpcc=nleyeonaiLinks and thoughts:“How Arm conquered the chip market without making a single chip, with CEO Rene Haas”Top 5 Tweets of the week:Footnotes:[1] https://probml.github.io/pml-book/book1.htmlWhat’s next for The Lindahl Letter?* Week 91: What are ensemble ML models?* Week 92: National AI strategies revisited* Week 93: Papers critical of ML* Week 94: AI hardware (RISC-V AI Chips)* Week 95: Quantum machine learningI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

A lot of people talk about deploying AI in the business world and almost all that conjecture is entirely based on deploying a machine learning model into a production environment or some interesting POC. When those same people deploy an actual AI product into production, they will hopefully see the difference. They are not the same. A lot of the AI hype is underpinned by advances in machine learning. Artificial general intelligence or more commonly abbreviated as AGI represents an interesting summation of possibility contained in a name. You have seen representations of AGIs in books, movies, comics, and all sorts of works of fictions. At the moment, machine learning models are generally trained to do one thing well and cannot generally pick up and learn tasking like a person would.That is why most of those fiction writers do not bother to include a machine learning model as the antagonist in stories. The expectation is that a person (or villain for that matter) would be able to generally pick up and learn tasking for a wide variety or purposes. That exception gets rolled up into what an AGI would be expected to achieve in practice. Generally, the expectation would be that the AGI could complete a mix of tasking just like a person would be able to handle. You would need a large number of machine learning models to complete the tasking that a person does in a single day. You could test this as a practical exercise with a sheet of paper and a pen throughout the day. As you built up the list of machine learning models you would need throughout the day to accomplish all the various tasking it would become very obvious that your ML model is not an AGI as your list would be much greater than a single model. Or even a small collection of models being sorted out by a piece of software upfront. To be fair to the idea contained within that point, we don’t even have a good method to switch between a collection of ML models to deploy a collection to complete a variety of tasks.Artificial general intelligence - Let’s begin by digging into a few books and papers related to AGI before introducing the ML part of the equation. This will help create a foundation for the concept and scholarly evaluation of AGI without spending as much time on ML. You will find a theme in the literature here were Goertzel is prominently featured.Goertzel, B., Orseau, L., & Snaider, J. (2015). Artificial general intelligence. Scholarpedia, 10(11), 31847. http://var.scholarpedia.org/article/Artificial_General_IntelligenceGoertzel, B. (2007). Artificial general intelligence (Vol. 2). C. Pennachin (Ed.). New York: Springer. https://www.researchgate.net/profile/Prof_Dr_Hugo_De_GARIS/publication/226000160_Artificial_Brains/links/55d1e55308ae2496ee658634/Artificial-Brains.pdfGoertzel, B. (2014). Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1), 1. https://sciendo.com/abstract/journals/jagi/5/1/article-p1.xmlI did discover along the way that Dr. Ben Goertzel who has papers referenced above has made a lot of content on YouTube. You may remember some of the Sophia the robot content (Hanson Robotics) from 2016 to 2018 as it was fairly prevalent in the media. You can read and article from The Verge about this one [1].If you wanted to dig into a more video based set of content, then feel free to check out this 7 video playlist on the general theory of general intelligence:https://www.youtube.com/playlist?list=PLAJnaovHtaFTK9E1xHnBWZeKtAOhonqH5Machine learning – This next set of research will consider both ML and AGI together.Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., ... & Shi, L. (2019). Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 572(7767), 106-111. https://aiichironakano.github.io/cs653/Pei-ArtificialGeneralIntelligenceChip-Nature19.pdfSilver, D. L. (2011, August). Machine lifelong learning: Challenges and benefits for artificial general intelligence. In International conference on artificial general intelligence (pp. 370-375). Springer, Berlin, Heidelberg. https://www.researchgate.net/profile/Daniel-Silver-3/publication/221328970_Machine_Lifelong_Learning_Challenges_and_Benefits_for_Artificial_General_Intelligence/links/00463515d5bc70ed5c000000/Machine-Lifelong-Learning-Challenges-and-Benefits-for-Artificial-General-Intelligence.pdfGoertzel, B. (2014). Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1), 1. https://sciendo.com/abstract/journals/jagi/5/1/article-p1.xmlConclusion – Back during week 62, I started to question how close we were to touching the singularity and that question aligns somewhat to when we will see a true AGI. A well referenced paper was mentioned titled, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion,” published in 2016 by Vincent C. Müller and Nick Bostrom [2].Müller, V. C., & Bostrom, N.

Brief aside: A bunch of shuffling has occurred in the forward-looking topics as we approach two years of The Lindahl Letter. Reworking the content for weeks 89 to 104 had to happen after the syllabus project. My focus and interest shifted a bit and due to that it made sense to go ahead and rework the pathing toward that extra special two-year anniversary of writing posts on Substack.That brief aside is now complete. Some congratulations are in order, you made it to the post where I am going to write about academic paper mills and the future of synthetic papers flooding the academy. Using GPT-2 and a million-word corpus of my own words I trained a model to mimic my writing style. Unlike some of the newer iterations of those models you could tell it was not ready to pass as human generated. Initially, I had thought this post was going to be about the nature of how the peer review academic system of gatekeeping was failing compared to writing about the problematic possibility of synthetic writing being able to approach something that could pass the gatekeeping. Instead of going in that direction it was during this research that I realized the flooding problem of endless content creation was far worse than the breakdown of the academy-based gatekeeping system. Academic gatekeeping is a function of the quality of the gatekeepers and the rules they apply. That is inherent within the academy system, but it is being tested in a way that it has not had to endure before. Extreme oversupply of content is not going to slow down any time soon. To that end, I have spent a lot of time wondering about the future of publishing.Large language models have created a scenario where a bit of prompt engineering can help generate blocks of prose. Previously I discussed a bit of the automation that is occurring within the instant news and financial reporting sections of the media. Using some type of model-based generation they take a bit of news and generate a story related to it and that can go out almost immediately. I have wondered about how many papers in the academic space get created in this way [1]. You can find examples of academics submitting papers to see if they can fool the reviewers into allowing them into journals [2]. Some scholars have taken this maybe a step too far and initially tried to publish fake papers [3]. I’m worried that flooding might occur within the world of academic publishing with fake journals and fake papers creating chaos.Any field of academic study where a key journal exists and the academics within that field have a strong network and focus on the work in that journal or maybe a handful of key journals the system of academic publishing is probably still working well enough to unify the field. Within the field of machine learning things have broken down to the point where a lot of the content that I read is not from peer reviewed academic journals or prestigious conferences. I read a lot of preprints and things that people have shared. You could go through my entire independent introduction to machine learning syllabus and only really consume open access academic works [4]. The number of academic journals focused on machine learning is really (really) large and appears to be growing. That is one of the reasons that I really focused on citations to see trends and papers that are bubbling up to the top of active consideration. While you cannot totally trust citation counts as a metric of authority of ideas it is a solid way that can be used to gain single out of the noise that a paper might be worth reading.If I was given a vote about things, then I would convene a regular conference cadence and associate a conference journal with it where submitted papers could be aggregated based on some peer review system of the conference attendees served as the gatekeeping system. That conference to journal system is probably my preferred method of journal aggregation as it is becoming community standard based. The people who want to be a part of it and read the journal are working together to uphold standards on the work they contribute to the academy. Right now, the opposite of that is occurring where people are defaulting back to reading preprints of papers and sometimes those preprints have more citations than the final location where the work is published. I’m pretty sure that based on the paywalls for some of the journals its entirely possible that the preprint reading rate is an order of magnitude larger. My preference here is keyed to building community vs. the totality of the contribution to the academy. I believe both elements are important and should be considered.Links and thoughts:“Everyone knows what YouTube is. Few know how it really works.”“GTA leaks, TikTok search, and Apple reviews hotline”Top 5 Tweets of the week:Footnotes:[1] https://www.nature.com/articles/d41586-021-00733-5 [2] https://undark.org/2020/11/26/fake-paper-predatory-journal/ [3] https://www.theatlantic.com/ideas/archive/2018/10

Research Note: You made it all the way to week 8 of 8 for the ML syllabus. You can find the files from the syllabus being built on GitHub. The latest version of the draft in PDF form can be found here. This lecture is going to be provided in two parts. First, I’m going to provide you with a few scholarly articles that dig into what MLOps involves and how researchers are addressing the topic. Second, I’ll provide you my insights on the topic of MLOps which I have been presenting for the last few years. When you get to the point of applying ML techniques in production you will end up needing MLOps.MLOps research papersAlla, S., & Adari, S. K. (2021). What is mlops?. In Beginning MLOps with MLFlow (pp. 79-124). Apress, Berkeley, CA. https://arxiv.org/pdf/2103.08942.pdf[Zugriffam09.09.2021Zhou, Y., Yu, Y., & Ding, B. (2020, October). Towards mlops: A case study of ml pipeline platform. In 2020 International conference on artificial intelligence and computer engineering (ICAICE) (pp. 494-500). IEEE. https://www.researchgate.net/profile/Yue-Yu-126/publication/349802712_Towards_MLOps_A_Case_Study_of_ML_Pipeline_Platform/links/61dd00575c0a257a6fdd62f3/Towards-MLOps-A-Case-Study-of-ML-Pipeline-Platform.pdfRenggli, C., Rimanic, L., Gürel, N. M., Karlaš, B., Wu, W., & Zhang, C. (2021). A data quality-driven view of mlops. arXiv preprint arXiv:2102.07750. https://arxiv.org/pdf/2102.07750.pdfRuf, P., Madan, M., Reich, C., & Ould-Abdeslam, D. (2021). Demystifying mlops and presenting a recipe for the selection of open-source tools. Applied Sciences, 11(19), 8861. https://www.mdpi.com/2076-3417/11/19/8861/pdfMy insights about MLOpsConceptually, I have been breaking down the categories of applied ML deployment based use cases into three buckets:* Bucket 1: “Things you can call” e.g. external API services* Bucket 2: “Places you can be” e.g. ecosystems where you can build out your footprint (AWS, GCP, Azure, and many others that are springing up for MLOps delivery)* Bucket 3: “Building something yourself” e.g. open source and self tooled solutionsThese buckets will impact your ability to run MLOps and how much control you have over the frameworks and underlying data pipes. Bucket one is the easiest to implement because all you have to do is go out and consume it. You just need to connect to it, send some information out to it, get some information back, and you're ready to go. Bucket two is really about places where you can be totally within an ecosystem where you can build out your footprint for the endeavor. AWS, Azure, and GCP and many others that are springing up for MLOps delivery. I do mean many others are ready to provide you an ecosystem. You should be starting to see other ecosystems become available besides the major three. They are popping up and they're going to provide a different workflow in a different place where you can serve up your ML models and to be able to get going in this space. Now the third category or bucket three is where you will be building something yourself. These are the open source and self-tooled solutions. A few years ago, this space was the primary place people were building and now we are seeing a shift. We're seeing that movement into other buckets. Those API based solutions are so readily available and you can get into these ecosystems where you can get going so quickly. Things are moving around and changing. That categorization of the three buckets helps me think about where things are for use cases and where things are gonna happen. It's a very tactical question versus strategic one.Some of the major players within the information technology space are trying to break into the machine learning operations (or MLOps) space. Like anything else, picking the right tools to get things done is about matching the right technology and use case to achieve the best possible results. We are really starting to see some solid maturity in the MLOps space. The next stage will be either a round of purchasing where established players buy up the upstart players building MLOps or the established players will build out the necessary elements to move past the newer players in the enterprise level market.Let's look at the first technology in Table 1 which happens to be TensorFlow. You should not be surprised to see that TensorFlow has by far the largest influence at 154,162 stars. Getting a star requires a GitHub user to click the star function. People have really placed a lot of attention on TensorFlow. It has 2,933 contributors that means that almost 3,000 people are contributing to TensorFlow. From that point you can see that PyTorch drops off considerably. It's going from around 154k stars to just 47k stars. The number of contributors drops off significantly as well. Now, you're down to around 1,785. Now on the PyTorch example, they do have 4,620 branches which honestly I don't know why you would want to look at that many branches. No human wants to manage that many branches of anything. That is unmana

This set of topics was either going to be the foundation to start this series or it was going to be collected as a set of thoughts at the end. You can tell that obviously I demurred from starting with ethics, fairness, bias, and privacy in machine learning until the full foundation was set for the topics under consideration. These topics are not assembled as an afterthought and are very important to any journey within the machine learning space. This technology in terms of machine learning and artificial intelligence has the potential to be near omnipresent in day to day life and certainly within anything where decision making or anything digital persists. Each of these topics is going to receive a solid overview followed by a series of scholarly articles like the previous lectures. You are now well aware from seeing dozens of other scholarly articles that these topics do not appear in each and every work and while they are conceptually foundational as intellectual guardrails they are not consistently presented that way in literature reviews or considerations for the practical work occurring within the machine learning space. I would clearly argue and have for years that just because you can do a thing does not mean that you should. You have to consider the consequences and realities of bringing that thing forward in a world where models and methods are so readily shared on GitHub and other platforms. Overlap certainly occurs between the topics of ethics, fairness, bias, and privacy within the machine learning academic space. I have tried to sort the articles to help enhance readability within the different categories, but you will see some overlap. Ethics - This topic got covered back in week 65. I’m going to rework part of that content here so if it feels familiar that is consistent with it appearing before about 20 weeks ago. Anybody preparing machine learning content should be comfortable with presenting ethics as a topic of consideration. I firmly believe and hope you would support that effort after coming along for this journey so far into this independent study syllabus. Ethics should be covered as a part of every machine learning course. Perhaps the best way to sum it up as an imperative would be to say, “Just because you can do a thing does not mean you should.” Machine learning opens the door to some incredibly advanced possibilities for drug discovery, medical image screening, or just spam detection to protect your inbox. The choices people make with machine learning use cases is where the technology and ethics have to be aligned.No one really solid essay or set of essays on AI/ML ethics jumped out and caught my attention this week during my search. Part of my search involved digging into results from Google Scholar that yielded a ton of different options to read about “ethics in machine learning” [1]. A lot of those articles cover how to introduce ethics to machine learning courses and about the need to consider ethics when building machine learning implementations. Given that those two calls to action are the first things that come up and they are certainly adjacent to the primary machine learning content being shared it might make you take a moment to pause and consider how much the field of machine learning should deeply consider the idea that just because it can do something does not mean you should. Some use cases are pretty basic and the ethics of what is happening is fairly settled. Other use cases walk right up to the edge of what is reasonable in terms of fairness and equity.Lo Piano, S. (2020). Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward. Humanities and Social Sciences Communications, 7(1), 1-7. https://www.nature.com/articles/s41599-020-0501-9.pdf Greene, D., Hoffmann, A. L., & Stark, L. (2019). Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. https://scholarspace.manoa.hawaii.edu/server/api/core/bitstreams/849782a6-06bf-4ce8-9144-a93de4455d1c/content Fairness and Bias - Implementing machine learning algorithms generally involves working with imperfect datasets that have different biases that have to be accounted for and ultimately corrected. Corbett-Davies, S., & Goel, S. (2018). The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023. https://arxiv.org/pdf/1808.00023.pdf Chouldechova, A., & Roth, A. (2018). The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810. https://arxiv.org/pdf/1810.08810.pdf Barocas, S., Hardt, M., & Narayanan, A. (2017). Fairness in machine learning. Nips tutorial, 1, 2. https://fairmlbook.org/pdf/fairmlbook.pdf Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6), 1-35. https://arxiv.org/pdf/1908.09635.pdf Yapo, A., & W

Neuroscience is a complex topic to dig into in general. Studying the nervous system is a complex thing to do before you add in the concept of machine learning or artificial intelligence. Within the context of machine learning it gets even more interesting for academic researchers, practitioners, and anybody building neural networks. Understanding that context of complexity within any inquiry into neuroscience, it will make sense here to focus on 5 scholarly articles that could help provide a solid context here for the relationship between neuroscience and machine learning. Within this section of inquiry the articles are really going to bring forward the complexity of the issue. The scholarly articles selected to cover neuroscience include a lot of focus on how the two subjects work together and the future of that collaboration. Savage, N. (2019). How AI and neuroscience drive each other forwards. Nature, 571(7766), S15-S15. https://www.nature.com/articles/d41586-019-02212-4 Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., ... & Kording, K. P. (2019). A deep learning framework for neuroscience. Nature neuroscience, 22(11), 1761-1770. https://www.nature.com/articles/s41593-019-0520-2Marblestone, A. H., Wayne, G., & Kording, K. P. (2016). Toward an integration of deep learning and neuroscience. Frontiers in computational neuroscience, 94. https://www.frontiersin.org/articles/10.3389/fncom.2016.00094/pdf Richiardi, J., Achard, S., Bunke, H., & Van De Ville, D. (2013). Machine learning with brain graphs: predictive modeling approaches for functional imaging in systems neuroscience. IEEE Signal processing magazine, 30(3), 58-70. https://archive-ouverte.unige.ch/unige:33936/ATTACHMENT01 Vu, M. A. T., Adalı, T., Ba, D., Buzsáki, G., Carlson, D., Heller, K., ... & Dzirasa, K. (2018). A shared vision for machine learning in neuroscience. Journal of Neuroscience, 38(7), 1601-1607. https://www.jneurosci.org/content/jneuro/38/7/1601.full.pdf Bonus PapersThis section includes a few additional papers that I have enjoyed and thought you might as well. They are not sorted in any particular order. This section may see the most updates between first publication and any updates of this syllabus. I’m sure that papers will get recommended to be included and if they don’t naturally fit into the main structure without overloading the reader, then they will end up here in the bonus papers section. Marcus, G. (2018). Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631. https://arxiv.org/ftp/arxiv/papers/1801/1801.00631.pdf Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12), 124003. https://arxiv.org/pdf/1912.02292.pdf Lake, B., & Baroni, M. (2018). Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. https://openreview.net/pdf?id=H18WqugAb Mitchell, M. (2021). Why AI is harder than we think. arXiv preprint arXiv:2104.12871. https://arxiv.org/pdf/2104.12871.pdf Biderman, S., & Scheirer, W. J. (2020). Pitfalls in machine learning research: Reexamining the development cycle. http://proceedings.mlr.press/v137/biderman20a/biderman20a.pdf Henderson, P., & Brunskill, E. (2018). Distilling information from a flood: A possibility for the use of meta-analysis and systematic review in machine learning research. arXiv preprint arXiv:1812.01074. https://arxiv.org/pdf/1812.01074.pdf Links and thoughts:“The Future of AI is Self-Organizing and Self-Assembling (w/ Prof. Sebastian Risi)”“The Man behind Stable Diffusion”“Lab Naming Controversy - WAN Show August 26, 2022”Top 6 Tweets of the week:Research Note:You can find the files from the syllabus being built on GitHub. The latest version of the draft is being shared by exports when changes are being made. https://github.com/nelslindahlx/Introduction-to-machine-learning-syllabus-2022What’s next for The Lindahl Letter?* Week 86: Ethics, fairness, bias, and privacy (ML syllabus edition 7/8)* Week 87: MLOps (ML syllabus edition 8/8)* Week 88: The future of publishing* Week 89: your ML model is not an AGI* Week 90: What is probabilistic machine learning?* Week 91: What are ensemble ML models?* Week 92: National AI strategies revisited* Week 93: Papers critical of ML* Week 94: AI hardware (RISC-V AI Chips)* Week 95: Quantum machine learningI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

You may find in the literature that this topic of neural networks is sometimes called the zoo or more specifically, “the neural network zoo.” Corresponding to the articles that make this reference is a wonderful included graphic that shows a ton of different neural networks and can really give you a sense of how they work at the most fundamental level. Two papers that make this reference and include that wonderful graphic are from researchers at the Asimov institute that had papers published in 2016 and 2022. Both of those papers are great places to start learning about neural networks. Van Veen, F., & Leijnen, S. (2016). The neural network zoo. The Asimov Institute. https://www.asimovinstitute.org/neural-network-zoo/ Leijnen, S., & Veen, F. V. (2020). The neural network zoo. Multidisciplinary Digital Publishing Institute Proceedings, 47(1), 9. https://www.mdpi.com/2504-3900/47/1/9 That brief introduction aside. We are now going to focus on specific types of neural networks and next week our focus will shift to the topic of neuroscience. I have separated the two topics on purpose. Briefly, I had considered trying to combine the two topics as one set of content, but I think it would have become unwieldy in terms of trying to present a distinct point of view on both topics. Digging into neural networks is really about digging into deep learning and trying to understand it as a subfield of machine learning. Keep in mind that while machine learning is exciting it's just a small part of the broader grouping of artificial intelligence as a field of study. I’m going to provide a brief introduction and some links to scholarly articles for 9 types of neural networks that you might run into. This list is in no way comprehensive and is built and ordered based on my interests as a researcher. A lot of speciality models and methods exist. One of them could end up displacing something on the list if it proves highly effective. I’m open to suggestions of course for different models or even orders of explanation.* Artificial Neural Networks (ANN)* Simulated Neural Networks (SNN)* Recurrent Neural Networks (RNN)* Generative Adversarial Network (GAN) * Convolutional Neural Network (CNN)* Deep Belief Networks (DBN)* Self Organizing Neural Network (SONN)* Deeply Quantized Neural Networks (DQNN)* Modular Neural Network (MNN)Artificial Neural Networks (ANN) - This is the model that is generally shortened to just neural networks and it is a very literal title. An ANN is really an attempt or more accurately a computational model designed to either mimic or create a neural network akin to what is used within a biological brain using hardware or software. You can assume this model to be fundamental to any consideration of neural networks, but you are going to quickly want to dig into other more targeted models based on your specific use case. What you are trying to accomplish will certainly help you focus on a model or method that best meets the needs of that course of action. However, in the abstract people will consider how to build ANNs and what they could be used for as the technology progresses.Jain, A. K., Mao, J., & Mohiuddin, K. M. (1996). Artificial neural networks: A tutorial. Computer, 29(3), 31-44. https://www.cse.msu.edu/~jain/ArtificialNeuralNetworksATutorial.pdf Hassoun, M. H. (1995). Fundamentals of artificial neural networks. MIT press. https://www.researchgate.net/profile/Terrence-Fine/publication/3078997_Fundamentals_of_Artificial_Neural_Networks-Book_Reviews/links/56ebf73a08aee4707a3849a6/Fundamentals-of-Artificial-Neural-Networks-Book-Reviews.pdf Simulated Neural Networks (SNN) - As you work along your journey in the deep learning space and really start to dig into neural networks you will run into those ANNs and very quickly a subset of machine learning adjacent to that type of model called the simulated neural networks. Creating a neural network that truly mimics the depth and capacity of the brain is something to strive for right now and with that constraint it makes sense that work is being done to simulate the best possible representation we can achieve currently or a very special use case that limits the simulation. Using models that generate a simulation based on some complex sets of mathematics, these SNNs are being created to challenge certain use cases. One of the papers shared below is associated with figuring out the shelf life of processed cheese for example. Kudela, P., Franaszczuk, P. J., & Bergey, G. K. (2003). Changing excitation and inhibition in simulated neural networks: effects on induced bursting behavior. Biological cybernetics, 88(4), 276-285. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.57.9281&rep=rep1&type=pdf Goyal, S., & Goyal, G. K. (2012). Application of simulated neural networks as non-linear modular modeling method for predicting shelf life of processed cheese. Jurnal Intelek, 7(2), 48-54. https://ir.uitm.edu.my/id/eprint/34381/1/34381.pdf Recurrent Neural Networks (

During the last lecture we jumped in and looked at 10 machine learning algorithms. This week the content contained within this lecture will cover from a machine learning perspective reinforcement learning and 3 types of supervised learning. Those types of supervised learning will include the general use case of supervised learning, unsupervised learning, and the super interesting semi-supervised learning. Like the model for consideration used in the last lecture I’ll cover the topics in general and provide links to papers covering the topic to allow people looking for a higher degree of depth to dive deeper into academic papers to achieve that goal. My general preference here is to find academic papers that are both readable and are generally available for you to actually read with very low friction. Within the machine learning and artificial intelligence space a lot of papers are generally available and that is great for literature reviews and generally for scholarly work and practitioners working to implement the technology. My perspective is a mix between those two worlds which could be defined as a pracademic view of things. All right; here we go. Reinforcement learning - Welcome to the world of machine learning. This is probably the first approach you are going to learn about in your journey. That’s right, it's time to consider for a brief moment the world of reinforcement learning. You are probably going to need to start to create some intelligent agents and you will want to figure out how to maximize the reward those agents could get. One method of achieving that result is called reinforcement learning. A lot of really great tutorials exist trying to explain this concept and one that I enjoyed was from Towards Data Science way back in 2018 [1]. The nuts and bolts of this one involve trial and error with an intelligent agent trying to learn from mistakes using a maximization of reward function to avoid going down paths that don’t offer greater reward. The key takeaway here is that during the course of executing a model or algorithm a maximization function based on reward has to be in place to literally reinforce maximization during learning. I’m sharing references and links to 4 academic papers about this topic to help you dig into reinforcement learning with a bit of depth if you feel so inclined. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of artificial intelligence research, 4, 237-285. https://www.jair.org/index.php/jair/article/view/10166/24110 Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning. https://login.cs.utexas.edu/sites/default/files/legacy_files/research/documents/1%20intro%20up%20to%20RL%3ATD.pdf Szepesvári, C. (2010). Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1), 1-103. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.308.549&rep=rep1&type=pdf Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. https://arxiv.org/pdf/1312.5602.pdf Supervised learning - You knew it would only be a matter of time before we went out to some content from our friends over at IBM [2]. They note that within a world where you have some labeled datasets and are training an algorithm to engage in classification or perhaps regression, but probably classification. In some ways the supervised element here is the labeling and guiding of the classification. Outside of somebody or a lot of people sitting and labeling training data the supervision is not from somebody outright sitting and watching the machine learning model run step by step. Some ethical considerations need to be taken into account at this point. A lot of people have worked to engage in data labeling. A ton of services exist to help bring people together to help do this type of work. Back in 2018 Maximilian Gahntz published a piece in Towards Data Science that talked about the invisible workers that are doing all that labeling in large curated datasets [3]. Within the world of supervised learning being able to get high quality labeled data really impacts the ability to make solid models. It’s our ethical duty as researchers to consider what that work involves and who is doing that work. Another article in the MIT Technology Review back in 2020 covered the idea of how gig workers are powering a lot of this labeling [4]. The first academic article linked below with Saiph Savage as a co-author will cover the same topic and you should consider giving it a read to better understand how machine learning is built from dataset to model. After that article, the next two are general academic articles about predicting good probabilities and empirical comparisons to help ground your understanding of supervised learning. Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burc

Welcome to the lecture on ML algorithms. This topic was held until the 3rd installment of this series to allow a foundation for the concept of machine learning to develop. At some point, you are going to want to operationalize your knowledge of machine learning to do some things. For the vast majority of you one of these ML algorithms will be that something. Please take a step back and consider this very real scenario. Within the general scientific community getting different results every time you run the same experiment makes publishing difficult. That does not stop authors in the ML space. Replication and the process of verifying scientific results is often difficult or impossible without similar setups and the same datasets. Within the machine learning space where a variety of different ML algorithms exist that is a very normal outcome. Researchers certainly seem to have gotten very used to getting a variety of results. I’m not talking about using post theory science to publish based on allowing the findings to build knowledge instead of the other way around. You may very well get slightly different results every time one of these ML algorithms is invoked. You have been warned. Now let the adventure begin. One of the few Tweets that really made me think about the quality of ML research papers and the research patterns impacting quality was from Yaroslav Bulatov who works on the PyTorch team back on January 22, 2022. That tweet referenced a paper on ArXiv called, “Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers,” from 2021 [1]. That paper digs into the state of things where hundreds of optimization methods exist. It pulls together a really impressive list. The list itself was striking just in the volume of options available. My next thought was about just how many people are contributing to this highly overcrowded field of machine learning. That paper about deep learning optimizers covered a lot of ground and would be a good place to start digging around. We are going to approach this a little differently based on a look at the most common ones. Here are some (10) very common ML algorithms (this is not intended to be an exhaustive list):* XGBoost* Naive Bayes algorithm* Linear regression* Logistic regression* Decision tree* Support Vector Machine (SVM) algorithm* K-nearest neighbors (KNN) algorithm* K-means* Random forest algorithm* DiffusionI’m going to talk about each of these algorithms briefly or this would be a very long lecture. We certainly could go all hands and spend several hours all in together in a state of irregular operations covering these topics, but that is not going to happen today. To make this a more detailed syllabus version of the lecture I’m going to include a few references to relevant papers you can get access to and read after each general introduction. My selected papers might not be the key paper or the most cited. Feel free to make suggestions if you feel a paper better represents the algorithm. I’m open to suggestions. XGBoost - Some people would argue with a great deal of passion that we could probably be one and done after introducing this ML algorithm. You can freely download the package for this one [2]. It has over 20,000 stars on GitHub and has been forked over 8,000 times [3]. People really seem to like this one and have used it to win competitions and generally get great results. Seriously, you will find references to XGBoost all over these days. It has gained a ton of attention and popularity. Not exactly to the level of being a pop culture reference, but within the machine learning community it is well known. The package is based on gradient boosting and provides parallel tree boating (GBDT, GBM). This package generally creates a series of models that boost the trees and help create overfitting in sequential efforts. You can read a paper from 2016 about it on arXiv called, “XGBoost: A Scalable Tree Boosting System” [4]. The bottom line on this one is that you get a lot of benefits from gradient boosting built into a software package that can get you moving quickly toward your goal of success.Chen, T., & Guestrin, C. (2016, August). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining (pp. 785-794). https://dl.acm.org/doi/pdf/10.1145/2939672.2939785 Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., & Chen, K. (2015). Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4), 1-4. https://cran.microsoft.com/snapshot/2017-12-11/web/packages/xgboost/vignettes/xgboost.pdf Naive Bayes algorithm - You knew I would have to have something Bayes related near the top of this list. This one is a type of classifier that helps evaluate the probability or relationship between classes. One of the classes with the highest probability will be considered the most likely class. It also assumes that those features are independent. I found a pap

You can find a lot of quality explanations of the differences between the various flavors of machine learning [1]. This second lecture in the introduction to ML syllabus series should open with a series of the best literature reviews I could find and pull together to share. That will be the second part of this lecture. The third part will cover the intersection of programming languages. Some rather high quality textbooks and manuscripts exist within the field of machine learning. You can even find ones for free on GitHub and other places. Instead of starting with the obvious way to go by digging into some weighty tomes. I’m going to spend some time sharing readouts of some of the most highly cited machine learning papers. For a lot of people jumping into the field they are working on something in a different field of study and find a use case or a business related adventure that could benefit from machine learning. Typically at this point they are going to start digging into software and can get going very rapidly. That part of the journey requires no real deep dive into the relevant literature. It’s great that people can just jump in and find machine learning accessible. However, (you knew that was coming) the next phase in the journey is when people start wondering about the why and how of what is happening or they dig deep enough that they may want to know about the foundations of the technology or techniques they are using. At that point, depending on what is being done people will see a massive number of papers published and shared online. The vast majority are available to freely download and read. Part 1: Highly cited machine learning papersWithin this section I’m going to try to build out a collection of 10 things you could read to start getting a sense of what papers within the machine learning space are highly cited. That is not a measure of readability or how solid of a literature review for machine learning they provide. You will find that most of them do not have really lengthy literature sections. The authors make the citations they need to make for related work and jump into the main subject pretty quickly. I’m guessing that is a key part of why they are highly cited publications. To begin with; from what I can tell, the most highly cited and widely shared paper of all time in the machine learning or deep learning space has over 125,285 citations that Google Scholar is aware of and can index. That is the first paper in the list below.1. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). https://arxiv.org/abs/1512.03385?context=cs This paper is cited a ton of times and has a pretty solid references section. If you read it after seeing the link above, then you would run into a bit of introduction on deep convolutional neural networks and then it would jump into some related work sections on residual representations, shortcut connections, and finally deep residual learning. While this paper is cited well over one hundred thousand times it is not designed to be an introduction to machine learning. It’s 12 pages and it provides a solid explanation of using deep residual learning for doing image recognition. To that end, this paper is highly on point and easy to read which is probably why so many people have cited it from 2016 to now. 2. Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255-260. https://www.science.org/doi/abs/10.1126/science.aaa8415Within the start of this review you are going to get a lot more of an introduction to what machine learning involves and I’m not surprised this work is highly cited. 3. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015). https://doi.org/10.1038/nature14539 This one is a very readable paper. It was certainly written to be widely read and is very consumable. It has 103 citations as well which is an intense number. 4. Ioffe, S., & Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). PMLR. https://arxiv.org/pdf/1502.03167.pdf 5. Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28. https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf 6. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf 7. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint

You might remember the Substack post from week 57 titled, “How would I compose an ML syllabus?” We have now reached the point in the program where you are going to receive 8 straight Substack posts that would combine together to compose what I would provide somebody as an introduction to machine learning syllabus. We are going to begin to address the breadth and depth of the field of machine learning. Please do consider that machine learning is widely considered just a small slice of the totality of artificial intelligence research. As a spoken analogy, you could say that machine learning is just one slice of bread in the loaf that is artificial intelligence. I did seriously entertain the idea of organizing the previous 79 posts into a syllabus based format for maximum delivery efficiency. That idea gave way quickly as it would be visually and topically overwhelming and that is the opposite of how this content needs to be presented. Let’s take this in the direction it was originally intended to take. To that end, let’s consider the framework that back during the week 57 writing process I thought was important. My very high level introduction to the creation of a machine learning syllabus from back in week 57 on February 25, 2022, would center on 8 core topics:* Week 80: Bayesian optimization (ML syllabus edition 1/8)* Week 81: A machine learning literature review (ML syllabus edition 2/8)* Week 82: ML algorithms (ML syllabus edition 3/8)* Week 83: Machine learning Approaches (ML syllabus edition 4/8)* Week 84: Neural networks (ML syllabus edition 5/8)* Week 85: Neuroscience (ML syllabus edition 6/8)* Week 86: Ethics, fairness, bias, and privacy (ML syllabus edition 7/8)* Week 87: MLOps (ML syllabus edition 8/8)That is what we are going to cover. At the end of the process, I’ll have a first glance at an introduction to machine learning syllabus. My efforts are annotated and include some narrative compared to a pure outline based syllabus. Bringing content together that is foundational is an important part about building this collection. At this point, just describing the edge of where things are in the field of machine learning would create something that would only be current for a moment and would fade away as the technology frontier curve advances. Instead of going that route it will be better to build a strong foundation for people to consume that will support the groundwork necessary to move from introductory to advanced machine learning. Yes, you might have caught from that last sentence that at some point I’ll need to write the next syllabus as a companion to this one. Stay tuned for a future advanced machine learning syllabus to go along with this introductory to machine learning edition. Enough overview has now occurred. It’s time to get started…Introduction to ML: Bayesian optimization (Lecture 1 of 8)I remember digging into Armstrong’s “Principles of forecasting” book which was published back in 2001 [1]. You can get a paper copy or find it online for a lot less than the $429 dollars Springer wants for the eBook. I thought the price was a typo at first, but I don’t think it actually is a typo. It’s just another example of how publishers are confused about how much academic work should cost for students to be able to read. Within that weighty tome of knowledge you can find coverage of the concept of Bayesian pooling which people have used for, “Forecasting analogous time series.” That bit of mathematics is always where my thoughts wander when considering Bayesian optimization. I have spent a lot of time researching machine learning and I really do believe most of the statistical foundations you would need to understand the field could be found in the book, “Principles of forecasting: A handbook for researchers and practitioners.” I do not think you should pay $429 dollars for it, but it is a wonderful book. Keep in mind that the book does not mention machine learning at all. It is from 2001 and does not really consider how forecasting tools would be extended within the field of machine learning. A lot of machine learning use cases are based on observation and the prediction of things. That is pretty much at the heart of the mathematics of forecasting. You need to understand the foundations of the statistical paradigm that Thomas Bayes introduced a couple hundred years ago in the 1700’s. The outcome of that journey will be the simple aside that we are about to work toward inferring some things. Yes, at this point in the journey we are about to work on inference. You could move directly to the point and examine Peter Frazier’s 2018 “A Tutorial on Bayesian Optimization” paper [2]. You may want to extend that analysis to figure out all the connected papers [3]. Instead of wandering off into the vast collection of papers that are connected to that one I started to wonder about a very different set of questions. You may have wondered as well if Bayesian optimization is an equation. Within the field of ma

Transformers were the thing. They were a big thing in the machine learning field. It was glorious. People talked about them a lot and papers were published. Oh so many papers were published. Now it feels like diffusion might be the thing. You will find that the thing of the moment in the field of machine learning shifts rapidly. I was looking at a GitHub repository based on, “high-Resolution Image Synthesis with Latent Diffusion Models,” and it has over 2,000 stars and has been forked 242 times [1]. I started reading this Tweet from Sebastian Raschka back on January 30, 2022 that asked the question, “Has anyone tried diffusion-based models, yet? Heard that they produce better results than GA” [2]. That Tweet linked out to a paper on ArXiv called, “Diffusion Models Beat GANs on Image Synthesis” [3]. It was published back during May of 2021 by OpenAI researchers and has seen 4 revisions so far. The paper loaded very slowly for me which was surprising. Rarely do I ever watch an update bar slowly creep across the screen waiting for a file to load up. It was 44 pages and 38 megabytes of data. That file should have arrived a lot faster. I took a look at another GitHub repository on guided diffusion from OpenAI that had 1,700 stars [4]. The audience for these diffusion code sets seems to be about 2,000 people which is interesting. Machine learning in general gets roughly 10,000 people focusing on things making this a subset within that slightly larger universe of attention. Given that we have moved into the 2nd paragraph it might be a good time to talk about what exactly diffusion might be in the context of machine learning. Over in the field of thermodynamics you could study gas molecules. Maybe you want to learn about how those gas molecules would diffuse from a high density to a low density area and you would also want to know how those gas molecules would reverse course. That is the basic theoretical part of the equation you need to absorb at the moment. Within the field of machine learning people have been building models that learn how based on degree of noise to diffuse the data and then reverse that process. That is basically the diffusion process in a nutshell. You can imagine that the cost to do this is computationally expensive. Let’s jump from OpenAI over to the Google AI team who wrote about, “High Fidelity Image Generation Using Diffusion Models” [5]. If you get to read that last link from the Google AI team then you will get to see a bunch of examples of how this works in practice. Imagine a lower quality image that is smaller, being increased in both quality and size. Now you need to imagine that happening again which is what makes the model they are using seem really novel and exciting. I ended up going back to a 2015 paper, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” and trying to get a little bit more detail on how the noise process works to create and reverse diffusion [6]. Links and thoughts:Top 5 Tweets of the week:Footnotes:[1] https://github.com/CompVis/latent-diffusion [2] [3] https://arxiv.org/abs/2105.05233 [4] https://github.com/openai/guided-diffusion[5] https://ai.googleblog.com/2021/07/high-fidelity-image-generation-using.html[6] https://arxiv.org/abs/1503.03585 What’s next for The Lindahl Letter?* Week 80: Bayesian optimization (ML syllabus edition 1/8)* Week 81: Deep learning (ML syllabus edition 2/8)* Week 82: ML algorithms (ML syllabus edition 3/8)* Week 83: Neural networks (ML syllabus edition 4/8)* Week 84: Reinforcement learning (ML syllabus edition 5/8)* Week 85: Graph neural networks (ML syllabus edition 6/8)* Week 86: Neuroscience (ML syllabus edition 7/8)* Week 87: Ethics (fairness, bias, privacy) (ML syllabus edition 8/8)I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

This week based on the backlog, I should be covering the topic of Bayesian optimization. During the course of sitting down to write this week something different happened. Apparently, I was a highly misbehaven backlog prompt this morning. Instead of digging into that topic I’m going to spend some time talking about a more pressing philosophical question related to the future of trust and digital images. This missive is more about expressing and recognizing concern than delivering information. Starting in April of 2022, OpenAi shared the DALL-E 2 model which the researchers noted, “DALL·E 2 is a new AI system that can create realistic images and art from a description in natural language” [1]. Strangely enough the word ethics does not really appear on the homepage for the DALL-E 2 system. It was apparently more important to spend time setting up an Instagram account for art created by the model [2]. You can pretty easily go see how photorealistic some of these images are. With this release from OpenAI, I never requested to join the waitlist to kick the tires on this one. I’m generally more interested in natural language processing than visual image processing. Let’s set the stage as clearly as possible on this one. People are used to being able to go to the photo finish. Races have been decided by photos and ultimately video for years. We trust video in replay for sports and it remains the visual record of our times. Things changed. Full stop. It used to require a lot of effort to make a deep fake or to alter photographs. It required software and spending time to accomplish that task. Right now a host of new models and other ML implementations are creating the possibility of asking a prompt for an image and within a few seconds getting a reasonable approximation. Some of these images are really high quality. They are photorealistic renderings. You can call up pictures of people who never existed doing things that never happened. All of this raises really interesting ethical questions about the use and propagation of such ML technology. Keep in mind that these models are just openly being used and served up online. No degree of concern for the potential social impact stopped the distribution and ultimately additional model development. Right now I’m going to say that you cannot trust the future of digital photography. Don’t believe what your eyes report at this point within the digital space. Right now these models very quickly change and make digital images. Soon enough people will develop that technology into a series of images and realistic video will be produced based on a prompt. Essentially they just have to extend the model to the context of a few frames in series and short videos will spring into existence. The evening news could galvanize popular opinion with a story and photograph. At this point, I’m not sure we can trust that type of evidence anymore. Lingering implementations for how civil society is going to change in the face of a zero trust image paradigm. I’m not sure people even understand the alternate realities that could be created and presented as fact. Somebody could bring forward the presentation of a very news forward YouTube channel powered by DALL-E 2 created images. For example, you could introduce a new continent and talk about the discovery of Atlantis and potentially go on for years presenting an alternate reality as truth. Somebody will probably make a living doing that or something equivalent to it. That is where the ethical considerations of this technology and the impacts on society as a whole took a backseat to race to share and demonstrate effectiveness. Take a moment and consider that just because a technology can do a thing does not mean it should be used to do those things. We make choices. You have to have ethical considerations at the forefront of that type of decision making. We are getting to a point where we have a zero trust image paradigm that will effectively make it a necessity to question everything you see in terms of digital photography and ultimately video. That realization and reality will reverberate across interactions in daily life. At this point, based on the evidence we have, I’m going to declare we have to embrace and ultimately enforce a zero trust image paradigm. How do we even label actual historical documents accurately at this point? Historians will have to be very careful going forward in the analysis of the times about to happen. This may very well be a watershed moment about how we evaluate the truth in front of us and how we verify and validate that narrative. My argument here is not intended to be hyperbolic or presented with any sarcasm whatsoever. A very real situation is developing within our ability to trust the visual world being presented to us. We have to consider the possibilities in front of us and begin to evaluate a path forward. I’m assuming that the path forward is zero trust. That should be clear within the argument being prese

One of the things I have seen trending around the internet places I visit is related to quantum machine learning. I went over to Google Trends and took a look at the last 12 months and could see a decent volume of people generating related queries. If you were to categorize interest by state, then the top 8 would look like: New Hampshire, Washington, California, Pennsylvania, New York, Indiana, Texas, and Florida [1]. I’m a little surprised that my research efforts within the last 12 months did not get Colorado on that list of interest by subregion. Apparently, I’m going to need to step up my quantum computing game.After getting a sense of where in the United States people are interested in quantum machine learning I started to consider what topics are slightly related. That effort is intended to help me understand the edges of the topic and be able to look 360 degrees around the idea being evaluated today. The Top 10 related queries would be: * quantum machine learning* open source quantum machine learning* quantum machine learning solutions* quantum machine learning tools* quantum machine learning system* quantum machine learning software* quantum machine learning services* quantum machine learning applications* free quantum machine learning solutions* free quantum machine learning toolsWithin the Google trends frameworks they offer another layer of insight beyond the Top 10 related queries. Let’s take a look at that next layer.The Top 10 rising queries would be: * free cloud based quantum machine learning tools* free cloud based quantum machine learning solutions* free quantum machine learning tools* free quantum machine learning applications* free open source quantum machine learning applications* free quantum machine learning solutions* open source quantum machine learning software* free cloud based quantum machine learning software* free cloud based quantum machine learning services* free open source quantum machine learning toolsTo help drill down to some context of how often quantum machine learning really comes up as a topic I ended up comparing machine learning, artificial intelligence, and quantum machine learning [2]. Searches for artificial intelligence are an order of magnitude bigger than searches for quantum machine learning. Overall the searches for machine learning are much larger than quantum machine learning, but still like 20% of the artificial intelligence related volume. The interest quantified as searches for artificial intelligence have jumped up incredibly high. That is wholesale due to an employee from Google who is an engineer that works with the technology noting publicly a concern that the AI had become sentient [3]. That disclosure has caused a stir online and in the media. I’m not going to cover the nature of what sentient AI describes in this newsletter. It will certainly be a topic that gets covered in more detail after the dust settles on this one. You can check out the first video in the links and thoughts section below for 20 minutes of coverage on this one from the one and only Yannic Kilcher. I’ll share two links to the Medium platform here, but I’m not going to cover them by naming the engineer [4][5]. I think it is entirely possible that the content from those two links will get pulled down at some point in the not so distant future. Unless all of this was just some sort of weird publicity play from Google to raise awareness about LaMDA. Links and thoughts:“Did Google's LaMDA chatbot just become sentient?”From CNET, “Why I’m More Excited About the Next UPS Truck Than the Next Tesla”“The Download: GitHub Achievements, LTT to the Rescue, and Goodbye to an Old Friend”Top 6 Tweets of the week:Footnotes:[1] https://trends.google.com/trends/explore?geo=US&q=%2Fm%2F012c1btp [2] https://trends.google.com/trends/explore?geo=US&q=machine%20learning,%2Fm%2F0mkz,%2Fm%2F012c1btp [3] https://www.npr.org/2022/06/16/1105552435/google-ai-sentient [4] https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 [5] https://cajundiscordian.medium.com/what-is-lamda-and-what-does-it-want-688632134489 What’s next for The Lindahl Letter?* Week 77: Is quantum machine learning gaining momentum?* Week 78: Why is diffusion so popular?* Week 79: What is GPT-NeoX-20B? Bonus topic: What is XGBoost?* Week 80: Deep learning Bonus topic: Bayesian optimization* Week 81: Classic ML algorithmsI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Last week some difficult questions were asked about overcrowding within the field of machine learning and the effect of that on engineering colleges [0]. This week things are getting even deeper into the hard philosophical questions we are starting to face. Working to answer the question, “What is post theory science?” will require a lot of consideration and a good bit of digging around. A lot of the ML things that are bubbling up right now in terms of ethical conundrums and rapidly changing ML delivery use cases are just ahead of the next wave of publications and methodology research articles, manuscripts, and textbook updates. This is another topic that I think is going to end up getting a lot of coverage at some point. Just like the potential effects of overcrowding within the machine learning space limiting other research within engineering colleges. The compounding problem from that is that the overcrowded research happens to be very shallow within the ML space resulting in papers without real and lasting contributions to the field. Let’s focus on the question at hand, “What is post theory science?” This is an interesting scenario to have in existence. Machine learning models can be built to seek out solutions without any theoretical methodology being applied to the potential solution. That type of possibility created a situation where Laura Spinney of The Guardian asked the question, “Are we witnessing the dawn of post-theory science?” [1]. Within that analysis Laura called back to an article from 2008 by Chris Andeson in Wired magazine titled, “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete” [2]. In this scenario without using any theoretical basis to create a testable hypothesis or any classic research method a machine learning use case could derive defendable answers. You could do a similar task with big data or just data in general. Things can be observed and built into a postulate or observation that is not directly based on scientific theory. We can learn things that are seen as being objectively true, but are not derived from theory. In this case that type of effort could very well be called a post theory scientific research methodology. You could build an AI model that aims to understand elements of the universe and it could just be allowed to run and work on that effort. Within that proposition the AI model could test and work with model settings that yield results, but are not based on theory. It could just be using random configurations or working down a path that was independently derived. A search within Google for “post theory science” does not yield a ton of results. At the time this post was written only about 1,740 results existed within the knowledge graph. I’m truly curious when research method books will contain post theory methods. For post theory science to really pick up steam as a method of research in academic institutions it will need to start showing up in research methods textbooks. I have a couple of them on the bookshelf next to me and while they cover mixed methods and a variety of approaches, nothing within that very large book presumes the idea of post theory science. I’m going to guess at some point that will change here in the next couple of years. We are going to see findings and research coming out of the ML space at record levels and a portion of it will be delivering results, conclusions, and data that are not derived from the traditional scientific method. Links and thoughts:I watched this entire video, “This Beat Up, Non-Running Omega Seamaster Has Big Potential! Vintage Watch Restoration” “What unions could mean for Apple with Zoe Schiffer”I’m in and out on Lex’s content, but sometimes I just enjoy listening to a good conversation. “Jonathan Haidt: The Case Against Social Media | Lex Fridman Podcast #291”“The Lab is a Disaster - WAN Show June 3, 2022”Top 5 Tweets of the week:Footnotes:[0] “Is ML destroying engineering colleges?”[1] https://www.theguardian.com/technology/2022/jan/09/are-we-witnessing-the-dawn-of-post-theory-science [2] https://www.wired.com/2008/06/pb-theory/ What’s next for The Lindahl Letter?* Week 77: What is GPT-NeoX-20B?* Week 78: A history of machine learning acquisitions* Week 79: Bayesian optimization* Week 80: Deep learning* Week 81: Classic ML algorithmsI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. Thank you and enjoy the day! This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Welcome to a more investigative journalism based issue of The Lindahl Letter this week. This one really made me think. It's a provocative question for sure. Emotional reactions to the premise of destruction aside the question of overcrowding within the field of ML has really caught my attention this week. One piece of prose stood out on this topic and I’m not the only one to react to it. Published back on January 22, 2022 on LinkedIn of all places was a post titled, “How a False Love for AI/ML is Destroying our Engineering Colleges,” [1]. This post seems like an article in waiting or a research note of some kind that was intended to be widely shared by Smruti Sarangi who is the Usha Hasteer Chair Professor at IIT Delhi. It has caused a lot of discussion for sure with over 500 comments and around 10,000 interactions.Obviously, at this point, I needed to do a bit more research on this one beyond just reading a post shared by a professor on LinkedIn. I ended up looking all over and found a few Tweets that were adjacent to the topic. Yes, I fell deeply into the rabbit hole of reading exchanges on Twitter that were long stale and otherwise left online like the remnants of previous sunrises. Sure they happened and outside of photograph evidence or in this case Tweets everyone has generally moved on to whatever is next on the agenda.One of them was from Yaroslav Bulatov who is on the PyTorch team over at Meta/Facebook. That Tweet included some arguments toward trying to illustrate what was wrong with ML research by putting papers with a theorem expressed or mentioned, but no significant improvement demonstrated. That paper referenced was from Schmidt, Schneider, and Hennig back in 2021 called, “Descending through a Crowded Valley – Benchmarking Deep Learning Optimizers,” [2]. It’s about selecting an optimizer in the deep learning space and it is an interesting read. They are essentially trying to empirically test optimizers and share the results. It’s an interesting thing to go out and work on in terms of answering questions about what optimizers to actually use. Within the conclusion the authors shared a truly interesting observation that, "Perhaps the most important takeaway from our study is hidden in plain sight: the field is in danger of being drowned by noise." I don’t normally include quotes in my weekly research notes, but this one just stood out and needed to be read aloud for the true impact to be appreciated. My interest here really is about a deeper question about if overcrowding within the ML space is causing negative externalities.Generally I avoid doing research by reading Twitter posts, but I thought sharing this thread from Tom Goldstein focusing on a history of AI winters. It's from a National Science Foundation town hall talk, “ML Needs Science,” but distilled on Twitter to be easily consumed. A counter argument was shared by Melanie Mitchell on Twitter as well. Melanie has also written a paper about AI Winters that is easy to read called, “Why AI is Harder Than We Think” [3].One of the things in all of that back and forth that caught my attention was a commencement address from 1974 delivered at the California Institute of Technology by Richard Feynman [4].I could not find enough content to really complete a full post on this topic so not only did I reach out to Gary Marcus on Twitter, but also I sent a note over to Smruti Sarangi on LinkedIn to see if anything additional had been written or published. It turns out that Smruti was just sharing a few thoughts on LinkedIn as a post back on January 22, 2022 and was not preparing any academic papers on the core question of this Substack post. That question was really about, “Is ML destroying engineering colleges?” I ended up wondering about an overcrowding effect within engineering colleges where ML is sucking up all the oxygen and research focus of a generation. That could be reframed into a testable hypothesis by grabbing the last 10 years of publications in some of the top engineering journals to see if publications related to ML are crowding out other academic contributions. I spent a bit of time trying to figure out if somebody had done some academic work on ML creating overcrowding within engineering fields, but I have not found anything directly addressing the elements that caught my attention. My Twitter interactions this week have been interesting and lively. Normally my Tweets and other research links shared during the weeks don’t get such a high level of interaction. This is a topic that people seem to be passionate about and a lot of different points of view exist on this one. Research is certainly happening and will continue to happen. My honest guess here is that all the efforts to publish within the general scope of ML will reach a peak and people will branch off into other things. My take is that veering off into new areas of academic exploration for most researchers will be a healthy thing to happen for the academy after a bit of i

Thank you for tuning in to this audio only podcast presentation. This is week 74 of The Lindahl Letter publication. A new edition arrives every Friday. This week the topic under consideration for The Lindahl Letter is, “ML content automation.”At this point, I started to ask myself if I’m the prompt of an exceedingly large language model. Perhaps a better trained and curated model than any foundational model. Prompt engineering is a wonderfully interesting part of ML content automation in terms of generative models. You have to sort of know how to prime the pump or in this case the prompt to get the right content to start to flow from something like GPT-3 or one of the larger foundational models that are starting to be floated around. Previously I have warned about content flooding and the potential for the entire internet to just be astroturfed with nearly endless content if the wrong Web3 comes into being. Machine learning elements can certainly be used for content generation and extending that functionality to the practice of content automation. You can set up a workflow that just automatically generates content. It could be a bot implementation use case that generates content in response to people. That is an easy method of feeding the model prompts as the only method to interact with a chatbot is to engage in the prompt base activity of sending something from the user to the chatbot. At that point, the prompt has been opened and something is being exchanged to the model which will cause the generation of content. The use case could be extended nearly indefinitely at that point. Let’s go beyond just a chatbot use case and jump into the complexity of “automated journalism” or more generally the use of machine learning or artificial intelligence to generate articles for the purpose of reporting news. This is where I worry about the use case being expanded from news rooms to a general effort of flooding or astroturfing topics. Bad actors could step in and create such a flood of content that figuring out what was real and what was synthetic could become the greatest challenge facing the internet. Truth could be swept away into a totality of coverage that covers all potential prompt lines given that synthetically generated content may have no association with reality whatsoever. My concern related to this path of course started with the advent of GPT-3 and the potential for the synthetic creation of prose that is believable like the article that was published in The Guardian in 2020 [1]. A lot of the content I run into during the course of reading news during the day could very well be synthetically created from a large language or foundational model. We are seeing things like the Microsoft corporation reducing news staff at MSN to replace them with automation [2].We are now starting to see models like DALL-E 2 from places like OpenAI that are able to make realistic images from a prompt [3]. That takes a prose based use case for the creation of content to a much wider range of use cases. I’m sure the model will go from images to videos at some point and that type of model would be a legitimate game changer for content creation. Automating the ability for a machine learning model to create video from a prompt would be a huge advancement within the world of content automation. You could open and stage an art gallery with live prompt based installations. I think it would actually be an interesting use case for OpenAI to demonstrate the potential of the model. Bad actors within this space could also decide to create endless images and videos to flood the online world with content dedicated to a specific topic or general theme. I have spent a lot of time worrying about how to deal with or manage the problematic elements content flooding could create for society in general. The very fabric that binds civil society together might already have seen the breakdown of a curated common thread based on shared experiences. We may have created such a curated content bubble that any shared experience might be limited to commercials and knowledge of products. Being willing to make that assumption might explain a lot of the things happening within society in general at the moment as we face the reality of the intersection of technology and modernity. Links and thoughts:The WAN Show was full of wild Linus stories about home automation today. “Story Time! - WAN Show May 27, 2022”“Stanford Seminar - Leveraging Human Input to Enable Robust AI Systems”“Ask the Experts: Scaling responsible MLOps with Azure Machine Learning | CATE21”Top 5 Tweets of the week:Footnotes:[1] https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gpt-3 [2] https://www.seattletimes.com/business/local-business/microsoft-is-cutting-dozens-of-msn-news-production-workers-and-replacing-them-with-artificial-intelligence/ [3] https://openai.com/dall-e-2/ What’s next for The Lindahl Letter?* Week 75: Is ML destroying engineering colleges?*

Within my staging Google document for Substack posts I reached the end of the originally planned out posts for this series of content. Earlier this morning I expanded the staging shell post outlines to week 104 which as you can imagine is a significant point in the publication lifecycle. 2 years of writing Substack posts will be here before you know it. I have enough content in the backlog for this Substack series to get to week 120. At the two year mark I’m planning on moving away from machine learning posts into just generally covering artificial intelligence and producing research notes related to a planned set of academic articles. That means that it is possible that weeks of ongoing coverage of a topic being worked on as a future academic article could be distributed during year 3 of this Substack series. That is probably a good method to really dig deep into a few topics along the way. One of the things I have worked pretty hard to avoid is producing coverage of the same topic over and over again. One of the things I have noticed in the last few months is that I may have reached conceptual exhaustion within the machine learning topics at around one hundred different concepts. At this point, I should probably go look at all the general conceptual models of the machine learning space and see how close I am to reaching comprehensive coverage. I jumped over into Google Trends and took a look at what topics are bubbling to the surface [0].That very meta aside about the future of The Lindahl Letter being complete; let’s jump into the topic at hand for today. Most of the time you will see people calling out Symbolic AI vs. symbolic machine learning. If you are interested in trying to build an artificial intelligence system that works similar to the human brain in terms of learning, then you are going to run into the idea of Symbolic AI. Think of things like deep learning, Bayesian networks, or evolutionary algorithms. What I was curious about this week was how many times people try to evaluate symbolic machine learning as a concept. Explicitly searching on Google Scholar for “symbolic machine learning” will yield just over 2,000 results [1]. Some of the academic coverage on this topic goes back to the 1990’s which was obviously where my reading started. Typically I try to rewind back to where articles were sparse and the content was more focused. Recently the volume of content has exploded, but a good portion of it is derivative. I ended up reading an article from Harries and Horn called, “Detecting Concept Drift in Financial Time Series Prediction using Symbolic Machine Learning,” that was published back in 1995 [2]. Sometimes I just enjoy reading about forecasting related concepts as it is grounded in a field of study that has always just made sense to me. Within that space of consideration a copy of Armstrong’s Principles of Forecasting (2001) is sitting on my bookshelf just a couple of feet away. I don’t plan on letting go of that weighty tome any time soon. Oddly enough this article seemed to focus on the potential promise of symbolic machine learning in the future. The phrase only occurs 4 times in the article and that includes the title and abstract. I’m wondering if maybe it was added after the article was written. I was reading a few academic articles and wondering what exactly people are doing within the practical applications of symbolic machine learning. Google Scholar indicates that 6 related searches stand out. Those searches include remote sensing, algorithms, neural networks, classifiers, reactive control systems, and European settlement maps. Obviously, I was super curious what symbolic machine learning had to do with settlement maps. I found a letter from IEEE Xplore called, “Application of the Symbolic Machine Learning to Copernicus VHR Imagery: The European Settlement Map,” [3]. It’s pretty much exactly what you might think it would be about in terms of a literal mapping of settlements. The scale of the data being processed on this one seems really interesting. This letter did mention symbolic machine learning within the body of the work related to model innovations. It’s a pretty dense publication in terms of concepts being blended together without a lot of explanation. That is probably a byproduct of the authors trying to keep this to 5 pages vs. around 20 pages where that additional commentary would be flushed out.Links and thoughts:“Computex 2022 laptops 💻 Elon vs Twitter bots 🤖 Apple ‘testing’ foldable E Ink display 📱”“See Where Your Electric Car’s Battery Will Go One Day”“My Investment Pays Off - WAN Show May 20, 2022”Top 5 Tweets of the week:Footnotes:[0] https://trends.google.com/trends/explore?q=machine%20learning [1] https://scholar.google.com/scholar?q=%22symbolic+machine+learning%22&hl=en&as_sdt=0&as_vis=1&oi=scholart [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.51.5260&rep=rep1&type=pdf [3] https://ieeexplore.ieee.org/abstract/document/8941071 What’s next f

Security is the element of open source software that has to always be considered. Depending on the size of the associated developer community participating and the rate of development the number of security vulnerabilities is going to rise and fall. It will be a constant battle between those people trying to take advantage of vulnerabilities and the people who fight the good fight of software security. I have taken a serious look at this topic before. Realities of risk associated with software security are a problem for both open source and proprietary software. Arguments have been made that bringing together more and more people who are using a piece of software via the open source model will create a scenario where risk is reduced via transparency and contribution from a multitude of sources.Back in January Kent Walker who is the President of Global Affairs for Google shared a blog post about, “Making Open Source software safer and more secure” [1]. That missive talked about log4j, a recent open source vulnerability that was impactful to a variety of industries. A reminder was included about a $100 million dollar donation to the Open Source Security Foundation [2]. Kent reduced the question down to figuring out the critical projects instead of trying to boil the ocean, being clear about security testing baselines, and figuring out methods for increased support from both public and private sources. Let’s zoom out for a second and look at public policy and regulation related to open source software security. Back on May 12, 2021 Executive Order 14028 was issued about “Improving the Nation’s Cybersecurity” [3]. The whole order is 15 pages long and may take you about 20 minutes to read. You can pivot from that to the update from May 11, 2022 to the National Institute of Standards and Technology (NIST) guidance on “Software Security in Supply Chains” [4]. That collection of online pages will take you a lot longer to read. It has a pretty high density of content. A lot of supply chains are now using machine learning and a mix of open source software elements to make things work along the path from production to delivery. As you can imagine a lot of policy makers are legitimately concerned about risks to supply chains. Now that we have considered concerns related to the developers, companies, and governments looking at open source software security you can see the scope of risk involved. I’m not sure I see any easy solutions on the horizon for this one. It is going to be something that has to be mitigated in real time and a lot of people are going to have to work together to make that happen on an ongoing basis. Does this week include some bonus edition content? Yes, it does. We are about to cover a bonus topic related to, “Machine learning and surveillance.”Welcome to the bonus topic this week. My backlog of topics has grown a bit out of control. This is week 72 for example and the backlog has 120 topics. Moving forward I’m going to grab a few of the topics and work on making a few double issues of The Lindahl Letter. Making sense of and working with mind boggling amounts of data is something that machine learning can help with based on anomaly detection and computer vision elements. You can quickly work through hours of security video footage from cameras at a building and only work with the footage where motion or some type of change occurs. In terms of overnight security and monitoring this means that a large portion of the effort can be almost immediately cleared away. No review is required. You can then move from anomaly detection to the more complicated elements of computer vision to tag elements in the video and flag things for manual review or intervention by alarming or notification. I jumped into a quick Google Scholar search for all of the academic papers that might include or be related to, “computer vision machine learning surveillance” [5]. This is an area where you can find some really solid and well understood use cases. Back during week 37 coverage one of the links referenced out to the CLIP technology from OpenAI [6]. You can grab an implementation of that from Johan Modin over on GitHub that will help you do contrastive language to image searches [7]. When you see people in movies just searching hours and hours of video for the needle in the haystack and coming back with a quick response of all the examples of “The Man with One Red Shoe” it would be based on a technology like this making that magic happen. If you have not seen the 1985 Tom Hanks comedy thriller by the same name, then you might be missing out on the rich comedic depth of that reference. With the right amount of investment and computing power you can do amazing things in the surveillance space with machine learning. Some of them are shockingly advanced compared to where we were before.The part of this topic that I really want to cover, but is again a deeper topic for conversation involves the various methods people stitch data

This week we are going to dig into content about ML newsletters. I’ll do my best to bring that content to life via the spoken word, but you may want to check out the actual written text this week if you are looking for hyperlinks to the actual newsletter locations. Generally, if you want to live dangerously, then you can find these newsletters with a quick Google search or you can visit this post for a more direct approach.Probably the newsletter that I have subscribed to the longest would be the Inside AI newsletter that has featured weekend commentary from Rob May [1]. Recently the entire Inside newsletter ecosystem has changed it up a little bit with a new platform where you can login and interact with posts. The Inside newsletter ecosystem includes tons of different topics thanks to the founder and CEO Jason Calacanis mixing things up. Way back on July 22, 2018 I did get to write a weekend commentary for that newsletter. I’ll leave that missive below at the end of this post.A lot of smaller newsletters exist and a few of them are wrapped around brands or larger institutions. Let’s talk about a few Substack newsletters: Last Week in AI which provides a weekly free edition or you can subscribe for more issues [2]. Subscribing to that Substack newsletter will suggest you subscribe to 3 other newsletters. I always think it is interesting to see what other people are suggesting in terms of subscriptions. 1. The AI Ethics Brief: Democratizing AI ethics literacy [3]2. The Gradient: Overviews on cutting-edge AI research and perspectives on the direction of the field [4]3. AI Weirdness: The weirdness of artificial intelligence [5]. Interestingly enough when I subscribed to the Substack of this one I got an email from the author advising me the newsletter had moved to Ghost and I should shift my subscription. I have read a lot of content from The Next Web (TNW) including the Neural newsletter [6]. You can find newsletters from O’Reilly and the MIT Technology review and a slew of other businesses that share AI/ML related content primarily associated with their interest. Over the years I have gotten subscribed to a ton of these and some of them I’m not entirely clear how those subscriptions occurred. A lot of them came about around the time I was working on my AIOps/MLOps research which makes sense. Here is my weekend commentary post from back in 2018, “Everyday AI: From open source tools with a growing library of free training to being accessible in the business world.”Sitting down to write a commentary on AI that will match Rob’s high standards made me pause and think deeply. These are amazing times and anybody that is actively seeking to take training on artificial intelligence or use the multitude of tools and languages being deployed within the open source community can open the door to truly interesting possibilities. That is the part of all of this that really caught my attention.Translating the newly accessible tools and training into action has been the big challenge facing practitioners of artificial intelligence, machine learning, and deep learning. People throughout business are trying to gain insights from data to make decisions. Some of those efforts help build out highly complex dashboards and compendiums of KPIs that help provide insights and drive decisions. Some of the AI use cases are more plumbing related vs. being a scoreboard by replacing a step in a workflow that drives productivity or introducing something new that brings tremendous value to a product. Between informing strategic decisions and driving value, people are hearing about AI and they want to use it in the workplace. They want to use it to be successful. Nobody wants to miss out on this wave of opportunity to describe how things are happening and make predictions about the future. A new host of tools and techniques related to deep learning exist to really perform complex analysis. The increase in data availability for training models has also risen exponentially. Within our everyday lives models have been deployed to improve the pictures we take and handle other tasks at the edge. Very few of those new tools have started to really be deployed within the day to day workflow of business. That is where I see the greatest opportunity for practitioners. My gut tells me that 2019 will be the year that AI becomes accessible to common workplace reporting, tasking, and IT deployments. During the demo of Google’s Duplex at Google IO a technology was delivered that will change more than a few games. Both call centers and personal digital assistants come to mind. Both of those advancements will help push engagements with consultant companies and business partners delivering new technology implementations. Just about everybody that has called a major corporation over the years has felt the frustration with poorly setup phone systems. Some of them are interactive and some of them were just a mix of old technologies that made it hard to reach cus

A lot of enthusiasm and money are flowing into cryptocurrency these days. That wave of 25 billion in venture backed financing started back in 2021 [0]. Most of that technology is based on blockchains. The co-founder of Ethereum Gavin Wood as far back as 2014 started talking about a blockchain technology being used to fundamentally change the internet [1]. Gavin is a big proponent of fixing the modern internet and very proud of the idea of coining the term “Web 3.0” certainly before the term became a part of the general lexicon of technology terms. Certainly fixing the modern internet is a noble quest to undertake. At this point, the fundamental technology allowing a free and open internet still exists, but we are seeing islands and speed traps popping up at a much larger pace. It is still an open question about if the metaverse will be pay to play or free and open with financial consequences after you enter the front door. At this point, it might be a good time to talk about what each of the different versions of the webs really mean and why that is important. You can learn about it from various cryptocurrency forms including Etherum [2]. They are very loud and eager to sell you on a vision of the future. A lot of people are betting venture capital on this version of the future. One of the writers over at Geeks for Geeks broke it down in a pretty decent way back on January 27, 2022 [3].We need to go back to the 1990’s and consider how Web1 came to be and was used and consumed by a lot of people. People started to stand up servers and have web pages with rather static content. In a lot of ways it was the billboards of the internet built for the online highway. It was amazing and this method of communication was based on servers with content being delivered in a mostly one way direction. Some people called this the mostly “read-only web” [4]. That description was and still remains a pretty solid way of describing Web1.Fast forward to 2000’s and you start to see Web2 begin to emerge. This is where the “read-only web” shifted to a way more collaborative web. You can think back to examples like Flickr for photo sharing, digg which mostly gave way to Reddit for links and things, Twitter, Facebook, and MySpace. I had to drop a MySpace reference into this post for those of you who miss your first online friend Tom. This is where network platforms started to gain steam and walled ecosystems began to flourish from collecting data and selling advertisements. Sometimes people call this the surveillance web and reference the immense infrastructure deployed to track people online using a combination of cookies and stitching data together. That is a topic that could end up being a future post. Nothing really stops me from weaving in bonus topics for future editions.While Gavin Wood might be very vocal about coining the term Web3 you can rewind back to San Francisco, California on November 12, 2006 and see that term shared by John Markoff in the New York Times [5]. That article however does not talk about or have any context for blockchain or Bitcoin. Given those considerations I can see how Gavin makes the argument to have coined the version of Web3 that Etherum or other blockchain based technologies would power. Leave it to John Bogna of PC Magazine to dig into “What Is Web3 and How Will it Work?” [6]. It basically comes down to having a decentralized digital infrastructure where your personal information is not the foundation of the interaction. I would describe it as a version of the internet powered in some way by a blockchain based technology. I’m very curious to see an implementation working in practice. You can go out to the Web3 foundation and learn about what they are doing [7]. On their about page it clearly talks about the Web 3.0 technology stack. Entire communities have developed to track and learn about Web3 jobs including one that tracks the job counts at the top 100 Web3 companies [8]. The top five with several hundred jobs each are Binance, Crypto.com, Coinbase, Ripple, and Consensys. I scrolled past the rest of the list and some of them were names I know something about, but the vast majority of them don’t have any meaning to me. One day they might occupy space in the public mind. It is also possible that most of them will never have the name recognition of Google, Facebook, Microsoft, or Apple. Before we wrap up this post I will acknowledge that machine learning and artificial intelligence are mentioned all over the Web3 landscape in a variety of ways. However, primarily the technology core of how the stacks are being built seems to be around some type of blockchain based method to be decentralized. Generally speaking both machine learning and artificial intelligence are centralizing technologies. Certainly distributed machine learning model delivery or decentralized artificial intelligence would be interesting topics to consider, but neither of those things are commonly in the lexicon of discussion cur

Don’t panic! We are going to take this topic in two different directions this week. First, the obvious path will be taken and we will talk about programming, code, and python cookbooks. Second, for those of you that were hoping this week will be a deep dive into artificial intelligence and machine learning derived recipes compiled into a cookbook you will not be disappointed either. Another way to talk about what we are covering today would be that not only are we talking python cookbooks, but also we are covering the epicurean adventures that could be created. Beginning with the first topic under consideration; coding related answers to the machine learning cookbook question are real. What exactly is a Python Cookbook? If you are a fan of the work by David Beazley and the 3rd edition of the Python Cookbook from the publisher O’Reilly, then you might already have an idea [1]. I personally did not have a copy of this book on my shelf or any of the other python cookbook compilations. These are compilations of code brought together like recipes to help people learn how to use and deploy Python as a computer programming language. You can get to all the code examples that David Beazley has shared on GitHub [2]. For those of you looking to dig into some Python code and learn about how it works, a collection of recipes compiled into a cookbook is a decent place to start. You might get lucky enough to find some recipes on the specific thing you are trying to solve. The other book that stood out to me was the Modern Python Cookbook from Steven F. Lott [3]. The code from that book is also published on GitHub for easy access [4].At this point we are pivoting to the second question about actual recipes. You might remember back to 2017 when Eric Schmidt shared an AI created cookie recipe built out by engineers at Google as part of a real world challenge [5]. You can find the recipe for the “chocolate chip and cardamom cookie” via the link above. Getting to that point apparently took 59 recipe iterations which is interesting. Oddly enough that article also brought in some of the warning elements that Eric Schmidt has shared about the potential misuse of AI. Cookie related AI examples have improved since 2017 and you could check out the Nestle Toll House cookie expert or alternatively the “cookie coach” AI now to help answer your baking questions [6]. It got announced on Twitter and they had a fun video about working with Ruth the cookie coach. It does appear that Ruth the Nestle Toll House cookie coach really is still online and working over at https://cookiecoach.tollhouse.com. From Eric Schimdit sharing recipes to interactive chat experiences people have really opened the door to recipe driving artificial intelligence when it comes to cookies. At this point it might be a good idea to move beyond cookies and look at recipes in general. You could go out to the Google Cloud blog and learn about “Baking recipes made by AI” from December 15, 2020 [7]. The embedded video digs into making recipes with ML and is a 7 minute journey of fun featuring Sara Robinson. You may already know that during the pandemic Sara Robinson collected a ton of recipes and used a TensorFlow model to make predictions on future iterations [8]. Between the code shared on the Google Cloud blog and by Sara Robinson you can actually work on a similar cookie effort to make your own machine learning model to work on recipes. Our friends at Google are not the only ones that have started to wonder about using machine learning for the practical activity of creating recipes. The team over at Towards Data Science have walked through the good and the bad of the process as well within an article titled, “Using machine learning to generate recipes that actually work” [9]. This article really walks you through the process and it is interesting. People really do seem to be having fun with using ML to create recipes. Another way to take a look at the process would be with an article from KDNuggets called, “Generating cooking recipes using TensorFlow and LSTM Recurrent Neural Network: A step-by-step guide” [10]. As you can tell from the number of sources of examples on how to start making your own recipes with machine learning it is a topic that people really seem to be passionate about. Some of that passion is translating further into action. Consider the article, “Forage: Optimizing Food Use With Machine Learning Generated Recipes” [11]. Within that article Angelica, Elbert, and Brian take a real look at reducing potential food waste with machine learning. It is an interesting way to apply machine learning to a practical real world problem. My efforts to find a real cookbook for sale that is made up exclusively of AI/ML made recipes were not successful this week. You can find websites like https://cookbook.ai/ that help you search recipes and build meal plans, but I did not find a cookbook you could buy from a bookstore. Links and thoughts: “Sparse Expert Models (Sw

Both OpenAI and Hugging Face have teams doing great things with respect to machine learning models. Their delivery models are very different. You can visit OpenAI at https://openai.com/api/ and look around at the machine learning models being sold to consumers like IBM, Salesforce, Intel, and Cisco. Delivering machine learning models via an API is one way to go about publishing and sharing your work. Alternatively, people are publishing models like what is happening over at Hugging Face https://huggingface.co/models where if I’m reading the page correctly you can search for various models from a list of 36,028. Both of these organizations are delivering excellent machine learning content to a world of people looking to operationalize machine learning within their corporate strategies. Deciding to publish a model or to sell it via an API is a major decision to make. Selfishly, I much prefer the open source models that I can play with and download. Before we move on to the rest of this analysis please consider my full disclosure that I participated in the OpenAI private beta for both Codex and GPT-3. Both of those sets of beta analysis provided API access and not full downloads of the models in question. That participation may have given me a good idea of how the system works and let me kick the tires, but it did not cloud my judgment or make me want to give OpenAI favorable treatment. I do agree with the original assessment by the OpenAI team that the GTP-3 and GTP-2 models open the door to misuse [1]. Back in 2019 The Verge team noted that, “OpenAI has published the text-generating AI it said was too dangerous to share,” [2]. We face a very real possibility that the models could be misused to flood our information streams and that it would become almost impossible for communication to function. Some people already believe that bots and other flooding techniques to AstroTurf and falsely drive news cycles are already breaking a problematic news ecosystem. A truly asymmetric delivery problem exists when the amount of content being produced is massively larger than what can be consumed by an individual. Traditional media has transformed from the highly curated view newspapers both national and local mixed with nightly news broadcasts provided to near real time broadcasting. The level of curation within a 24 hour broadcasting channel is even fundamentally different from the single serving real time publishing cycle that happens online. While the topic of information flooding deserves an entire post of consideration especially related to the mechanics of how it works I’m going to move on to the ethics part of the question.Ethicists have been debating the potential release of dangerous machine learning models for some time [3]. It is a serious debate that needs to be had probably at a governmental and ultimately international consensus level given the potential influences on civil society as a whole from a dangerous intersection of technology and modernity. You can easily provide a model like GPT-3 a prompt for a topic and it will very quickly spit out content. If you elected to do that over and over again for a nefarious purpose, then you could flood comments, posts, news, and other points of information. It is a truly great tragedy of the public information commons that takes the power of sharing information online and tips it to an extreme. Outside of the ethical considerations of these large language or foundational models. We are probably going to see the heavily used and curated machine learning models deployed via the API method of selling and providing accessibility. Reducing the friction to be able to access and use an API which is generally going to be curated by an organization that is handling all the maintenance and training has a certain value proposition going forward. You almost get to set it and forget about the ongoing cost of training, enhancing, and maintaining the machine learning model. Your machine learning return on investment model may very well allow for some additional cost per transaction within an externally sold API to get the benefits of speed to access and ongoing scalability. That creates an advantage for the biggest companies that can provide proven uptime and reliable service. My attention turned to looking at Google Scholar for “machine learning API marketplace” to see what publications surfaced [4]. A lot of the articles felt like pitches or introductions to specific technology. They were describing parts of the landscape, but were missing the bigger picture of what would happen in the overall marketplace.Links and thoughts:1. I watched Linus and Luke during the WAN show episode from April 8, 2022. They were super excited about launching a screwdriver, backpack, and maybe getting a second giant product testing studio. It's good that Linus is getting into the product testing part of the review space as that is an area where we need more focus and professional attention for the p

Things got shuffled around a little bit and instead of hearing about, “Who still does ML tooling by hand?,” this week my attention shifted to writing down my thoughts on NFTs. This happened in part due to a lengthy conversation I had about the future of NFTs this week. Part of that conversation was distilled into this series of thoughts on the subject. This was almost an interview edition of the podcast. Given the length of my current topic backlog the topic being replaced may not surface again or will have to be incorporated into something else along the way as a sort of bonus topic. I’m on the lookout for the first mainstream use case for NFTs. To qualify that statement I’m going to describe mainstream as millions of users doing something (that something is the basis of what I would consider a use case) on a regular basis with NFTs facilitated by a marketplace. For those of you who wanted a quick primer on NFTs, then please consider watching the 8 minute video by The Verge team called, “NFTs and the $13B marketplace, explained,” from February 2, 2022 or reading the “NFTs, explained” post they have [1].Earlier this month Mark Zuckerberg said that NFTs are coming to Instagram [2]. We are already seeing NFTs being added to Twitter profile pictures in a special way [3]. Both of these use cases for NFTs create a use case for the technology or the foundation for where a marketplace could exist with a sizable audience. That is the key element I have been waiting to see within the NFT space. People can 100% get non fungible tokens associated with a blockchain and keep that token in their cryptocurrency wallet. This whole process requires a bunch of things to work which is why I think having some of this background stuff built into an application like Twitter or Instagram will help create a more mainstream experience. What exactly do you need to make an NFT work?* Blockchain - You need a functioning blockchain that can be accessed to view and confirm ownership of the token. This is a really important thing as if the blockchain ceases to exist or becomes so slow from lack of operational activity your token could become worthless. It is a digital asset and is inherently ephemeral.* Marketplace - The person needs a marketplace or some method of transacting the exchange of an NFT. This is where I think the mainstream social media applications getting involved will make this easier for everybody.* Wallet - Somebody has to have a cryptocurrency wallet to be able to store the information related to the token and other elements for you. This is a collection of keys that are necessary to prove ownership of your token or cryptocurrency. * Minting - Within the process one of the parties has to be able to generate NFts and bring them to a marketplace or distribute them as gifts. I’m assuming that Instagram will give people the ability to make NFTs from pictures they upload. Right now as it stands somebody who is reasonably tech savvy could go out and buy, own, and trade NFTs right now. I don't believe that all of the blockchains being used for these NFTs will survive. I think a bunch of people are going to be disappointed when this happens. However, a bunch of people will probably have forgotten they have had the NFTs on that blockchain to begin with so they won’t be disappointed. Sure people are working on some interoperability between blockchains so you can move NFTs if you think your current blockchain might be on the path to ruin. All of that complexity will fade away when people are just using the ecosystem related to Instagram or Twitter to make the process frictionless. A process for NFTs that includes millions of people within the marketplace and zero barrier to entry in terms of technology will kickstart an actual economy around the content. It is also a lot less likely that Instagram will fade away. People will keep sharing photos in some form now and in the metaverse. Soon enough I’ll be focused on building out 8 sections of that machine learning syllabus. My plan for that effort is to really build out and potentially revise that syllabus into something really good. I’m talking about taking it from academic articles all the way to easy to consume lectures from other academics. My goal with that effort is to pull together the very best content from all over the place into a one stop shop for somebody to come up to speed on the subject of machine learning. I’ll put each one of them into a Jupyter notebook format and publish them on GitHub as well to allow people to do push and pull requests against the content. That seems like the most reasonable way to begin the sharing process in an open and earnest way. Links and thoughts:1. You can watch this video about “NFTs and the $13B marketplace, explained” from the team over at The Verge. I enjoyed it and thought it was a good introduction. 2. This episode of the Decoder podcast really digs into the possibility of what is going to happen with NFTs and crypto. Nilay Patel r

Let’s start with a brief aside. You may have noticed the podcast audio from last week’s episode was a little different. I accidentally pressed the pattern button on the back of my Blue Yeti X microphone and moved from stereo mode to omni mode. The change did not take away from the listenability of the overall podcast episode, but it does change the temperature or color of the overall audio recording feel. Personally, I strongly prefer the results of the stereo mode that have a slight natural reverb and increased depth. You may recall that I tried the cardioid mode as well earlier as it is recommended by Blue microphones for podcasts, but accidental button presses aside I’m going to stick with the stereo mode for recording going forward. I’ll take a moment now to kindly remember the iconic words of science fiction writer Douglas Adams, “Buttons aren't toys!” That is good advice to remember in the future when moving the Yeti X around my desk. We can now return to the question at hand related to the title of this essay, “Does a digital divide in machine learning exist?” Yes. A digital divide exists. A world of online content exists, but it has a certain barrier to entry or access. You need a smartphone, computer, or tablet with access to the internet to participate with and access the digital world. It is distinct and separate. To that end a divide exists. It really is a digital divide between technology usage and a normative set of functions distinctly separate. Within the machine learning landscape I would argue that a digital divide exists and we could probably categorize it in multiple ways. First, you have the layer of digital divide that exists between those who have access to technology and elect to use it. Beyond that first layer you have to consider the complexity of machine learning models and the underlying data. As a second layer, you probably have to consider that the digital divide creates an inherent bias where under representation or even complete exclusion exists inside the data being used to train and implement machine learning models. That is probably a structural data inequality that is not easily corrected to remove bias related to a lack of inclusion. I did read an article titled, “Exploring the Intersection of the Digital Divide and Artificial Intelligence: A Hermeneutic Literature Review,” from 2020 [1]. The paper is free to download and does look at a lot of literature. You can generally jump to “Appendix A: Digital Divide Research” to see some extra content about the digital divide if that is an area you are interested in better understanding. They had a focus on visible and invisible AI that I found interesting. A focus on AI that is visible to a user and elements that would not be visible. A lot of the machine learning models at work today are not visible to the users they are impacting. The metaverse is not visible to most people. For most of us the metaverse is an example where a very real digital divide will exist within two distinct worlds of interaction. I don’t participate in the metaverse. A more pressing example than the metaverse would be access to care between those that can utilize digital access as a vector and people who are unable to use technology for scheduling. Within that frame of reference, I read an article from Anita Ramsetty and Cristin Adams about the “Impact of the digital divide in the age of COVID-19” from the Journal of the American Medical Informatics Association [2]. It was only 2 pages long, but it was directly looking at the topic I wanted to read about. The authors very carefully argued about how underserved communities could exist from a healthcare perspective based on a digital divide. Machine learning itself also creates problems based on the efforts required to make it work [3]. A lot of gig workers help train and work with the data necessary to make machine learning work. The article listed above from the MIT Technology Review really was focused on the content the title indicated, “AI needs to face up to its invisible-worker problem.” Within the article a NeurIPS talk was referenced by Saiph Savage [4]. That talk is over an hour long and will make you really think about how the largest datasets got labeled and who did that work. It really is something to consider and understand about how foundations are built within the largest language models. Links and thoughts:1. I listened to a New York Times podcast from Kara Swisher this week called Sway that covered how Elon Musk might shape the future of Twitter. Casey Newton showed up and they dug into the potential changes at Twitter now that Elon Musk is the largest single shareholder. 2. This week you are getting a second podcast link. This one was titled “Is streaming just becoming cable again? Julia Alexander thinks so” from the Decoder podcast with Nilay Patel. It was an interesting conversation with two people who have obviously spent a lot of time talking. Top 5 Tweets of the week:Footnotes:[1] C

Ethics should be a part of every machine learning course. It has to be a part of every machine learning journey. Perhaps the best way to sum it up as an imperative would be to say, “Just because you can do a thing does not mean you should.” Machine learning opens the door to some incredibly advanced possibilities for drug discovery, medical image screening, or just spam detection to protect your inbox. The choices people make with machine learning use cases is where the technology and ethics have to be aligned. Full stop. That is the point I’m trying to make today and this essay could stop right here. I’m going to carry on anyway to celebrate the point as I consider it to be vitally important. No one really solid essay or set of essays on AI/ML ethics jumped out and caught my attention this week during my search. Part of my search involved digging into results from Google Scholar that yielded a ton of different options to read about “ethics in machine learning” [1]. A lot of those articles were about how to introduce ethics to machine learning courses and about the need to consider ethics when building machine learning implementations. Given that those two calls to action are the first things that come up and they are certainly adjacent to the primary machine learning content being shared it might make you take a moment to pause and consider how much the field of machine learning should deeply consider the idea that just because it can do something does not mean you should. Some use cases are pretty basic and the ethics of what is happening is fairly settled. Other use cases walk right up to the edge of what is reasonable in terms of fairness and equity.An open access article from Nature did catch my attention by Samuele Lo Piano called, “Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward” [2]. That 7 page article has almost 2 pages of references which was pretty intense as citation to content ratios go in published articles. Within my search I was looking for a foundational article or essay that is commonly referenced. I never really did find one. I ended up moving on to an industry driven essay from the team over at Toward Data Science about, “Ethics in machine learning” [3]. That essay did scale back to the basics of the question at hand in terms of how ethical considerations are applied to building machine learning models.I wanted to refocus my efforts on the macro considerations related to ethics in machine learning at this point. I remembered that Rob May shared a weekend commentary as a part of the Inside AI newsletter recently about the dark side of reducing friction in taking action with advanced technology [4]. Rob even went as far as sharing an article from one of my favorite technology related sources “The Verge” about just how easy and low friction it was to use machine learning to suggest new chemical weapon builds [5]. That is a very real example of where reducing friction to doing a thing opens the door to very problematic actions that illustrate the need for a foundational set of ethics.If my call to action and introduction of an imperative to the machine learning ethics space were not enough to compel you to ground your efforts, then please consider a hand curated selection of three videos to assist you in your journey. Maybe one of them will catch your attention and help spread the word about ethics in machine learning.1. In under 2 minutes Dan Frey who is a professor of mechanical engineering at the Massachusetts Institute of Technology (MIT) introduces this video, “Exploring fairness in machine learning.” This pitch goes back to the Comprehensive Initiative on Technology Evaluation (CITE) from the MIT D-Lab which was launched in 2012 and provides a framework you can use for evaluation [6].2. If you were looking for something less tactical and more discussion oriented, then please consider this much longer 90 minute video from the New York University (NYU) Stern School of Business, Fubon Center for AI, Business Speaker Series, titled, “Machine Learning, Ethics, and fairness.” The video is from back on Monday, April 15, 2019 by Dr. Solon Barocas of Cornell University and Professor Foster Provost who is director of the Fubon data analytics and AI intuitive and it really digs into the question of ethics in machine learning.3. Finally the third curated selection for you is a shift to a 6 minute video from an industry leader. The IBM Technology and the IBM Cloud group shared a video with Phaedra Biondiris whose title in the video is noted as, “Trustworthy AI Leader: IBM Global Business Services.” This video is more grounded and is probably a good place to wrap up this essay.Links and thoughts:https://www.youtube.com/playlist?list=PLZHnYvH1qtOYXzWxVdIU1ZDpbLvxbZdyQ Top 5 Tweets of the week:Footnotes:[1] https://scholar.google.com/scholar?q=ethics+in+machine+learning&hl=en&as_sdt=0&as_vis=1&oi=scholart [2] Lo Piano, S. Ethical prin

Maybe revisiting large language models should have been saved for a few weeks from now, but we are going to begin that journey into the foundations of machine learning anyway. My opening question within this chautauqua should be about how large language models in the machine learning space will change society. To that end it might be good to read a post from the Stanford University HAI or Human Centered Artificial Intelligence Institute, “How Large Language Models Will Transform Science, Society, and AI,” by Alex Tamkin and Deep Ganguli [1]. That institute has a mission of, “Advancing AI research, education, policy, and practice to improve the human condition.” While that sounds like an interesting mission statement to attempt to fulfill, it probably ignores the darker possibilities of what could happen. I went out and read the 8 page paper from the post Alex and Deep that they shared, “Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models” [2]. Understanding how large language models could impact the economy and potential misuses are considered in that research which made it a very solid place to start my analysis for this week.Some really large blocks of content for machine learning exist online. The amount of written work being shared related to machine learning is exponentially growing. It is seriously out of control and beyond what anybody can really reasonably track anymore. One of those blocks of content that caught my attention this week was the ML Compendium by Dr. Ori Cohen [3]. First, this pretty deep work made me wonder about how GitBook works and what other content might be on that platform. Second, it made me wonder about how interactive delivery formats might change the future of textbooks in college settings. A quick search for language models in that collection of links and other content took me to a section on “attention” that included BERT, GPT-2, and GPT-3 [4]. It was not really what I was looking to read about this week and my attention quickly turned elsewhere. What I was expecting to dig into was the paper on foundational models from a bunch of Stanford University related contributors noted as, “On the Opportunities and Risks of Foundation Models: A new publication from the Center for Research on Foundation Models (CRFM) at Stanford University, with contributions by Shelby Grossman and others from the Stanford Internet Observatory” [5]. You can get the full 212 paper over on ArXiv [6]. By this time in our journey together you have downloaded that paper a couple of times. Yannic theorized that the paper will end up being a key referenced work due to the number of contributors and the volume of things covered. I can see it becoming a part of curriculums for years to come as it has so much reference material in one place and it is free to download.I’m going to backtrack for a minute here and let you know that after a bit of review it appears that GitBook was designed to provide living documentation [7]. Teams use it to maintain and share technical documentation for software and APIs. It appears that it is also used for some projects like the one shared above. I really do think that type of content curation is probably the future of academic publishing for coursework. Really large static textbooks will be replaced by interactive content that could survive in the metaverse. Students' expectations for the delivery of content to them will fundamentally change in the next 10 years and courses that demand a rigid reading of chapter by chapter in a textbook will fall out of favor. Links and thoughts:Top 6 Tweets of the week:Footnotes:[1] https://hai.stanford.edu/news/how-large-language-models-will-transform-science-society-and-ai[2] https://arxiv.org/abs/2102.02503 [3] https://mlcompendium.gitbook.io/machine-and-deep-learning-compendium/ [4] https://mlcompendium.gitbook.io/machine-and-deep-learning-compendium/deep-learning/deep-neural-nets#attention [5] https://fsi.stanford.edu/publication/opportunities-and-risks-foundation-models [6] https://arxiv.org/pdf/2108.07258.pdf [7] https://docs.gitbook.com/ What’s next for The Lindahl Letter?* Week 65: Ethics in machine learning* Week 66: Does a digital divide in machine learning exist?* Week 67: Who still does ML tooling by hand?* Week 68: Publishing a model or selling the API?* Week 69: A machine learning cookbook?I’ll try to keep the what’s next list for The Lindahl Letter forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. Thank you and enjoy the week ahead. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Thank you for tuning in to this audio only podcast presentation. This is week 63 of The Lindahl Letter publication. A new edition arrives every Friday. This week the machine learning or artificial intelligence related topic under consideration is, “Sentiment and consensus analysis.”My coding experience probably most closely aligns to this topic. Crawlers, bots, and other automation use sentiment analysis. A lot of my original automated coding efforts were related to trying to understand sentiment analysis. A lot of people built web crawling software that ran on some pretty tight schedules to collect a set of target pages. That content was then scraped to find companies trading on public stock exchanges. Those company names and more importantly stock symbols would be evaluated for settiment. Initially that analysis was brute force or you could say explicitly hardcoded values to see how many words near the stock symbol or name were positive or negative. You can get lists of words like a dictionary with sentiment scoring that make that relatively easy to accomplish. You probably can imagine that information was used to try to figure out if a stock was going to go up or down. I personally only mapped out the system for paper trading, but it was a very interesting technology to build. You can go out to Google Scholar and find a ton of articles to read with a search for, “machine learning sentiment analysis,” [1]. You will see a ton of natural language processing and machine learning topics that intersect with the phrase sentiment analysis. Understanding the sentiment of a block of prose is something that is highly desirable for a variety of reasons. The use cases in some cases are very valuable which makes this particular topic something that a lot of different researchers have worked to understand. That Google Scholar search directed me to an academic publication with 501 citations to initially take a look at for this review. It was another Springer publication where they wanted $39.95 to download that single PDF. Those prices for access to academic works is why pre-prints and other open access journals are so popular. Erik Boiy and Marie-Francine Moens produced a work called, “A machine learning approach to sentiment analysis in multilingual Web texts,” [2]. You could go out to Research Gate and get the publication without the paywall as it was shared by one of the authors [3]. The article is surprisingly easy to read and happens to be very direct in the delivery of content. Back in 2008 and 2009 they spent time talking about manually annotating and working with different techniques. Current machine learning efforts have come a long way thanks to sample training datasets where a lot of this type of mapping or annotation has been done already. To that end you can get a lot of off the shelf machine learning models for this type of sentiment analysis that outperform the 83% accuracy they found for English texts. After digging into that foundational article in my review I’ll admit my focus shifted to looking at how people use sentiment analysis to mine data from Tweets on Twitter. One of the articles that caught my attention was, “Machine Learning-Based Sentiment Analysis for Twitter Accounts,” from 2018 [4]. That article also very quickly referred to sentiment analysis as opinion mining which seems to be a popular convention at the start of research papers. On the Twitter development webpage you can even find instructions related to, “How to analyze the sentiment of your own Tweets,” that comes complete with code snippets [5]. Other coding examples exist as well that show step by step methods of doing this type of sentiment analysis [6]. The paper referenced above and the step by step guide I shared both use the Tweepy code package to work with the Twitter API. Using that as the starting point for gathering Tweets the next step in the process involves getting ready to do some type of sentiment analysis. That happens to be the part of the equation I generally find interesting. The researchers from the above article basically made a classification model that yielded percent positive and percent negative. The actual methods for determining the positive or negative sentiment are very interesting. That type of explicit encoding hardly requires a machine learning model. You would not have to train the model on anything as every word is already encoded with a value. You could spend an entire day reading the guides hosted on the Berkeley Library website [7]. You can go find a bunch of different sentiment analysis dictionaries with scoring and other elements to do this type of analysis. It used to be a lot harder to achieve before the scoring was just something you can call or reference. I wonder about how much the scoring shifts over time and based on the collective national mode. Languages are always shifting based on the way we use and apply meaning to words. Something with a relative low score could shift rapidly with

For years it has been a running joke with some of my close friends that I have been perpetually writing a book about the intersection of technology and modernity. To that end my consideration of how we build and what it means for our shared social fabric in terms of civility and technology has been a major part of my creative efforts over the years. Modernity as a word is an attempt to signal that the now or more to the point something very current is being considered. Singularity is a word that has been used in a variety of ways. When you are describing a phenomenon in space the word singularity could be used to describe the formation of a black hole event. Some type of force would compel matter to keep compressing into a single spot until the density of that formation becomes a singularity. In this case the word singularity is being applied to a technological based scenario where in a similar fashion the advance of technology continues to the point where the advance itself cannot be stopped anymore. Within that metaphor technology would have advanced to a point where no matter what action was taken the advance of technology would continue in an uncontrolled way. A lot of science fiction authors have dealt with the rise and fall of the singularity. You could easily hide under a pile of them and read for years. It is a very interesting subject to take on as it allows for a myriad of very advanced problems to deal with and typically creates all sorts of drama. My title for this particular essay deals with the possibility of touching the singularity. To that end I’m describing a point of time before the intersection of technology and modernity. You can argue with the advance of large language models or foundation models, potentially the advance of technology is starting to become a snowball that will roll down the mountain either slowly or very quickly toward the singularity. I went back to look at the paper published by a ton of researchers at Stanford University, “On the Opportunities and Risks Foundation Models,” only to find that the words “singularity” and “modernity” do not occur [1]. All four of the references to “awareness” were about public understanding and not related to the state of an AI’s understanding of the world. The idea of an artificial consciousness is typically adjacent to arguments related to the formation of a technological singularity. This problem is a complex one to begin to dig into and it took me about 500 words split between two paragraphs to start to set up the groundwork necessary to begin to question the intersection of technology and modernity. A simpler way to get to this point probably exists, but it will take some more practice on my part to introduce in the future. At this point, I’m very curious and you may be as well about just how close we are to touching the singularity. I looked at a very well referenced paper, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion,” published in 2016 by Vincent C. Müller and Nick Bostrom [2]. That analysis of expert opinions found that there is a 50/50 chance between 2040 and 2050 that a general artificial intelligence or AGI would spring into existence or be created. Arguments can be made and are being made about if the singularity is inherently good or bad for civil society and civility in general. That is not a consideration I’m working with at the moment. My consideration of this is as an event or more to the point right before the event occurs. I did go back and read an article from a 2015 issue of the New Yorker magazine online called, “The Doomsday Invention: Will artificial intelligence bring us utopia or destruction?” [3]. That article is principally about Nick Bostrom and does consider utopia and destruction if you want to go give that a read. We probably do not have the robotic automation available for an AGI to manifest as both being capable of awareness and constructing things. We are building highly complex 3D printing and other automation methods, but the supply chain for that is pretty complex at the moment. We would probably be able to see the rise of that type of technology before a more gradual intersection of technology and modernity occurred. It would have to get pretty far along to be a true singularity in the sense that it would be something that could continue on as an unstoppable force. Right at that point of moving toward the singularity is where my thoughts refocus on the social impacts to civility, our social fabric, and the normative patterns that make up our shared perspective. As I continue to dig into that point of time and really try to think about the branching of outcomes I do think it will eventually turn into a lengthy manuscript or a full book. Over the last twenty years or so I have not been ready to complete that writing effort. I have written extensively about normative patterns, civil society, and how technology relates to those concepts. I have always just fallen short of re

Without question a good number of you reading these words right now have interacted with at least one network platform today. A futurist who was trying to predict the rise or fall of network platforms would probably side with a foreseen expansion and increased clustering of network platforms going forward. Facebook (Meta), Twitter, Slack, and other places where a technology driven platform brings a community together by hosting and maintaining a network are going to end up being categorized as network platforms. Even this very Substack post that you are reading right happens to have been distributed by a network platform of a much smaller scale than the previously mentioned ones. Adding an element of artificial intelligence as the base of a network platform is an interesting proposition. First you have to assume that an artificial intelligence could be created that is capable of being the foundation of a network platform. Second, artificial intelligence would have to be interesting enough to maintain or sustain the network platform without destroying the hosted community. That very well could be a case of destruction through stability or interaction. You have to remember that an artificial intelligence would be able to create an asymmetric amount of content compared to the user community. Orders of magnitude would be required as a measure to describe the potential flooding that could very well occur without some degree of regulated order and consistency. For the most part all of that could be done without the general artificial intelligence being aware. It could be an AI network platform capable of a multitude of tasking, but not directly aware of what the culmination of that tasking really involves. Last week while we were digging into what you get with a general artificial intelligence more time should have been spent on the concept of awareness and what that means. A lot of science fiction works have referred to computers, homes, or ships that have built in general artificial intelligence (GAI). That amazingly advanced GAI will handle requests and complete tasks both requested and simply needed. All of that could end up getting transferred to the needs of future AI network platforms, but we are pretty far away from getting to that point based on what we have right now. Voice assistants on phones are just barely branching into completing more than a few tasks. They are making progress for sure and we are getting closer to being able to ask the computer to complete a task with the advent of models like GPT-3 and how it is being extended to create things. I’m working through the process of incorporating my links and top 5 tweets of the week into the main podcast recording each week. Getting to that point will require providing coverage of the content in a written format within the newsletter that can be read which would facilitate inclusion in the podcast. Early issues of The Lindahl Letter did include more commentary with the links and Tweets, but over time I shifted over to a method of just including 7-10 pieces of content that caught my attention that week and ended up becoming curated content. Providing clear voice overs for why a YouTube video was included will require more effort than the simple act of embedding the link as a curator of content. For this penultimate episode which essentially is an homage to Marvin the android who Douglas Adams brought to life in the Hitchhiker's Guide to the Galaxy, I’m going to give integrating the content a try. Links and thoughts:1. Some of you are aware that I keep recordings of classic political speeches on my phone and listen to and break them apart for fun. That interest in politics has existed for decades. From being captain of a high school debate team to just loving deep political trivia this podcast episode from actor Rob Lowe with White House Press Secretary Jen Psaki was an excellent conversation. Rob did an amazing job of bringing trivia to life during the episode, “Jen Psaki: The Go-To Person.”2. This week Linus and Luke are super excited about the launch of the Steam Deck handheld gaming experience. They shared an episode of the WAN show called, “Steam Deck Review: PC Gaming in Your Palm, at Long Last.” Linus indicated this week that drama was off the table, but that never really happens on the WAN show. 3. You might be wondering why this episode was included based on just seeing the thumbnail alone. AJ and the team over at DFB shared an episode titled, “The Truth About Disney's Star Wars Hotel -- Star Wars: Galactic Starcruiser.” It really is a video with 25 minutes of commentary and thoughts on a really expensive Disney experience. I found it fascinating and included it this week. In order to facilitate the process of reading the commentary some numbering has been added above to help signpost the change in what link is being discussed. Including this content could end up pushing the recording length from the 5 minute range to somewhere near 10 minute

Thank you for tuning in to this audio only podcast presentation. This is week 60 of The Lindahl Letter publication. A new edition arrives every Friday. This week the machine learning or artificial intelligence related topic under consideration is, “General artificial intelligence.”We made the ten hour drive from Denver, Colorado to Kansas City, Kansas this week. Last time we made the lengthy drive on I-70 we listened to the audio book recording of, “The Age of AI: And Our Human Future,” by Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher that was published toward the end of last year on November 2, 2021. It was a pretty decent cut at the current state of artificial intelligence and its limitations. I had looked around for another artificial intelligence related audiobook to listen to during this trip. Instead of going with an audiobook this time around, I ended up listening to some of my favorite podcasts during the 10 hours of driving time.According to a Pew Research Center survey of adults in the United States nearly a quarter of people get their news from podcasts [1]. The other key finding from that survey was that younger adults 18-29 were even more likely to consume news by podcast at 33%. Some of the curated news podcasts are just a higher quality of content than what you would see on cable news. Consolidated and well packaged content comes from solid editing and well thought out efforts. Getting news on demand when you want it is much easier than waiting for cable news to gather enough momentum to get to the point. Based on the survey data a shift is occurring. Podcast subscriber numbers are also indicating that based on audience size the shift may have already occurred.For those of you who are curious what podcasts made up my 10 hours of traveling time I made a list. Included in that list were The Daily from the New York Times, Start Here from ABC News, The Vergecast, The NPR Politics Podcast, This Week in Google by Leo Laporte, Bourbon Pursuit, and Sway by Kara Fisher. The episode of Sway we listened to was about, “Tech’s Love Affair With Miami,” which was a very interesting look at how the clustering of technology focused people is changing. Kara and Keith talked about the migration from Silicon Valley to Miami. I previously shared a look at where the main AI labs are located back during my week 34 post about, “Where are the main AI Labs?” I had referenced a report that came out from The Brookings Institute by Mark Muro and Sifan Liu titled, “The geography of AI: Which cities will drive the artificial intelligence revolution?” [2]. The locations according to that report, “...include eight large tech hubs—New York; Boston; Seattle; Los Angeles; Washington, D.C.; San Diego; Austin, Texas; and Raleigh, N.C.—and five smaller metro areas that have substantial AI activities relative to their size: Boulder, Colo.; Lincoln, Neb.; Santa Cruz, Calif.; Santa Maria-Santa Barbara, Calif.; and Santa Fe, N.M.” You may have noticed that none of that seemed to include Miami, Florida. Previously I shared a book called, “Artificial Intelligence: A Modern Approach,” by Stuart Russell and Peter Norvig [3]. This is probably my favorite topic within the artificial intelligence space. It happens to be the topic that powers a ton of science fiction movie and book plots. General artificial intelligence is much more exciting than any special or specific applications of artificial intelligence. The ability to solve a variety of problems and work beyond a single use case is exciting. I started reading an article from Ragnar Fjelland about, “Why general artificial intelligence will not be realized” [4]. Working through the article was like a refresher on philosophy and the nature of intelligence. It certainly is one perspective on general artificial intelligence. You could go with a more positive perspective from McKinsey and Company called, “An executive primer on artificial general intelligence” [5]. That article starts out by acknowledging that technology is changing, but getting to a true general intelligence is probably pretty far off. However, unlike Ragnar they don’t argue that it is an impossible task. Next I turned to Forbes from 2021 to read a little more about, “The Future Of Artificial General Intelligence” [6]. Some of the coverage is hopeful and some of it theorizes the endeavor is hopeless. Now at the end of this post is the time to share a short update on my audio recording methods. This week's audio was recorded using the Yeti X professional microphone’s cardioid mode instead of the stereo mode which apparently kicks on automatically. The cardioid mode is geared toward recording sounds directly in front of the microphone which should be better for a podcast. You are welcome to let me know in the comments if you prefer the reverb of the stereo mode of previous episodes or the more targeted audio of the cardioid mode that delivered this recording.Footnotes:[0] https://www.pewresearch.org/fact-tank/2022/02

You might well be aware that multimodal machine learning (MMML) is a slice of the machine learning universe. Even typing the title of this post was challenging. I really wanted to type multi model vs. modal with an “E” before the last “L” instead of an “A” like the actual wording requires in this case. The definition of multimodal is really direct and does not include a ton of mystery. The word is used to describe something with more than one mode. You might end up quickly running down the path of multimodal deep learning to help describe it. As a person is capable of taking in the world with multiple senses and converting those signal paths into one stream for analysis. A multimodal deep learning network could be built to evaluate multi types of inputs. That really becomes a lot more complicated than it sounds within the modeling space. Our current class of models does not demonstrate the practical skill a person would at differentiating senses and understanding them in real time [1].Within the machine learning space the first generation of models were really focused on achieving very specific tasks. They received highly defined training on very specific data problems with very curated training datasets. At some point, machine learning models or builds will need to be able to take on more than one type of tasking. I think that is a really fascinating part of machine learning to study. Expanding the capabilities and ultimately what is possible changes future trajectories. I ended up reading a paper titled, “Recent Advances and Trends in Multimodal Deep Learning: A Review,” from 2021 [2]. It was pretty much the exact paper I was looking to read to really dig into the topic under consideration today. That paper really focused on video and language examples which really put things in context. One of the things that I realized during the course of my research on this topic was that a treasure trove of recorded lectures exist on YouTube. A lot of them are related to computer science, machine learning, and artificial intelligence. That is a really good thing if you were trying to put together a syllabus geared toward providing an introduction to machine learning. The video that I spent the most time watching this time around was from Victoria Dean’s, “MIT 6.S191 Lecture 5 Multimodal Deep Learning,” lecture from 2017 [3]. Somebody who was willing to put in the work to curate the content could easily pull together all the lectures necessary from a multitude of different sources. After reviewing the current trends in multimodal deep learning my interests shifted to one particular topic related to automated ICD coding. A quick Google Scholar search for "automated icd coding" in quotes and multimodal machine learning produced a ton of interesting results [4]. My search returned 94 results which was pretty surprising given the targeted nature of the terms used. Some of the articles were related to feature extraction and others were really keyed in on trying to get to the point of working with an ICD code or completing the action of coding content. One of the results that dug into automated ICD coding caught my attention was titled, “A Deep Learning Framework for Automated ICD-10 Coding” [5]. Ultimately the automation would help physicians be more productive and accurate in working toward a diagnosis. That seems like a noble effort to assist physicians and help patients.Links and thoughts:“MIT 6.S191 Lecture 5 Multimodal Deep Learning”“Steam Deck: What I Didn't Say In My Review - WAN Show February 25, 2022”Top 5 Tweets of the week:Footnotes:[1] https://towardsdatascience.com/multimodal-deep-learning-ce7d1d994f4[2] https://arxiv.org/pdf/2105.11087.pdf [3] [4] https://scholar.google.com/scholar?hl=en&as_sdt=0,6&as_vis=1&qsp=6&q=%22automated+icd+coding%22+multimodal+machine+learning&qst=br [5] https://books.google.com/books?id=81A2EAAAQBAJ&lpg=PA347&ots=2k6gK3lL_d&dq=%22automated%20icd%20coding%22%20multimodal%20machine%20learning&lr&pg=PA347#v=onepage&q=%22automated%20icd%20coding%22%20multimodal%20machine%20learning&f=false What’s next for The Lindahl Letter?* Week 60: General artificial intelligence* Week 61: AI network platforms* Week 62: Touching the singularity* Week 63: Sentiment and consensus analysis* Week 64: Language models revisitedI’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Let me start with a quick update on something Substack related. Throughout the last month I attended an invitation-only weekly forum hosted by Substack called Substack Go where writers were brought together to help encourage each other. For me writing on Substack is generally a solitary activity that happens mostly on Saturday morning. That effort is followed by a Sunday morning of editing and expansion. Finally the rest of the week is spent tinkering to get to a final post that goes out on Friday. A lot of the content being presented during Substack Go is about how to craft and write a newsletter which is interesting to hear about. This particular series has reached 58 weeks and the weekly rhythm of writing and publishing seems to be working. Even my new podcast variation of this series has been working. The two topics that I have spent the most time writing about within the machine learning space are strategy (ROI/KPIs/budgeting) and training efforts like bringing somebody or a team up to speed. Training is wholesale an element that should be a part of any organizational ML strategy. This post certainly could start by harkening back to Week 4 that appeared on February 19, 2021. During that week I tackled the topic, “Have an ML strategy… revisited.” Contained within that analysis were two questions targeting “What exactly is an ML strategy?” and “Do you even need an ML strategy?” The answer to that pivotal question is still of course an organization should have a machine learning strategy. Beginning with the end in mind and having that direction tied to budget level KPI is really a minimum standard at this point. Within this analysis the question really is if teaching or training machine learning skills is the right path to take. Inherent within that analysis has to be a question about the reason for wanting to learn the machine learning skills or acquire that specific knowledge. Bringing team members up to speed is certainly a reason to champion the training process for introducing machine learning skills. On the other side of that consideration is a scenario where learning about the foundations and building up knowledge inspired the need to pick up machine learning skills. In that case the collecting of knowledge simply compels more collecting. Continuing this general line of thought about training; I am going to break up the next two sections into concentrated coverage of individual enrichment followed by coverage of team building. Individual enrichment as a reason for advancing machine learning skills covers a significant, but not overwhelming number of people that comprise this use case universe. Students and lifelong learners alike see the world of machine learning and either have a fear of missing out or are wondering what the hype is all about. The hype in the marketplace is all about a new technology that can be applied to business use cases. People look at potential and are willing to make a leap to what it could possibly mean for them or what it could possibly do for the organization. Within the machine learning space going from model to production is a journey. Even the process of keeping a model well tuned and running in production is a consideration that means that the journey never really ends. Team building or more to the point the individual practitioners within teams or solo efforts that advance machine learning within organization certainly comprise the overwhelming bulk of the people contained in this use case universe. People at all sorts of different skill levels are working and learning within the machine learning space. Very few of them have the knowledge, skills, and abilities to create a machine learning system like TensorFlow or PyTorch. The preponderance of work in the machine learning space is done on the shoulders of giants based on using frameworks and systems that were already built. However, we are starting to see a lot of the newest innovations coming out of organizations like Hugging Face and EleutherAI. The developers with the skills to produce those foundational platforms and tooling that people are going to be using in the machine learning space will continue to be a tier above the much larger pool of people using the products. That asymmetrical dynamic is unlikely to change. To end this post, I thought I would provide you with a short update on my audio editing strategy. Within the Audacity software I’m using three different effects to clean up my podcast audio. First, a noise reduction effect which I have used for a long time in the product is kicked off and executed. The noise gate effect would probably catch it all, but I still run it first. Second, a loudness normalization effect is run which is key for any podcast as it makes sure nothing shocking in terms of a volume spike happens to the listener. Third, I have started using the built in noise gate effect function to trim out background noises and any breaths that might remain. Between those three effects in Aud

This is the third week of including audio integration as a part of the weekly Substack post for The Lindahl Letter. Recording a reading of the first section of the post that contains the main content is easy enough to achieve using Audacity with a basic use of both the noise reduction and loudness normalization effects. Between those two effects it will keep a noise floor in place to keep anything from being too loud during the recording. For those of you who are curious to complete the recordings I’m using a Blue Microphones Yeti X professional microphone on my desk in my office. It has proven to be a little more dynamic than the Samson microphone that normally accompanies my audio recordings. My very high level introduction to the creation of a machine learning syllabus would center on 8 core topics:* Bayesian optimization* Deep learning* Classic ML algorithms* Classic neural networks* Neuroscience* Reinforcement learning* Graph neural networks* Ethics (fairness, bias, privacy)I would probably break out the systems and tools part of the conversation to a separate course or keep that content included as supplemental reading. Getting to the point of being able to work toward an 8 topic introduction to machine learning involved really digging into a few other machine learning course introductions. I’m going to share links to six of them with you today, but that should not be considered an exhaustive list. It is just a list of content that caught and held my attention enough that I felt curating it in a list would be helpful. * Stanford CS229: Machine Learning [1]* MIT Opencourseware: Introduction to Machine Learning [2]* NYU Introduction to Machine learning [3]* Tufts CS COMP 134 Intro ML [4]* Berkeley CS 189/289A Introduction to Machine Learning [5]* Washington: CSE/STAT 416, Summer 2020: Introduction to Machine Learning [6]Taking a step back and thinking about the most direct solution moving forward on this path I would probably just assign a class or at least highly recommend the students to read a book called, “Artificial Intelligence: A Modern Approach,” by Stuart Russell and Peter Norvig [7]. The current version appears to be the 4th edition [8]. My bookshelf has a copy of the 3rd edition and that is the one that I picked up and started reading a few years ago. Oddly enough during the course of reading about the 4th edition the Pearson website suggested that I read, “Quantum Computing Fundamentals,” [9]. That would take this dialogue into another direction. At some point, that pivot might happen for a few weeks. I did check eBay to see if anybody was selling a used copy of that book on quantum computing. Nobody has let go of that tome of quantum goodness just yet. At some point, I’m sure one will show up for resale or my research interests will veer that direction enough to justify the price.Getting back to the main point of talking about that book by Russell and Norvig. Machine learning elements always feel like a foundational build up to work with more advanced topics in artificial intelligence. Machine learning models that are able to handle a multitude of problems are potentially becoming more likely. Week 79 to 86 of this newsletter will be devoted to creating outlines or a first draft of what the 8 core topics in my proposed syllabus would need to contain. I’m going to try to intermix a lot of artificial intelligence related observations into the machine learning focused content for each of those Substack posts. Bringing the two content sets together in a very intentional way should provide a solid written outcome.Links and thoughts:“Newegg... More Like Rotten Egg! - WAN Show February 11, 2022”“AI Show Live - Episode 51 - NVIDIA DeepStream development with Microsoft Azure” “HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning (w/ Author)” “[ML News] Uber: Deep Learning for ETA | MuZero Video Compression | Block-NeRF | EfficientNet-X” Top 5 Tweets of the week: Footnotes:[1] Stanford University. (n.d.). Syllabus and Course Schedule. CS229. Retrieved February 13, 2022, from https://cs229.stanford.edu/syllabus.html [2] Massachusetts Institute of Technology. (n.d.). Introduction to Machine Learning. MIT OPENCOURSEWARE. Retrieved February 13, 2022, from https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-036-introduction-to-machine-learning-fall-2020/ [3] Mehryar Mohri. (n.d.). Introduction to Machine Learning. Retrieved February 13, 2022, from https://cs.nyu.edu/~mohri/mlu11/ [4] Hughes, M. (n.d.). Introduction to Machine Learning. Syllabus. Retrieved February 13, 2022, from https://www.cs.tufts.edu/comp/135/2020f/index.html [5] Shewchuk, J. (n.d.). CS 189/289A Introduction to Machine Learning. Retrieved February 13, 2022, from https://people.eecs.berkeley.edu/~jrs/189/ [6] Swamy, V. (n.d.). CSE/STAT 416, Summer 2020: Introduction to Machine Learning. Retrieved February 13, 2022, from https://courses.cs.washington.edu/courses/cse416/20su/ [7] Ru

This topic has been added to my list of academic papers that I need to one day complete. Beyond that scholarly realization the core of this topic really has me conflicted about how to approach consideration about what a national AI strategy really entails. I’m fundamentally questioning what the trajectory of the strategic action would be for the different players involved. Within that conflict I have to wonder if the corporate players within a nation taking action would be inclusive of the strategy or only the official government plan should be included. Those sets of actors do not have to work together. It is an asymmetrical dynamic where one group needs the other a lot more to implement a strategy. To that end I set out to find government documents on the subject. Finding a national AI strategy document in English for a variety of different nations is actually harder to achieve from basic searching than I expected at the start of this journey. It might be an opportunity for an industrious or benevolent researcher to index all of the current plans and build a dataset for comparative analysis. Getting to the National AI Strategy for the United Kingdom online is really easy [1]. The PDF of that content comes in at a slim 35 pages. It is well written and easy to read. Clearly a 10 year plan exists to make sure Britain is competitive in the AI space. The only really big problem with that is that several of the key corporations in the artificial intelligence space are located several thousand miles away. Getting to the Artificial Intelligence Strategy document from the United States Department of Health and Human Services was fairly easy to achieve. It was published in January of 2021 and is only 8 pages in PDF format [2]. Reading both of those reports was pretty easy and only took a few minutes. From that point forward I started to wonder about the possibility of reading some comparative reports that might have been recently widely published. One of those reports that caught my attention was from the Brookings Institute titled, “Winners and losers in the fulfillment of national artificial intelligence aspirations,” [3]. That document let me know that 44 countries have entered the race by creating a national AI strategic plan. I immediately wanted to get the list and hoped that they shared links to this potential treasure trove. The Brookings Institute had two previous reports one about differing views and the other focusing on the analysis of 34 plans [4][5]. Yes, my primary focus of this inquiry so far was to dig into the plans of the United States and the United Kingdom and then reading three different reports from the Brookings Institute. I have no real preference for their reporting or research, but in this instance it was good quality reading material on the exact topic I wanted to take into consideration. The researchers did take the time to prepare a figure that shows the leaders in national AI strategy implementation which included India, Germany, China, South Korea, United Kingdom, and Canada. The United States actually fell distinctly below the line and was in an entirely different quadrant than the leaders called technology prepared. You could go read the Global AI Vibrancy Tool page from Stanford University to see a different perspective on rankings [6]. After reading all that content and starting to wonder why they did not share the actual dataset with links to all these national AI strategies I may have to reach out to some of the researchers to see if they published it somewhere else that I may have just missed during my exploration.It seemed like a good idea to go and dig into Google Scholar to see if anything stood out comparing national AI strategies [7]. It does look like some of the comparative studies schools in public policy and political science have done some groundwork. None of it really dug in and provided the higher level statistical analysis with good coding on key metrics I was looking for to better understand which strategy and trajectory would be the most successful or is being primarily implemented. It is entirely possible that the free market will end up defining where AI is going to end up flourishing. Regulation could certainly constrain the free market innovation machine enough to diminish a national AI strategy. The other consideration is that the multinational nature of modern corporations means that even the best national AI strategy could be confounded by organizational reach.Links and thoughts:“[ML News] DeepMind AlphaCode | OpenAI math prover | Meta battles harmful content with AI”“Is AI just statistics? | Yann LeCun and Lex Fridman”From SpaceX, “Starship Update”Top 5 Tweets of the week:Footnotes:[1] National AI Strategy. United Kingdom. (2021, September 22). Retrieved February 12, 2022, from https://www.gov.uk/government/publications/national-ai-strategy or you can go directly to the document here: https://assets.publishing.service.gov.uk/government/uploads/system/upl

Considering this topic really involved taking things in two different directions. First, I started to really try to understand the world and landscape of machine learning patents. Second, it became abundantly clear that acquisitions have fueled a lot of change in the machine learning space. A history of machine learning acquisitions could be a topic one day. That will probably happen in week 78. Some of that acquisition has been about bringing talent to internal teams and some of it has been about patent acquisition [1]. After those two considerations bubbled up to the forefront of my thoughts I also started to wonder about patenting ML models compared to patenting the underlying technology. You can very quickly go out and find out that last year the United States Patent and Trademark Office’s (USPTO) released a new artificial intelligence patent dataset [2].That gigabyte sized dataset download included 13.2 million patents and pre-grants that you can study on the topic of artificial intelligence [3]. The actual download speed to get that file was exceedingly slow and even with my gigabit internet connection took over 20 minutes. While I was waiting to review the actual data I tried to read the linked journal article about the dataset from Giczy, A.V., Pairolero, N.A. & Toole, A.A. Identifying artificial intelligence (AI) invention: a novel AI patent dataset. The Journal of Technology Transfer (2021) [4]. That article is behind a Springer paywall and costs $39.95 which is a little wild for access to a full PDF of an article. However, it was much easier and surprisingly informative to just watch a narrated PowerPoint presentation from Nicholas A. Pairolero from June 25, 2021 that was presented at the AI and Patents Workshop at ICAIL 2021 [5]. Let’s pivot back to understanding the landscape of machine learning patents. You will very quickly find out that the team over at Google is proudly displaying a patent from December 1, 2005 about machine learning systems and methods [6]. The patent in PDF format is 17 pages long and a pretty easy read. Pivoting from that patent I started to wonder about who is getting patents now and if that really matters anymore in the machine learning space [7]. I learned during that research that it might be fun to work with a website called Patent Guru (patentguru.com) to visualize frequency [8]. I pretty quickly hit the limits of what free access to that website using the analysis features would provide and ended up circling back to the dataset I downloaded. During the course of getting that dataset loaded up I started to consider the idea of patenting AI algorithms and even the future consideration of if an AI built for drug discovery could own the patent or resulting intellectual property from that discovery.Links and thoughts:“Identifying Artificial Intelligence Invention: A Novel AI Patent Dataset (AI & Patents 2021)”Data Science Grandmaster Series from NVIDIAhttps://www.youtube.com/playlist?list=PL5B692fm6--uXbxtmPJz5nu3Xmc1JUm3F “tinyML Talks: Energy-Efficiency and Security for TinyML and EdgeAI: A Cross-Layer Approach”“GPT-NeoX-20B - Open-Source huge language model by EleutherAI (Interview w/ co-founder Connor Leahy)”“Getting Started with Sentiment Analysis using Python”https://huggingface.co/blog/sentiment-analysis-python “They Almost Got Away With It! - WAN Show February 04, 2022”Top 5 Tweets of the week:Footnotes:[1] Sawers, P. (n.d.). 13 acquisitions highlight Big Tech’s AI talent grab in 2020. Venture Beat. Retrieved February 6, 2022, from https://venturebeat.com/2020/12/25/13-acquisitions-highlight-big-techs-ai-talent-grab-in-2020/ [2] USPTO releases new Artificial Intelligence Patent Dataset. USPTO. (n.d.). Retrieved February 6, 2022, from https://www.uspto.gov/about-us/news-updates/uspto-releases-new-artificial-intelligence-patent-dataset [3] Artificial Intelligence Patent Dataset. USPTO. (n.d.). Retrieved February 6, 2022, from https://www.uspto.gov/ip-policy/economic-research/research-datasets/artificial-intelligence-patent-dataset [4] Giczy, A.V., Pairolero, N.A. & Toole, A.A. Identifying artificial intelligence (AI) invention: a novel AI patent dataset. J Technol Transf (2021). https://doi.org/10.1007/s10961-021-09900-2[5] [6] https://patents.google.com/patent/US20050267850A1/en [7] Obee, E. (2021, August 18). Artificial Intelligence Patent 101. Towards Data Science. Retrieved February 6, 2022, from https://towardsdatascience.com/artificial-intelligence-patent-101-3eebf93f5297 [8] "machine learning". PatentGuru. (n.d.). Retrieved February 6, 2022, from https://www.patentguru.com/analysis?area=US&q=%22machine+learning%22What’s next for The Lindahl Letter?* Week 56: Comparative analysis of national AI strategies* Week 57: How would I compose an ML syllabus?* Week 58: Teaching or training machine learning skills* Week 59: Multimodal machine learning revisited* Week 60: General artificial intelligenceI’ll try to keep the what’s next list forward looking with at leas