Regular Programming

Stories about Ecto quickly redeeming itself, and of what it takes to introduce foreign keys.Some of us are super comfortable referencing the ID. Lars dislikes that Ecto needs to be more complicated because of SQL, but the abstractions do hold.Also: the biggest reason to ever use a ORM! It can be reallynice to come back to one after a tour of plain SQL-land.Some people have just been bitten so hard by cowboys.LinksEctoForeign keysRethinkDBReferential integrityAXALantmännenModelForm in DjangoCowboy and PlugDSLUpsertsFragmentsHaxl - DSL for creating queriesSQLAlchemyets

Fredrik wants to think about long-lived code. Lars is offended, Andreas only a little bit so.Are there other good software development practices out there? Other than the ones focusing on building something quickly? Practices for building software which lives on and is maintained for much longer than we seem to care to admit? Should we remove dependencies over time? The swamp of dependency management and vendoring is probed, gradually shifting into firmware, the horrors of floating point (proper excuses are made), small language models.Finally, of course, indecent cups of tea.LinksLagomReactFlux architectureReduxChangelog episode with Justin Searls about dependencies as liabilitiesKent Beck talking about managing risks in software developmentKent Beck drawing on a whiteboard and staring at the audienceMithril.jsInteract.jsVendoringWorking effectively with legacy code - the book about legacy systemsDelphi 5FlaskDynamic linkingSAMLPOSIXLibcGlibcMuslH.264MicrocodeOxide and friendsCoral TPU:sTensorflow lite286PentiumCUDAROCmQuantizationLLaMA

The Saint Valentine's peak passed without issue. Andreas had time for semlor.Lars has opinions on semlor, and can imagine many possible improvements. Like having an apple. Or a pizza.Lars has had a nice influx of work, including hardware work using Nerves. Testing and very hackish hot code reloading are both included.Finally, some thoughts on Linux audio, and musings about the possibility of creating really nice audio tools for the platform.LinksSaint ValentineThe strangler fig patternThe strangler patternPhoenixCowboySemlaMudcakeThe Swedish chef making chocolate mooseFinnish fastlagsbulle with jamOne of Lars' blog posts about NervesFrank Hunleth - also hot code reloads the way Lars has doneLars' Stream deck library for ElixirStream deckElgato key lightPulseAudioPipeWireRogue amoeba's audio tools for MacJACKCustom APT repositoryQuotesThe Nordics go all awkward and weirdIn my heart, it was a catastrophyHad time for semlorAn unimpressive pastryIt's less messy to have an appleProfessional nervesBuilding with nervesA reasonable enough abstractionThe Rogue Amoeba for Linux

Andreas tells the story of a old system which suddenly exhibited a new and frightening bug. Lars shares similar experiences of things going wrong in new and novel ways.When things do go wrong, it is so nice to have supervision trees or other things which allow you to hear about problems, not to mention recover from them.Also covered are some stories about TCP, networks, and timeouts. And a realization that testing the frameworks upon which you build could have saved some bacon, had it just been done a long time ago.LinksDjangoModel-view-controllerDrupalUnicode collationSupervision treesOxide and friends - episode 27TCP_NODELAYQUIC and HTTP/3UDPNyqvist-Shannon sampling theoremHexagonal designQuotesGaming convention management systemWhen I say view, I mean controllerView is a better wordIf I ignore it, it will go awayDestructive favouritesAlternative class hierarchiesFailed in new and novel waysBoth a mistake, and interestingAaah, circumflex!TCP the good parts

Lars dove into data pipelines, and emerged bearing arrows and wishing for a lot fewer copies.What is there to think about regarding data pipelines, what is interesting about them?Which tools are out there, and why might you want to use them?Why all this talk about making fewer copies of data?What does Lars' current ideal pipeline look like, and where does Elixir fit in?LinksMatt TopolApache ArrowLarge language modelsVector searchBigQuerysedAWKjqReplacing Hadoop with bash - "Command-line Tools can be 235x Faster than your Hadoop Cluster"HadoopMapReduceUnix pipesDirected acyclic graphtee - to "materialize inbetween states"Apache BeamApache SparkApache FlinkApache PulsarAirbyte - shoves data between systems using connectorsCronjobFivetran - Airbyte competitorApache AirflowETL - Extract, transform, loadDesigning data-intensive applicationsStream processingEphemeralityData lakeData warehouseThe people's front of JudeaDBT - SQL-SQL batch-work-thingySQL with Jinja templatesSnowflake - data warehouse thingScalaBroadwayOban - "robust job processing for Elixir"Dashbitpandas - Python data libraryAPLArrow flightGRPCDataFusion - query execution enginePolars - "DataFrames in Rust"Explorer - built on top of PolarsVoltron dataThe Composable CodexPyarrow - Arrow bindings for PythonQuotesI've been reading a lot about data pipelinesWhat's so special about data pipelines?There's a lot of special toolingThere's a lot of bad, bad toolingLess than optimal toolingConverging on something biggerlkHe got me eventuallyAll of your steps in one bucketWhat tools do you associate with data?I inherited a data pipelineBashReduceIterate on the L and the TThe modern data stackAnd then you demand more workNo unnecessary copiesBarely a copyReconnecting with my Python roots

GenServers are fun! Andreas gives all the context. Things were learned, knowledge was aquired. You can do so much with GenServers, but make sure you have a good reason.If you don't watch out, this is where concurrency goes to die.Dynamic supervisors, and their children, are thoroughly considered.Also delved into is the mess other ecosystems make of doing things at the same time, waiting, and so on.The strange worlds of C and other unusual languages are considered.Finally, an interesting bug.LinksAlan TuringTuring machineGenServerCowboyPlugUmbrellaETS - Erlang Term StorageØredevThe actor modelVirding's first rule of programmingRegistryDynamicSupervisorThe Goth library - Google auth library for ElixirThe GIL - the global interpreter lockFriday afternoon deployPromisesEsbuildUiua - "A stack-based array programming language"Prefix treePackmatic library, by Evadne Wu - streaming zip archivesQuotesWhere the system grows horizontallyThe kind of thing that starts happening when you hire developersIt was missing a hatI have become nothing, the simplifier of thingsWhere all the concurrency goes to dieA whole dance party of sad, dark peopleThe children of the dynamic supervisorHomes can be nodesHundreds of interested partiesTurns life into promisesPoking some C programmers

Every web app starts out fine, the tabula rasa of an unwritten BODY. But sooner or later you need users. And a million other things which live in trees.Also: email.And that layer between the controller and the database where things like fine-grained access control goes.I'd like to have an admin, please.Eventually, web apps grows up. And while a larger framework with solutions and conventions for all those grown-up features may not necessarily be fun, it can certainly be useful.LinksAPM - Application Performance ManagementDjangoTeams should be an MVP feature!Bullet Train - a "Ruby on Rails SaaS framework"FlaskExpressSinatraScottyPhoenixAuth0OktaPostfixPostmarkDjango AnymailSwooshModel-view-templateACL:s - access-control listsEctoMultitenancyZack Daniel on Beam RadioZack's Elixirconf talkAsh frameworkPlugDSL - domain-specific languageBigquerygRPCHIPPAPostgrestFunction based viewsDjango RESTLaravelTitlesCheck in on your applicationDo you want details?The view is the controllerBecause namesI'd like to have an admin, pleaseThe admin is kind of roughAll the data is introspectableEndgame applicationNot another user management systemA very special can of worms

The software development industry is very much built for code nerds. It shouldn’t be.Many of us know many people who are really into coding. Not every working developer can, or even should, be though. Doesn't that create kind of a weird gap between professionals who live and breathe code both on and off work, and those who have a more balanced life?Being passionate about your job shouldn't be an expectation or requirement for anyone or anything.Is there too little space for learning - are we assumed to know too much, and assumed to spend our own time figuring out things we don't?Your path into coding is not, can not, and should not be the only path possible.LinksThe Python 2 to 3 transitionRobert A. Heinlein in 999 Words: What Every Human Should KnowGhost in the shellHarvest moon4x - Explore, expand, exploit, exterminateTDD - test-driven developmentBDD - behavior-driven developmentCharity Majors 2017 blog post about career paths for developers. (Bonus: 2019 follow-up about engineering managers)Late-stage capitalismQuotesI think that's perfectly healthySurrounded by themDelving into softwareSurrounded by nerdsMuch more reasonable answersWhere the nerd doesn't go so deepComputers are troublesomeWhy should you be passionate about your job?Squeeze the passion juiceToo passionate to defend themselvesExperience or scar tissue?Many developers have livesPopping out for the big pictureDoing good work takes all kinds

Data has moved to a real database. Next, there may be brave attempts to add actual structure. Working with a real database is nice, as is not losing data, and being able to restore.Not everything is ephemeral, after all.Database service providers and cool stuff they do are discussed. The deal with Elastic is clarified. Finally, it is revealed where you should store your traces.It is actually probably fine.LinksMongoDBRethinkDBEctoEcto changesetsDatabase schemaOpenAPIAsh frameworkDjangoDjango REST frameworkX-Request-IDTransactionsWrite-ahead logpg_restorepg_dumpBigqueryLogical replicationFly.ioElectricSQLFlutterReact nativeCRDT:sPhoenix liveviewFireflyWebassemblySQLiteDuckDBClickhouseDeveloper voices podcastEpisode about ClickhousePlausible analyticsFathom analytics podcastPaul Jarvis - the "Company of One " guyJack EllisLaravelFathomElasticsearchMeilisearchLuceneCardinalityHoneycombOpenTelemetryNew relicHyperDXDatadogQuotesA worse MongoDBMigration completeEverything is ephemeralThe idea is to add lots of columnsIt seems a bit more DjangoThe stakes were high but the budget was almost zeroIt is actually probably fine

It seems a mingle is a thing, and not just in Swedish! But what do we want to get out of them, how do we go into them, and how do we create good ones?Do you want resonance or hole-poking when you tell people about your plan to arm toddlers with nuclear weapons? Do you want to successfully mingle nerds, or just hit the snacks hard?The foood, the cake, the coffee, and the old classmates. Too hot, too loud, too crowded.LinksMingle (noun)Ben OrensteinTupleThougtbot podcasts - The bike shed and Giant robots. Ben was on episode 183 of The bike shed and episode 136 of Giant robots.ØredevThe art of gatheringThe Whova appQuotesInteresting and funArm toddlers with nuclear weaponsWe don't trust solutionsExcitement and resonanceIntensively and excitedly and indefinitelyThe active rubberduck strategyTalking to fish in a barrellSuccessfully mingle nerdsHit the snacks pretty hard

Performance: we wish the incentives were there to focus on it more often.Lars would like more opportunities and incentives to focus on making things fast, rather than just making them not slow. Unfortunately, things tend to line up so that fast enough and more features are in focus. Plus, performance and optimization can be very context sensitive and age out without anyone really noticing.Also pondered: IRC, Gentoo, and the eldritch horrors buried within the x86 architecture.LinksGrepos.walk() in PythonWhy GNU grep is fastRipgrepAg - the silver searcherTravelling salesman problemBin packingMnesiaORMProject EulerFibonacci numbersD. Richard Hipp - the guy behind SQLiteChangelog episodes with RichardXKCD and IRCIRCIRCCloudMatrix and ElementElmSVGCanvasNeovimLispArchNixGentooFuntooDartFlutterSkia - the graphics library under FlutterLinked listPointersCISC - Complex instruction set computerRISC - Reduced instruction set computerio_uringQuotesGiven up on old man ElixirGotta go fastI never really needed itGrep, naivelyAll the problems at the same timeTravelling knapsack problemMy ORM-infected brainMeasuring things and muttering under my breathI have a hobby, I do job interviewsTools by toolmakers for toolmakersI'm the IRC guyMachine-whispering optimization

CTOs want the ability to get prototypes built and out into production fast. Others preach the gospel of building things properly. How fast can you be? How much can you perpare before you hit the ice? And one you built and shipped that prototype, how can you get any kind of speed trying to maintain and evolve something where many corners were cut for speed?How do we want things to work then? Having an algebra for things might be nice. A sprinkling of interface, things that break noisily, and nice toolboxes to work with structs are all discussed.LinksThe Scott - Amundsen race to the South poleAccelerate, by Nicole ForsgrenParse, don't validateMnesiaDeep modulesPure functionsPlugElmBruce TateCRC - Create reduce convertEctoRocHappy Path Programming. Episode 47 features Richard Feldman and RocRichard Feldman, creator of RocQuotesThe gospel of building things properlyThe key to speed on the iceBefore you hit the iceBare mapsEvery step made senseThe original intent very easily gets lostThe curse of all softwareStrive for maintainabilityIt must not sprawlA little sprinkling of interfaceAt dawn, we roadmapThings that break noisilyA quantity unitlessThe simple case of HTTP

Did they do design, or did they just do a system?Distributed systems are hard in many ways. Andreas describes a system communicating between backends and mobile phones in exciting ways with many exciting possibilities for errors. Like data format changes, loss of messages, having 1.5 source of truths, and of course ordering.In certain cases, nobody likes an optimist.The discussion then moves to discuss the working well-windows for various networking solutions, before diving into WebRTC and finishing up with the various dangers of auto.LinksRecursionEventual consistencyPubsubRethinkDBEvent sourcingReact nativeAndroid studioMnesia - a "distributed, soft real-time database management system" written in ErlangDirty reads and writesWebsocketsQUICUDPTCPWebRTCNATHTTP live streamingLars' ElixirConf talkZoom H4Zoom H4n proQuotesWorking with systems and feeling the pain Coping with system designEventually consistent, on a good dayEventually sourcedA disappointment to work withYour internal representation of the userThis is the shape of the data, deal with it1.5 source of thruthsOh, it's an optimistI don't like optimists at allOptimist databasesWithin its working well-windowOutside of the working well-windowA crash of serversBad connections over long distancesI don't do math

Lars went to ElixirConf EU. Going to a conference can be a credibly incredible experience. Elixir has more clarity than Erlang.Lars also gave a talk, a fact he was comfortably uncomfortable with. Giving a talk also comes with benefits such as being able to talk to fish in a barrel. But why did he choose to make the whole talk a demo? What is the goal of it all?Gotta build things! Dive in, make stuff.LinksElixirConf EULars' conference report blog postCode BEAMSverokPieter Hintjens about giving talks by talking to the audienceWindows 98 (not 95) demo failLars' presentation codeVoice Driven Development: Who needs a keyboard anyway? - presentation by Emily SheaHugging FaceQuotesBorn during ElixirConfLess clarity to itGenservers and stuffMainstream ElixirComfortable with that discomfortTalking to fish in a barrelA buddy from the internetThe first one I bothered to countYour loose coupling to anythingWhat do you hypothetically know?

Text editors - which ones do we enjoy, which ones have we used, and what do we actually want and need in them?Andreas has read about vim, sed and awk. Lars is quite comfortable in vim, but finds Visual studio code more than acceptable enough. Andreas is excited to show Lars how to use Vim properly. Lars considers advanced setups something of a hellscape.Lars has held a lecture about functional programming and wishes to provide a path for new .Net developers (dotnet dots?) to become free software zealots.They both share their history of editors.There are dreams of ergonomic editing - of code as well as text in general - on mobile devices.Any other editors we should be trying? No, but you could hack together collaborative vim editing. LinksHumble bundleLearning the Vi and Vim Editors - bookVimThe Anarchist CookbookThunderdomeMonadex - line editor which inspired viedsed & awk - bookAWKsedSublime textZedNeovimTmuxI3GNOMEPop!_OSKDETreesitterElixirLSFZF - fuzzy finder for the command lineRipgrepFunctional programmingMonadsFakerootNotepad.exeBorland DelphiNotepad++EclipseIntellijAndroid studioXcodeWrite/WordpadNanoPicoGeditKateNetbeansAtomScratchGNU ScreenLive Share for Visual Studio CodeQuotesLearning violent vimLike Thunderdome, but nobody leaves, everI could do that with monads insteadC's strange cousinThere's a new sed on the blockThe power of just good enoughTwo terminals beside each otherIt's all a mess in hereMy sword and lots of configuration filesThe dotnet dotsQuitters don't use VimReal code is done on the serverNotepad the way I want it to workA load-bearing noteExciting and fun, and incredibly unsafe

How do we feel about working remotely? Pretty good, on the whole.Chairs and other basics are of course important, as is making your way of remote work a nice way of doing remote work for you. It is also nice to need to wear your work face less.The challenges are more around the social sides - communicating differently, but generally replacing and rebuilding ways of being social with people both inside and outside of your work interests. That takes work.Also, some talk about audio and video gear for remote meetings. It's nice to come off as full-fidelity people!LinksConan the barbarianEventual consistencyGamers Nexus on gamin chairsUllman Nite-FliteRFC processRøde procasterXLRGhost power (phantom power)Shotgun microphoneThe Kodsnack SlackQuotesMy real comfy legendary office chairMy chair was kinda goodFluffy partsIt's me and ConanI go for the floofEventually ergonomicEventually comfortableWhenever I don't have one, I create oneYour spine has a very particular taste in chairsA prosumer phase of lifeDefinitely dialled inMake sure you have a social lifeI fetch a lot fewer coffees than most peopleGhost power!Full-fidelity peopleIt's very much my officeI don't have to wear my work face all dayMy work face

Lars is thinking about distributed systems, and Andreas kind of fears them. The best thing to do for most cases might be to avoid distributing things at all. But if you do end up needing to distribute, you may run into one of the places in the world where worse is better is not necessarily better? Adding distribution on top of something not really built for it is one of the hard problems.There are deep dives into reconciliation, vector clocks, normalization, and places where fun goes to die. And there, still, are no magical solutions.LinksPhoenix pubsubWorse is betterElectricSQLCRDT:s - conflict-free replicated data typeThe CAP theoremSoft real timeHighlanderN-tier architecturePostgres replicationVector clockElixir outlawsPhoenix presenceOperational transformationsSplit-brainRiakCouchDBRaftPaxosNormal forms for databasesGoogles' MapreduceGoogle SpannerCockroachDBCassandraContentfulThe Cambria paper - schema evolution in distributed systems with edit lensesQuotesDistributed systems are interestingI'm doing an insert!A special little serverThe devil is always in the failure detailsThe naive thresholdThe absolute wrong number of machinesWhere all the fun goes to dieA good, sortable nameThey lie and they driftA simple incrementing number is incredibly usefulGit merge for vector clocksThree is the best number

About Hackers Thinking about the term "hacker". Time to take it back to mean something rather down to earth, rather than a pedistal requiring years of C and a black hoodie?What do airlines have against Erlang anyway?There's also the mindset angle: the hacking mindset can be when exploring, versus when needing to solve a specific problem.The discussion goes into labels one feels comfortable with, switching between different modes, and the ever present, ever hard to find dark matter developers.Over time, labels can easily go bad in one way or another. But regardless of labels, we can all agree on duct tape and enthusiasm, right?LinksLet it crashBurning manActivitypubStatic site generatorCMSThe Unix philosophyCOMFFmpegviPerlEctoDark matter developersQuotesSo security, very programmingJoy and playfulnessMy mind goes offCreative systems thinkingThink through as many eyes as possibleMany things are intended as complete packagesHanding you the fun bitsThings that provide you the entire worldNot very together-bashableThe media version of ViCreating SQL that you didn't intendMostly mindsetWhat happens in the outliersNeutron programmersThe unsung programmersDuct tape and enthusiasm

About Being WrongWherein polite gentlemen at gaming conventions explain how people didn't have their variables separate enough with regard to the Dunning-Kruger effect. Lars thinks Andreas has drawn the wrong learnings from this.It's a good idea to be humble … but strong opinions loosely held may not be the perfect thing, either?Also discussed is the curse of the expert - teaching across a large gap in experience, and how to actually go about changing systems and having better discussions. Have you considered being god's advocate instead of the devil's when in a discussion?LinksGothconThe Dunning-Kruger effectAutocorrelationThe Dunning-Kruger effect is autocorrelationGolden ratioThe curse of the expertThe halting problemLinus TorvaldsStrong opinions loosely heldSteelmanning an argumentYes, and …Two's complement podcastQuotesI attempted to make friendsHard to know what you don't knowIf you don't have your variables separateThey fumbled on the input dataI think you have the wrong takeawayThe curse of the expertHave you looked at presidents recently?Exhaust the universeThe halting problem of the universeSons of pedagogyI feel comfortable, but I don't feel certainA really badass judo throwI can ignore many thingsA multitude of partsBit by bit, you shift the systemTaking small standsVery happy to be wrongGod's advocateRandom ideas, loosely sharedA good crowd for this question

About EstimatesEstimates are a nasty subject, Andreas doesn't know how to handle it.Fortunately, Lars has one weird trick, which doctors hate.When you have plenty of control, estimates can be useful.Not useful: unexplained deadlines.Finally: when things get stuck. (Lars is usually available to blame.)(In an alternate timeline, Andreas' tells us everything his relatives taught him about quark cake.)LinksDeadlines whooshing pastThe XKCD about determining if you're in a national park, and check if your photo is of a birdBasecampShape upElmZenos' paradoxes - you can't run past a tortoiseThe travelling salesman problemNP-complete problemsCRUDLean manufacturingQuotesTheir due dates, their deadlinesI have this one weird trick, that doctors hateA constraint for the workThe magnitude of the taskSome real dumb things, and some very decent ideasSkate curveThe smallest unit is always a dayNot agile enoughSlightly confused and maybe a little bit sadIf you think that's a map (, I think you're using it wrong)Assorted concernsYou can't run past a tortoiseYou can always split a cake in twoQuark cakeAccelerate a cakeA fixed estimate on the travelling salesman problemUsually available to blame