Python Bytes | Listen Free on Castbox.

122 Episodes

Reverse

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanBrian #1: Combining and separating dictionariesPEP 584 -- Add + and - operators to the built-in dict class.Steven D'ApranoDraft status, just created 1-March-2019d1 + d2 would merge d2 into d1like {**d1, **d2}or on two lines d = d1.copy() d.update(d2)of note, (d1 + d2) != (d2 + d1)Currently no subtraction equivalentGuido’s preference of + over |Related, Why operators are useful - also by GuidoMichael #2: Why I Avoid Slackby Matthew RocklinI avoid interacting on Slack, especially for technical conversations around open source software. Instead, I encourage colleagues to have technical and design conversations on GitHub, or some other system that is public, permanent, searchable, and cross-referenceable.Slack is fun but, internal real-time chat systems are, I think, bad for productivity generally, especially for public open source software maintenance.Prefer GitHub because I want toEngage collaborators that aren’t on our SlackRecord the conversation in case participants change in the future.Serve the silent majority of users who search the web for answers to their questions or bugs.Encourage thoughtful discourse. Because GitHub is a permanent record it forces people to think more before they write.Cross reference issues. Slack is siloed. It doesn’t allow people to cross reference people or conversations across SlacksBrian #3: Hunting for Memory Leaks in Python applicationsWai Chee YauConquering memory leaks and spikes in Python ML products at Zendesk.A quick tutorial of some useful memory toolsThe memory_profiler package and matplotlib to visualize memory spikes.Using muppy to heap dump at certain places in the code.objgraph to help memory profiling with object lineage.Some tips when memory leak/spike hunting:strive for quick feedbackrun memory intensive tasks in separate processesdebugger can add references to objectswatch out for packages that can be leakypandas? really?Michael #4: Give Me Back My Monolithby Craig KerstiensFeels like we’re starting to pass the peak of the hype cycle of microservicesWe’ve actually seen some migrations from micro-services back to a monolith.Here is a rundown of all the things that were simple that you now get to re-visitSetup went from intro chem to quantum mechanicsOnboarding a new engineering, at least for an initial environment would be done in the first day. As we ventured into micro-services onboarding time skyrocketedSo long for understanding our systemsBack when we had monolithic apps if you had an error you had a clear stacktrace to see where it originated from and could jump right in and debug. Now we have a service that talks to another service, that queues something on a message bus, that another service processes, and then we have an error.If we can’t debug them, maybe we can test themAll the trade-offs are for a good reason. Right?Brian #5: Famous Laws Of Software DevelopmentTim Sommer13 “laws” of software development, includingHofstadter’s Law: “It always takes longer than you expect, even when you take into account Hofstadter's Law.”Conway’s Law: “Any piece of software reflects the organizational structure that produced it.”The Peter Principle: “In a hierarchy, every employee tends to rise to his level of incompetence.”Ninety-ninety rule: “The first 90% of the code takes 10% of the time. The remaining 10% takes the other 90% of the time”Michael #6: Beer Garden PluginsA powerful plugin framework for converting your functions into composable, discoverable, production-ready services with minimal overhead.Beer Garden makes it easy to turn your functions into REST interfaces that are ready for production use, in a way that’s accessible to anyone that can write a function.Based on MongoDB, Rabbit MQ, & modern PythonNice docker-compose option tooExtrasMichael:Firefox SendEthical ads on Python Bytes (and Talk Python)Brian: T&C 69: The Pragmatic Programmer — Andy Huntnot up yet, but will be before this episode is releasedJokesFrom Derrick Chambers“What do you call it when a python programmer refuses to implement custom objects? self deprivation!Sorry, that joke was really classless.”via pyjokes: I had a problem so I thought I'd use Java. Now I have a ProblemFactory.

#122 Give Me Back My Monolith

2019-03-2200:29:061

Sponsored by Datadog: pythonbytes.fm/datadogBrian #1: Futurize and Auto-FuturizeStaged automatic conversion from Python2 to Python3 with futurize from python-future.orgpip install futureStages:1: safe fixes: exception syntax, print function, object base class, iterator syntax, key checking in dictionaries, and more2: Python 3 style code with wrappers for Python 2more risky items to changeseparating text from bytes, quite a few morevery modular and you can be more aggressive and more conservative with flags.Do that, but between each step, run tests, and only continue if they pass, with auto-futurize from Timothy Hopper.a shell script that uses git to save staged changes and tox to test the code.Michael #2: Tech blog writing live streamvia Anthony ShawLive stream on "technical blog writing"Talking about how I put articles together, research, timing and other things about layouts and narratives.Covers “Modifying the Python language in 6 minutes”, deep articleListicals, “5 Easy Coding Projects to Do with Kids”A little insight into what is popular.Question article: Why is Python Slow?Tourists guide to the CPython source codeBrian #3: Try out walrus operator in Python 3.8Alexander HultnérThe walrus operator is the assignment expression that is coming in thanks to PEP 572. # From: https://www.python.org/dev/peps/pep-0572/#syntax-and-semantics # Handle a matched regex if (match := pattern.search(data)) is not None: # Do something with match # A loop that can't be trivially rewritten using 2-arg iter() while chunk := file.read(8192): process(chunk) # Reuse a value that's expensive to compute [y := f(x), y**2, y**3] # Share a subexpression between a comprehension filter clause and its output filtered_data = [y for x in data if (y := f(x)) is not None]This article walks through trying this out with the 3.8 alpha’s now available.Using pyenv and brew to install 3.8, but you can also just download it and try it out.3.8.0a1: https://www.python.org/downloads/release/python-380a1/3.8.0a2: https://www.python.org/downloads/release/python-380a2/Ends with a demonstration of the walrus operator working in a (I think) very likely use case, grabbing a value from a dict if the key exists for entry in sample_data: if title := entry.get("title"): print(f'Found title: "{title}"')That code won’t fail if the title key doesn’t exist.Michael #4: bullet : Beautiful Python Prompts Made SimpleHave you ever wanted a dropdown select box for your CLI? Bullet!Lots of design optionsAlsoPassword “boxes”Yes/NoNumbersLooking for contributors, especially Windows support.Brian #5: Hosting private pip packages using Azure ArtifactsInteresting idea to utilize artifacts as a private place to store built packages to pip install elsewhere.Walkthrough is assuming you are working with a data pipeline.You can package some of the work in earlier stages for use in later stages by packaging them and making them available as artifacts.Includes a basic tutorial on setuptools packaging and building an sdist and a wheel.Need to use CI in the Azure DevOps tool and use that to build the package and save the artifactNow in a later stage where you want to install the package, there are some configs needed to get the pip credentials right, included in the article.Very fun article/hack to beat Azure into a use model that maybe it wasn’t designed for.Could be useful for non data pipeline usage, I’m sure.Speaking of Azure, we brought up Anthony Shaw’s pytest-azurepipelines pytest plugin last week. Well, it is now part of the recommended Python template from Azure. Very cool.Michael #6: Async/await for wxPythonvia Andy BulkaRemember asyncio and PyQt from last week?Similar project called wxasync which does the same thing for wxPython!He’s written a medium article about it https://medium.com/@abulka/async-await-for-wxpython-c78c667e0872 with links to that project, and share some real life usage scenarios and fun demo apps.wxPython is important because it's free, even for commercial purposes (unlike PyQt).His article even contains a slightly controversial section entitled "Is async/await an anti-pattern?" which refers to the phenomenon of the async keyword potentially spreading through one's codebase, and some thoughts on how to mitigate that.ExtrasMichael: Mongo license followupWill S. told me I was wrong! And I was. :)The main clarification I wanted to make above was that the AGPL has been around for a while, and it is the new SSPL from MongoDB that targets cloud providers.Also, one other point I didn't mention -- the reason the SSPL isn't considered open source is that it places additional conditions on providing the software as a service and the OSI's open source definition requires no discrimination based on field of endeavor.Michael: python2 becomes self-aware, enters fifth stage of griefFunny thread I startedpython2 -m pip list DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.Michael: PyDist — Simple Python PackagingYour private and public dependencies, all in one place.Looks to be paid, but with free beta?It mirrors the public PyPI index, and keeps packages and releases that have been deleted from PyPI. It allows organizations to upload their own private dependencies, and seamlessly create private forks of public packages. And it integrates with standard Python tools almost as well as PyPI does.JokeA metajoke: pip install --user pyjokes or even better pipx install pyjokes. Then:$ pyjoke[hilarity ensues! …]

#121 python2 becomes self-aware, enters fifth stage of grief

2019-03-1600:23:341

Sponsored by pythonbytes.fm/digitaloceanBrian #1: The Ultimate Guide To Memorable Tech TalksNina Zakharenko7 part series that covers choosing a topic, writing a talk proposal, tools, planning, writing, practicing, and delivering the talkI’ve just read the tools section, and am looking forward to the rest of the series.From the tools section: “I noticed I’d procrastinate on making the slides look good instead of focusing my time on making quality content.”Michael #2: Running Flask on Kubernetesvia TestDriven.io & Michael HermanWhat is Kubernetes?A step-by-step tutorial that details how to deploy a Flask-based microservice (along with Postgres and Vue.js) to a Kubernetes cluster.Goals of tutorialExplain what container orchestration is and why you may need to use an orchestration toolDiscuss the pros and cons of using Kubernetes over other orchestration tools like Docker Swarm and Elastic Container Service (ECS)Explain the following Kubernetes primitives - Node, Pod, Service, Label, Deployment, Ingress, and VolumeSpin up a Python-based microservice locally with Docker ComposeConfigure a Kubernetes cluster to run locally with MinikubeSet up a volume to hold Postgres data within a Kubernetes clusterUse Kubernetes Secrets to manage sensitive informationRun Flask, Gunicorn, Postgres, and Vue on KubernetesExpose Flask and Vue to external users via an IngressBrian #3: Changes in the CI landscapeTravis CI joins the Idera family - TravisCI blog#travisAlums on Twitter“TravisCI is laying off a bunch of senior engineers and other technical staff. Look at the #travisAlums hashtag and hire them!” - alicegoldfussoptions: GitHub lists 17 options for CI, including GitLab & Azure PipelinesSome relevant articles, resources:The CI/CD market consolidation - GitLab articleAzure Pipelines with Python — by example - Anthony Shawpytest-azurepipelines - Anthony ShawAzure Pipelines Templates - Anthony SottileMichael #4: Python server setup for macOS 🍎what: hello world for Python server setup on macOS why: most guides show setup on a Linux server (which makes sense) but macoS is useful for learning and for local devSTEP 1: NGINX ➡️ STATIC ASSETSSTEP 2: GUNICORN ➡️ FLASKSTEP 3: NGINX ➡️ GUNICORNBrian #5: Learn Enough Python to be Useful: argparse How to Get Command Line Arguments Into Your Scripts - Jeff Hale“argparse is the “recommended command-line parsing module in the Python standard library.” It’s what you use to get command line arguments into your program.“I couldn’t find a good intro guide for argparse when I needed one, so I wrote this article.”Michael #6: AWS, MongoDB, and the Economic Realities of Open SourceRelated podcast: https://soundcloud.com/exponentfm/episode-159-inverted-pyramidsLast week, from the AWS blog:Today we are launching Amazon DocumentDB (with MongoDB compatibility), a fast, scalable, and highly available document database that is designed to be compatible with your existing MongoDB applications and tools. Amazon DocumentDB uses a purpose-built SSD-based storage layer, with 6x replication across 3 separate Availability Zones. The storage layer is distributed, fault-tolerant, and self-healing, giving you the the performance, scalability, and availability needed to run production-scale MongoDB workloads.Like an increasing number of such projects, MongoDB is open source…or it was anyways. MongoDB Inc., a venture-backed company that IPO’d in October, 2017, made its core database server product available under the GNU Affero General Public License (AGPL).AGPL extended the GPL to apply to software accessed over a network; since the software is only being used, not copiedMongoDB’s Business ModelWe believe we have a highly differentiated business model that combines the developer mindshare and adoption benefits of open source with the economic benefits of a proprietary software subscription business model.MongoDB enterprise and MongoDB atlasBasically, MongoDB sells three things on top of its open source database server:Additional tools for enterprise companies to implement MongoDBA hosted service for smaller companies to use MongoDBLegal certaintyWhat AWS Sellsthe value of software is typically realized in three ways:First is hardware. Second is licenses. This was Microsoft’s core business for decades: licenses sold to OEMs (for the consumer market) or to companies directly (for the enterprise market). Third is software-as-a-service.AWS announced last week:> The storage layer is distributed, fault-tolerant, and self-healing, giving you the the performance, scalability, and availability needed to run production-scale MongoDB workloads.AWS is not selling MongoDB: what they are selling is “performance, scalability, and availability.” DocumentDB is just one particular area of many where those benefits are manifested on AWS.Thus we have arrived at a conundrum for open source companies:MongoDB leveraged open source to gain mindshare.MongoDB Inc. built a successful company selling additional tools for enterprises to run MongoDB.More and more enterprises don’t want to run their own software: they want to hire AWS (or Microsoft or Google) to run it for them, because they value performance, scalability, and availability.This leaves MongoDB Inc. not unlike the record companies after the advent of downloads: what they sold was not software but rather the tools that made that software usable, but those tools are increasingly obsolete as computing moves to the cloud. And now AWS is selling what enterprises really want.This tradeoff is inescapable, and it is fair to wonder if the golden age of VC-funded open source companies will start to fade (although not open source generally). The monetization model depends on the friction of on-premise software; once cloud computing is dominant, the economic model is much more challenging.Extras:PyTexas 2019 at #Austin on Apr 13th and 14th. Registrations now open. More info at pytexas.org/2019/Michael: Sorry Ant!Michael: RustPython follow up: https://rustpython.github.io/demo/Joke:Q: Why was the developer unhappy at their job?A: They wanted arrays.Q: Where did the parallel function wash its hands?A: Async

#120 AWS, MongoDB, and the Economic Realities of Open Source and more

2019-03-0500:25:255

Sponsored by pythonbytes.fm/datadogSpecial guestsEric ChouDan BaderTrey HunnerMichael #1: Incrementally migrating over one million lines of code from Python 2 to Python 3Weighing in at over 1 million lines of Python logic, we had a massive surface area for potential issues in our migration from Python 2 to Python 3First Py3 commit, hack week 2015Unfortunately, it was clear that many features were completely broken by the upgradeOfficial start H1 2017Armed with Mypy, a static type-checking tool that we had adopted in the interim year, they made substantial strides towards enabling the Python 3 migration:Ported our custom fork of Python to version 3.5Upgraded some Python dependencies to Python 3-compatible versions, and forked some others (e.g. babel)Modified some Dropbox client code to be Python 3 compatibleSet up automated jobs in our continuous integration (CI) to run the existing unit tests with the Python 3 interpreter, and Mypy type-checking in Python 3 modeCrucially, the automated tests meant that we could be certain that the limited Python 3 compatibility that existed would not have regressed when the project was picked up again.PrerequisitesBefore we could begin working on migrating any of our application logic, we had to ensure that we could load the Python 3 interpreter and run until the entry point of the application. In the past, we had used “freezer” scripts to do this for us. However, none of these had support for Python 3 around this time, so in late 2016, we built a custom, more native solution which we internally referred to as “Anti-freeze” (more on that in the initial Python 3 migration blog post).Incrementally enabling unit tests and type-checking‘Straddling’ Python 2 and Python 3Letting it bakeLearnings (tl;dr)Unit tests and typing are invaluable.String encoding in Python is hard.Incrementally migrate to Python 3 for great profit.Eric #2: Network Automation Development with Python (for fun and for profit)Terms: NetDevOps (Cisco), NRE (Network Reliability Engineer)Libraires: Netmiko, NAPALM, Nornir Free Lab Resources: NRE Labs, dCloud, DevNetConferences: AnsibleFest (network automation track), Cisco DevnetCreateTrey #3: Alkali file as DBIf you have structured data you want to query (like RSS feed, CSV, JSON, or any custom format of your own creation) you can use a Django ORM-like syntax to query itSave it to the same format or a different format because you control both the reading and the writingKurt is at PyCascades so I got to chat with him about thisDan #4: Carnegie Mellon Launches Undergraduate Degree in Artificial Intelligence **Carnegie Mellon University's School of Computer Science will offer a new undergraduate degree in artificial intelligence beginning this fallThe first offered by a U.S. university"Specialists in artificial intelligence have never been more important, in shorter supply or in greater demand by employers," said Andrew Moore, dean of the School of Computer Science.The bachelor's degree in AI will focus more on how complex inputs — such as vision, language and huge databases — are used to make decisions or enhance human capabilitiesMichael #5: asyncio + PyQt5/PySide2via Florian Dahlitzasyncqt is an implementation of the PEP 3156 event-loop with Qt. This package is a fork of quamash focusing on modern Python versions, with some extra utilities, examples and simplified CI.Allows wiring events to Qt’s event loop that run on asyncio and leverage it internally.Example: https://github.com/gmarull/asyncqt/blob/master/examples/aiohttp_fetch.pyDan #6: 4 things I want to see in Python 4.0JIT as a first class featureA stable .0 releaseStatic type hintingA GPU story for multiprocessingMore community contributionsExtras:Michael: My Python Async webcast recording is now available.Michael: PyCon Israel in the first week of June (https://il.pycon.org/2019/), and the CFP opened today: https://cfp.pycon.org.il/conference/cfpDan: Python Basics BookJoke:Q: Why did the developer ground their kid?A: They weren't telling the truthy

#119 Assorted files as Django ORM backends with Alkali

2019-02-2600:22:131

Sponsored by pythonbytes.fm/digitaloceanBrian #1: Frozen-Flask“Frozen-Flask freezes a Flask application into a set of static files. The result can be hosted without any server-side software other than a traditional web server.”2012 tutorial, Dead easy yet powerful static website generator with FlaskSome of it is out of date, but it does point to the power of Frozen-Flask, as well as highlight a cool plugin, Flask-FlatPages, which allows pages from markdown.Michael #2: pipxby Chad SmithLast week we spoke about pythonlocExecute binaries from Python packages in isolated environments"binary" to describe a CLI application that can be run directly from the command lineFeaturesSafely install packages to isolated virtual environments, while globally exposing their CLI applications so you can run them from anywhereEasily list, upgrade, and uninstall packages that were installed with pipxRun the latest version of a CLI application from a package in a temporary virtual environment, leaving your system untouched after it finishesRun binaries from the __pypackages__ directory per PEP 582 as companion tool to pythonlocRuns with regular user permissions, never calling sudo pip install ... (you aren't doing that, are you? 😄).You can globally install a CLI application by running: pipx install PACKAGE"Just the “pipx upgrade-all” command is already a huge win over pipsi"Check out How does this compare to pipsi?Brian #3: Data science is different nowVicki BoykisThere’s lots of buzz around data science.This has resulted in loads of new data scientists looking for junior level positions.Coming from boot camps, MOOCs, self taught, remote degrees, and other training.“.. now that data science has changed from a buzzword to something even larger companies outside of the Silicon Valley bubble hire for, positions have not only become more codified, but with more rigorous entry requirements that will prefer people with previous data science experience every time.”“ … the market can be very hard, and very discouraging for the flood of beginners.”Data science is a misleading job req“The reality is that “data science” has never been as much about machine learning as it has about cleaning, shaping data, and moving it from place to place.”Advice:Don’t get into data science (this amuses me).“Don’t do what everyone else is doing, because it won’t differentiate you.”“It’s much easier to come into a data science and tech career through the “back door”, i.e. starting out as a junior developer, or in DevOps, project management, and, perhaps most relevant, as a data analyst, information manager, or similar, than it is to apply point-blank for the same 5 positions that everyone else is applying to. It will take longer, but at the same time as you’re working towards that data science job, you’re learning critical IT skills that will be important to you your entire career.”Learn the skills needed for data science todayCreating Python packagesPutting R in productionOptimizing Spark jobs so they run more efficientlyVersion controlling dataMaking models and data reproducibleVersion controlling SQLBuilding and maintaining clean data in data lakesTooling for time series forecasting at scaleScaling sharing of Jupyter notebooksThinking about systems for clean dataLots of JSONData science is turning more and more into a mostly engineering field.Data scientists need to have “good generalist engineering skills with a data background.”Michael #4: RustPythonvia Fredrik AverpilA Python-3 (CPython >= 3.5.0) Interpreter written in Rust.Seems pretty active: Latest commit ac95b61 an hour ago…GoalsFull Python-3 environment entirely in Rust (not CPython bindings)A clean implementation without compatibility hacksContributingTo start contributing, there are a lot of things that need to be done.Most tasks are listed in the issue tracker. Check issues labeled with good first issue if you wish to start coding.Rust does have direct WebAssembly support…Brian #5: Jupyter Notebook: An IntroductionMike Driscoll on RealPythonNot the “all the cool things you can do with it”, but the “really, how do I start” tutuorial.I think it should have included a mention of installing it in a venv and how to use %pip install, so I’ll include those things in these notes.Installing with pip install jupyter . Also a note that Jupyter is included with the Anaconda distribution.Note: Like everything else, I always install it in a virtual environment, if using pip, so the real installation instructions I recommend is:python3 -m venv venv --``prompt jupytersource venv/bin/activate OR venv\scripts\activate.bat if windowspip install jupyterpip install [HTML_REMOVED]jupyter notebookThat will launch a localhost web interface.Creating a new notebook within the web interface.Changing the “Untitled” name by clicking on the name. This was not obvious to me.Running cells, including the shift-enter keyboard shortcut.A run through the menu, stopping at non-obvious places“File” has “Save and Checkpoint” which is super cool.“Edit” has cell cut, copy, paste. But also has delete, split, merge, and cell movement.“Cell” menu has lots of cool run options, like “Run all above” and “Run all below” and others.Not just Python, but you can have a terminal sessions and more from within Jupyter.A look at the “Running” tab.Quick overview of the markdown support for markdown cellsExporting notebooks using jupyter nbconvertExtra notes on installing packages from Jupyter:To pip install from the notebook, do this: %pip install numpy in a code cell.Michael #6: Python Developers Survey 2018 ResultsPython usage as a main language is up 5 percentage points from 79% in 2017 when Python Software Foundation conducted its previous survey.What do you use Python for? (2018/2017)59%/51% Data analysis56%/54% Web dev39%/32% MLWeb development is the only category with a large gap (56% vs. 36%) separating those using Python as their main language vs. as a supplementary language. For other types of development, the differences are much smaller.What do you use Python for the most? (single answer)29%/29% web dev17%/17% data analysis11%/8% MLLike last year:27% (Web development) ≈ 28% (Scientific development) Science = 17% + 11% for Data analysis + Machine learningPython 3 vs Python 284% Python 3 vs 16% Python 2. The use of Python 3 continues to grow rapidly. According to the latest research in 2017, 75% were using Python 3 compared with 25% for Python 2.Top 4 web frameworks (majority to the first two):FlaskDjangoTornadoPyramidDatabasesPostgreSQLMySQLSQLiteMongoDBORMsSQLAlchemy and Django ORM tiedExtras:“Mentored sprints for diverse beginners” at PyCon“A newcomer’s introduction to contributing to an open source project”https://us.pycon.org/2019/hatchery/mentoredsprints/Call for applications for projects open Feb 8 to March 14Call for contributors, participants in the sprint also open Feb 8 to March 14“If you are wondering if this event is for you: it definitely is and we would love to have you taking part in this sprint.”“This mentored sprint will take place on Saturday, May 4th, 2019 from 2:35pm to 6:30pm”Joke:via FlorianQ: If you have some pseudo code (say in sample.txt) how do you most easily convert it to Python?A: Change the extension to .py Extra Joke: Python Song (with chapters!)

#118 Better Python executable management with pipx

2019-02-2200:25:541

Sponsored by pythonbytes.fm/datadogBrian #1: Goodbye Virtual Environments?by Chad Smithvenv’s are great but they introduce some problems as well:Learning curve: explaining “virtual environments” to people who just want to jump in and code is not always easyTerminal isolation: Virtual Environments are activated and deactivated on a per-terminal basisCognitive overhead: Setting up, remembering installation location, activating/deactivatingPEP 582 — Python local packages directoryThis PEP proposes to add to Python a mechanism to automatically recognize a __pypackages__directory and prefer importing packages installed in this location over user or global site-packages. This will avoid the steps to create, activate or deactivate “virtual environments”. Python will use the __pypackages__ from the base directory of the script when present.Try it now with pythonlocpythonloc is a drop in replacement for python and pip that automatically recognizes a __pypackages__ directory and prefers importing packages installed in this location over user or global site-packages. If you are familiar with node, __pypackages__ works similarly to node_modules.Instead of running python you run pythonloc and the __pypackages__ path will automatically be searched first for packages. And instead of running pip you run piploc and it will install/uninstall from __pypackages__.Michael #2: webassetsBundles and minifies CSS & JS filesBeen doing a lot of work to rank higher on the sitesThat lead me to Google’s LighthouseDespite 25ms response time to the network, Google thought my site was “kinda slow”, yikes!webassets has integration for the big three: Django, Flask, & Pyramid.But I prefer to just generate them and serve them off disk def build_asset(env: webassets.Environment, files: List[str], filters: str, output: str): bundle = webassets.Bundle( *files, filters=filters, output=output, env=env ) bundle.build(force=True)Brian #3: Bernat on Python Packaging3 part series by Bernat GaborMaintainer of tox and virtualenv Python packages.The State of Python PackagingPython packaging - Past, Present, FuturePython packaging - Growing PainsMichael #4: What the mock? — A cheatsheet for mocking in PythonNice introductionSome examples @mock.patch('work.os') def test_using_decorator(self, mocked_os): work_on() mocked_os.getcwd.assert_called_once()And def test_using_context_manager(self): with mock.patch('work.os') as mocked_os: work_on() mocked_os.getcwd.assert_called_once()Brian #5: Transitions: The easiest way to improve your tech talkBy Saron YitbarekJeff Atwood of CodingHorror noted “The people who can write and communicate effectively are, all too often, the only people who get heard. They get to set the terms of the debate.”Effectively presenting is part of effective communication.I love the focus of this article. Focused on one little aspect of improving the performance of a tech talk.Michael #6: Steering council announcedOur new leaders areBarry WarsawBrett CannonCarol WillingGuido van RossumNick CoghlanVia Joe CareyWe both think it’s great Guido is on the council.Brian: Got interviewed on IT Energizer Podcast:The one with Brian: https://itcareerenergizer.com/e123/The one with Michael: https://itcareerenergizer.com/e83/PyCon LATAMAugust 29, Puerto Vallarta, MexicoWe should go. Anyone want to sponsor our travel/hotel to this event?CFP open till May 31, 2019, https://www.pylatam.org/en/speaking/From the list from Ant, my votes. Q: What's the second movie about a database engineer called?A: The SQL.!falseIt's funny 'cause it's true.A programmer's spouse tells them, "Run to the store and pick up a loaf of bread. If they have eggs, get a dozen."The programmer comes home with 12 loaves of bread.

#117 Is this the end of Python virtual environments?

2019-02-1400:28:204

Sponsored by pythonbytes.fm/digitaloceanBrian #1: Inside python dict — an explorable explanationInteractive tutorial on dictionariesSearching efficiently in a listWhy are hash tables called has tables?Putting it all together to make an “almost”-Python-dictHow Python dict really works internallyYes this is a super deep dive, but wow it’s cool.Tons of the code is runnable right there in the web page, including moving visual representations, highlighted code with current line of code highlighted.Some examples allow you to edit values and play with stuff.Michael #2: Embed Python in Unreal Engine 4You may notice a theme throughout my set of picks on this episodeGames built on Unreal Engine 4 includeFortnite: Save the WorldGears of War 4Marvel vs. Capcom: InfiniteMoto Racer 4System Shock (remake)Plugin embedding a whole Python VM in Unreal Engine 4 (both the editor and runtime).This means you can use the plugin to write other plugins, to automate tasks, to write unit tests and to implement gameplay elements.Here is an example usage. It’s a really nice overview and tutorial for the editor.For game elements, check out this section.Brian #3: Redirecting stdout with contextlibWhen I want to test the stdout output of some code, that’s easy, I grab the capsys fixture from pytest.But what if you want to grab the stdout of a method NOT while testing?Enter [contextlib.redirect_stdout(new_target)](https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout)so cool. And very easy to read.ex: f = io.StringIO() with redirect_stdout(f): help(pow) s = f.getvalue()also a version for stderrMichael #4: Panda3Dvia Kolja LubitzPanda3D is an open-source, completely free-to-use engine for realtime 3D games, visualizations, simulations, experimentsNot just games, could be science as well!The full power of the graphics card is exposed through an easy-to-use API. Panda3D combines the speed of C++ with the ease of use of Python to give you a fast rate of development without sacrificing on performance.Features:Platform PortabilityFlexible Asset Handling: Panda3D includes command-line tools for processing and optimizing source assets, allowing you to automate and script your content production pipeline to fit your exact needs.Library Bindings: Panda3D comes with out-of-the-box support for many popular third-party libraries, such as the Bullet physics engine, Assimp model loader, OpenALPerformance Profiling: Panda3D includes pstats — an over-the-network profiling system designed to help you understand where every single millisecond of your frame time goes.Brian #5: Why PyPI Doesn't Know Your Projects DependenciesSome questions you may have asked:> How can I produce a dependency graph for Python packages?> Why doesn’t PyPI show a project’s dependencies on it’s project page?> How can I get a project’s dependencies without downloading the package?> Can I search PyPI and filter out projects that have a certain dependency?If everything is in requirements.txt, you just might be able to, but…setup.py is dynamic. You gotta run it to see what’s needed.Dependencies might be environment specific. Windows vs Linux vs Mac, as an example.Nothing stopping someone from putting random.choice() for dependencies in a setup.py file. But that would be kinda evil. But could be done. (Listener homework?)The wheel format is way more predictable because it limits some of this freedom. wheels don’t get run when they install, they really just get unpacked.More info on wheels: Kind of a tangent, but what why not:From: https://pythonwheels.com“Advantages of wheelsFaster installation for pure Python and native C extension packages.Avoids arbitrary code execution for installation. (Avoids setup.py)Installation of a C extension does not require a compiler on Linux, Windows or macOS.Allows better caching for testing and continuous integration.Creates .pyc files as part of installation to ensure they match the Python interpreter used.More consistent installs across platforms and machines.”Michael #6: PyGame seriesvia Matthew WardLearn how to program in Python by building a simple dice gameBuild a game framework with Python using the PyGame moduleHow to add a player to your Python gameUsing PyGame to move your game character aroundWhat's a hero without a villain? How to add one to your Python gamePut platforms in a Python game with PyGameAlso: Shout out to Mission Python book: Code a Space Adventure Game!via @realpythonWhy do Pythons live on land? They are above C-level!

#116 So you want Python in a 3D graphics engine?

2019-02-0600:17:562

Sponsored by pythonbytes.fm/datadogSpecial guest: Nina ZakharenkoBrian #1: Great ExpectationsA set of tools intended for batch time testing of data pipeline data.Introduction to the problem doc: Down with Pipeline debt / Introducing Great Expectationsexpect_[something]() methods that return json formatted descriptions of whether or not the passed in data matches your expectations.Can be used programmatically or interactively in a notebook. (video demo).For programmatic use, I’m assuming you have to put code in place to stop a pipeline stage if expectations aren’t met, and write failing json result to a log or something.Examples, just a few, full list is big:Table shape: expect_column_to_exist, expect_table_row_count_to_equalMissing values, unique values, and types: - expect_column_values_to_be_unique, expect_column_values_to_not_be_nullSets and rangesexpect_column_values_to_be_in_setString matchingexpect_column_values_to_match_regexDatetime and JSON parsingAggregate functionsexpect_column_stdev_to_be_betweenColumn pairsDistributional functionsexpect_column_chisquare_test_p_value_to_be_greater_thanNina #2: Using CircuitPython and MicroPython to write Python for wearable electronics and embedded platformsI’ve been playing with electronics projects as a hobby for the past two years, and a few months ago turned my attention to Python on microcontrollersMicroPython is a lean and efficient implementation of Python3 that can run on microcontrollers with just 256k of code space, and 16k of RAM. CircuitPython is a port of MicroPython, optimized for Adafruit devices. Some of the devices that run Python are as small as a quarter.My favorite Python hardware platform for beginners is Adafruit’s Circuit PlayGround Express. It has everything you need to get started with programming hardware without soldering. All you’ll need is alligator clips for the conductive pads.The board features NeoPixel LEDs, buttons, switches, temperature, motion, and sound sensors, a tiny speaker, and lots more. You can even use it to control servos, tiny motor arms.Best of all, it only costs $25.If you want to program the Circuit PlayGround Express with a drag-n-drop style scratch-like interface, you can use Microsoft’s MakeCode. It’s perfect for kids and you’ll find lots of examples on their site.Best of all, there are tons of guides for Python projects to build on their website, from making your own synthesizers, to jewelry, to silly little robots. Check out the repo for my Python-powered earrings, see a photo, or a demo. Sign up for the Adafruit Python for Microcontrollers mailing list here, or see the archives here. Michael #3: Data class CSV readerMap CSV to Data Classes You probably know about reading CSV filesMaybe as tuplesBetter with csv.DictReaderThis library is similar but maps Python 3.7’s data classes to rows of CSV filesIncludes type conversions (say string to int)Automatic type conversion. DataclassReader supports str, int, float, complex and datetimeDataclassReader use the type annotation to perform validation of the data of the CSV file.Helps you troubleshoot issues with the data in the CSV file. DataclassReader will show exactly in which line of the CSV file contain errors.Extract only the data you need. It will only parse the properties defined in the dataclassIt uses dataclass features that let you define metadata properties so the data can be parsed exactly the way you want.Make the code cleaner. No more extra loops to convert data to the correct type, perform validation, set default values, the DataclassReader will do all this for youDefault fallback values, more.Brian #4: How to Rock Python Packaging with Poetry and BriefcaseStarts with a discussion of the packaging (for those readers that don’t listen to Python Bytes, I guess.) However, it also puts flit, pipenv, and poetry in context with each other, which is nice.Runs through a tutorial of how to build a pyproject.toml based project using poetry and briefcase.We’ve talked about Poetry before, on episode 100.pyproject.toml is discussed extensively on Test & Code 52.briefcase is new, though, it’s a project for creating standalone native applications for Mac, Windows, Linux, iOS, Android, and more.The tutorial also discusses using poetry directly to publish to the test-pypi server. This is a nice touch. Use the test-pypi before pushing to the real pypi. Very cool.Nina #5: awesome-python-security *🕶🐍🔐, a collection of tools, techniques, and resources to make your Python more secure*All of your production and client-facing code should be written with security in mindThis list features a few resources I’ve heard of such as Anthony Shaw’s excellent 10 common security gotchas article which highlights problems like input injection and depending on assert statements in production, and a few that are new to me:OWASP (Open Web Application Security Project) Python Resources at pythonsecurity.orgbandit a tool to find common security issues in Pythonbandit features a lot of useful plugins, that test for issues like:hardcoded password stringsleaving flask debug on in productionusing exec() in your code& moredetect-secrets, a tool to detect secrets left accidentally in a Python codebase& lots more like resources for learning about security concepts like cryptographySee the full list for moreMichael #6: pydbgPython implementation of the Rust dbg macroBest seen with an example. Rather than printing things you want to inspect, you: a = 2 b = 3 dbg(a+b) def square(x: int) -> int: return x * x dbg(square(a))outputs: [testfile.py:4] a+b = 5 [testfile.py:9] square(a) = 4Extras:Brian:pathlib + pytest tmpdir → tmp_path & tmp_path_factoryhttps://docs.pytest.org/en/latest/tmpdir.htmlThese two new fixtures (as of pytest 3.9) act like the good old tmpdir and tmpdir_factory, but return pathlib Path objects. Awesome.Michael: The Art of Python is a miniature arts festival at PyCon North America 2019, focusing on narrative, performance, and visual art. We intend to encourage and showcase novel art that helps us share our emotionally charged experiences of programming (particularly in Python). We hope that by attending, our audience will discover new aspects of empathy and rapport, and find a different kind of delight and perspective than might otherwise be expected at a large conference.StackOverflow Survey is Open! https://stackoverflow.az1.qualtrics.com/jfe/form/SV_1RGiufc1FCJcL6B NumPy Is Awaiting Fix for Critical Remote Code Execution Bugvia Doug SheehanThe issue was raised on January 16 and affects NumPy versions 1.10 (released in 2015) through 1.16, which is the latest release at the moment, released on January 14The problem is with the 'pickle' module, which is used for transforming Python object structures into a format that can be stored on disk or in databases, or that allows delivery across a network.The issue was reported by security researcher Sherwel Nan, who says that if a Python application loads malicious data via the numpy.load function an attacker can obtain remote code execution on the machine.Get your google dataAll google docs in MS Office format via https://takeout.google.com/settings/takeoutAll Gmail in MBOX format from there as wellHint: Start with nothing selected ;)Nina:I’m teaching a two day Intro and Intermediate Python course on March 19th and 20th. The class will live-stream for free here on each day of or join in-person from downtown Minneapolis. All of the course materials will be released for free as well.I recently recorded a series of videos with Carlton Gibson (Django maintainer) on developing Django Web Apps with VS Code, deploying them to Azure with a few clicks, setting up a Continuous Integration / Continuous Delivery pipeline, and creating serverless apps. Watch the series here: https://aka.ms/python-videosI’ll be a mentor at a brand new hatchery event at PyCon US 2019, mentored sprints for diverse beginners organized by Tania Allard. The goal is to help underrepresented folks at PyCon contribute to open source in a supportive environment. The details will be located here (currently a placeholder) when they’re finalized.Catch my talk about electronics projects in Python with LEDs at PyCascades in Seattle on February 24th. Currently tickets are still for sale. If you haven’t tried the Python extension for VS Code, now is a great time. The December release included some killer features, such as remote Jupyter support, and exporting Python files as Jupyter notebooks. Keep up with future releases at the Python at Microsoft blog. Q: What do you call a snake that only eats desert? A: A pie-thon. (might not make sense read out loud)Q: How do you measure a python? A: In inches. They don't have any feet! Q: What is a python’s favorite subject? Hiss-tory!

#115 Dataclass CSV reader and Nina drops by

2019-02-0200:28:585

Sponsored by pythonbytes.fm/digitaloceanBrian #1: What should be in the Python standard library?on lwn.net by Jake EdgeThere was a discussion recently about what should be in the standard library, triggered by a request to add LZ4 compression.Kinda hard to summarize but we’ll try:Jonathan Underwood proposed adding LZ4 compression to stdlib.Can of worms openedzlib and bz2 already in stdlibBrett proposed making something similar to hashlib for compression algorithms.Against adding it:lz4 not needed for stdlib, and actually, bz2 isn’t either, but it’s kinda late to remove.PyPI is easy enough. put stuff there.Led to a discussion of the role of stdlib.If it’s batteries included, shouldn’t we add new batteriesSome people don’t have access to PyPI easilyDo we never remove elements? really?Maybe we should have a lean stdlib and a thicker standard distribution of selected packageswho would decide?same problem exists then of depending on it. How to remove stuff?Steve Dower would rather see a smaller standard library with some kind of "standard distribution" of PyPI modules that is curated by the core developers.A leaner stdlib could speed up Python version schedules and reduce burden on core devs to maintain seldom used packages.See? can of worms.In any case, all this would require a PEP, so we have to wait until we have a PEP process decided on.Michael #2: Data Science portal for Home Assistant launchedvia Paul CutlerHome Assistant is launching a data science portal to teach you how you can learn from your own smart home data. In 15 minutes you setup a local data science environment running reports.A core principle of Home Assistant is that a user has complete ownership of their personal data. A users data lives locally, typically on the SD card in their Raspberry PiThe Home Assistant Data Science website is your one-stop-shop for advice on getting started doing data science with your Home Assistant data.To accompany the website, we have created a brand new Hass.io Add-on JupyterLab lite, which allows you to run a data science IDE called JupyterLab directly on your Raspberry Pi hosting Home Assistant. You do your data analysis locally, your data never leaves your local machine. When you build something cool, you can share the notebook without the results, so people can run it at their homes too.We have also created a Python library called the HASS-Data-Detective which makes it super easy to get started investigating your Home Assistant data using modern data science tools such as Pandas.Check out the Getting Started notebookIoT aside: I finally found my first IoT project: Recording in progress button.Brian #3: What's the future of the pandas library?Kevin Markham over at dataschool.iopandas is gearing up to move towards a 1.0 release. Currently rc-ing 0.24Plans are to get there “early 2019”.Some highlightsmethod chaining - encouraged by core teamto encourage further, more methods will support chainingApache arrow likely to be part of pandas backend sometime after 1.0Extension arrays - allow you to create custom data typesdepricationsinplace parameter. It doesn’t work with chaining, doesn’t actually prevent copies, and causes codebase complexityix accessor, use loc and iloc insteadPanel data structure. Use MultiIndex insteadSparseDataFrame. Just use a normal DataFramelegacy python supportMichael #4: PyOxidizerPyOxidizer is a collection of Rust crates that facilitate building libraries and binaries containing Python interpreters.PyOxidizer is capable of producing a single file executable - with all dependencies statically linked and all resources (like .pyc files) embedded in the executableThe Oxidizer part of the name comes from Rust: executables produced by PyOxidizer are compiled from Rust and Rust code is responsible for managing the embedded Python interpreter and all its operations.PyOxidizer is similar in nature to PyInstaller, Shiv, and other tools in this space. What generally sets PyOxidizer apart is Produced executables contain an embedded, statically-linked Python interpreterhave no additional run-time dependency on the target systemruns everything from memory (as opposed to e.g. extracting Python modules to a temporary directory and loading them from there).Brian #5: Working With Files in Pythonby Vuyisile Ndlovu on RealPythonVery comprehensive write up on working with files and directoriesIncludes legacy and modern methods. Pay attention to pathlib parts if you are using 3.4 plusAlso great for “if you used to do x, here’s how to do it with pathlib”.Included:Directory listingsgetting file attributescreating directoriesfile name pattern matchingtraversing directories doing stuff with the files in therecreating temp directories and filesdeleting, copying, moving, renamingarchiving with zip and tar including reading thoselooping over filesMichael #6: $ python == $ python3?via David FurphyHomebrew tried this recently & got "persuaded" to reverse. Also in recent discussion of edits to PEP394, GvR said absolutely not now, probably not ever.Guido van RossumRE: python doesn’t exist on macOS as a command: Did you mean python2 there? In my experience macOS comes with python installed (and invoking Python 2) but no python2 link (hard or soft). In any case I'm not sure how this strengthens your argument.I'm also still unhappy with any kind of endorsement of python pointing to python3. When a user gets bitten by this they should receive an apology from whoever changed that link, not a haughty "the PEP endorses this".Regardless of what macOS does I think I would be happier in a future where python doesn't exist and one always has to specify python2 or python3. Quite possibly there will be an age where Python 2, 3 and 4 all overlap, and EIBTI.Extras:Michael: A letter to the Python community in Africavia Anthony ShawBelieve the broader international Python and Software community can learn a lot from what so many amazing people are doing across Africa.e.g. The attendance of PyCon NA was 50% male and 50% female.Joke:via Luke Russell:A: “Knock Knock”B: “Who’s There"A: ……………………………………………………………………………………….“Java”Also: Java 4EVER video is amazing: youtube.com/watch?v=kLO1djacsfg

#114 What should be in the Python standard library?

2019-01-2600:28:333

Sponsored by https://pythonbytes.fm/digitaloceanBrian #1: Advent of Code 2018 SolutionsMichael FoglemanEven if you didn’t have time or energy to do the 2018 AoC, you can learn from other peoples solutions. Here’s one set written up in a nice blog post.Michael #2: Python Lands on the Windows 10 App StorePython Software Foundation recently released Python 3.7 as an app on the official Windows 10 app store. Python 3.7 is now available to install from the Microsoft Store, meaning you no longer need to manually download and install the app from the official Python website.there is one limitation. “Because of restrictions on Microsoft Store apps, Python scripts may not have full write access to shared locations such as TEMP and the registry. Discussed with Steve Dower over on Talk Python 191Brian #3: How I Built A Python Web Framework And Became An Open Source MaintainerFlorimond MancaBocadillo - “A modern Python web framework filled with asynchronous salsa””maintaining an open source project is a marathon, not a sprint.”Tips at the end of the article include tips for the following topics, including recommendations and tool choices:Project definitionMarketing & CommunicationCommunityProject managementCode qualityDocumentationVersioning and releasingMichael #4: Python maintainability score via Wilyvia Anthony ShawA Python application for tracking, reporting on timing and complexity in testsEasiest way to calculate it is with wily https://github.com/tonybaloney/wily … the metrics are ‘maintainability.mi’ and ‘maintainability.rank’ for a numeric and the A-F scale.Build an index: wily build srcInspect report: wily report fileGraph: wily graph file metricBrian #5: A couple fun awesome listsAwesome Python Security resourcesToolsweb framework hardening, ex: secure.pymulti toolsstatic code analysis, ex: banditvulnerabilities and security advisoriescryptographyapp templatesEducationlots of resources for learningCompaniesAwesome Flake8 Extensionsclean codetesting, including flake8-pytest - Enforces to use pytest-style assertionsflake8-mock - Provides checking mock non-existent methodssecuritydocumentationenhancementscopyrightsMichael #6: fastloggingvia Robert YoungA faster replacement of the standard logging module with a mostly compatible API.For a single log file it is ~5x faster and for rotating log file ~13x faster.It comes with the following features:(colored, if colorama is installed) logging to consolelogging to file (maximum file size with rotating/history feature can be configured)old log files can be compressed (the compression algorithm can be configured)count same successive messages within a 30s time frame and log only once the message with the counted value.log domainslog to different fileswriting to log files is done in (per file) background threads, if configuredconfigure callback function for custom detection of same successive log messagesconfigure callback function for custom message formatterconfigure callback function for custom log writerMichael: My webcast on async, Jan 24, 11am PTMichael: Watch your YAML! Joke: >>> import antigravity

#113 Python Lands on the Windows 10 App Store

2019-01-1800:23:221

Comments (10)

Raj

Thanks for the kubernetes example, and overall good episode

Mar 22nd

Mian A. Shah

ypf

Jan 28th

GreatBahram

As usual, perfect!

Jan 27th

Antonio Andrade

I think you missed to highlight all the nice work of realphlython and your podcasts, these are key stuffs for Python in 2018!

Dec 27th

Vignesh Anand Krishnan

The jokes are good but let brian do it. 😂

Dec 13th

Congrats Python Bytes. This episode was really great 😎

Oct 27th

pyguy

Joel Grus talk can be found here: https://youtu.be/7jiPeIFXb6U

Oct 8th

víbora means in Spanish: snake. umm, just thinking about Phyton

Aug 4th

It's intetesting the title is flask but you guys spoke more about Django? kidding? hahaha please dont mess with us《Mico framework fans Thanks

Jun 28th

nice, another super good Python postcast

May 20th